Thursday Sep 12, 2013

Stream Relational Transactions into Big Data Systems

Are you one of the organizations adopting ‘big data systems’ to manage and analyze a class of data typically referred to as big data? If so, you may know that big data includes data that could be structured, semi-structured or unstructured, each of which originates from a variety of different sources.  Another characterization of big data is described by the data's volume, velocity, and veracity. Due to its promise to help harness the data deluge we are faced with, the adoption of big data solutions is becoming quite pervasive. In this blog post I’d like discuss how to leverage Oracle GoldenGate’s real-time replication for big data systems.

The term 'big data systems' is an umbrella terminology used in general to discuss a wide variety of technologies each of which is used for a specific purpose. Broadly speaking, big data technologies address the needs for batch, transactional, and real-time processing requirements. Using the appropriate big data technology is highly dependent on the use case being addressed.

While gaining business intelligence from transactional data continues to be a dominant factor in the decision making process, businesses have realized that gaining intelligence from other forms of data they have been collecting will enable them achieve a more complete view, address additional business objectives, and lead to better decision making. The following table illustrates some examples of various industry verticals, forms of data, and the objective the business attempts to achieve using the other forms of data.

Industry

Data

Objective

Healthcare

Practitioner’s notes, machine statistics.

Best practices and reduced hospitalization.

Retail

Weblog, click streams.

Micro-segmentation recommendations.

Banking

Weblogs, fraud reports.

Fraud detection, risk analysis.

Utilities

Smart meter reading, call center data.

Real-time and predictive utilization analysis.

Role of transactional data

When using other forms of data for analytics, better contextual intelligence is obtained when the analysis is combined with transactional data. Especially low-latency transactional data brings additional value to dynamically changing operations that day-old data cannot deliver. In organizations, a vast majority of applications' transactional data is captured in relational databases. In order to ensure an efficient supply of transactional data for big data analytics, there are several requirements that the data integration solution should address:

<!--[if !supportLists]-->· <!--[endif]-->Reliable change data capture and delivery mechanism

<!--[if !supportLists]-->· <!--[endif]-->Minimal resource consumption when extracting data from the relational data source

<!--[if !supportLists]-->· <!--[endif]-->Secured data delivery

<!--[if !supportLists]-->· <!--[endif]-->Ability to customize data delivery

<!--[if !supportLists]-->· <!--[endif]-->Support heterogeneous database sources

<!--[if !supportLists]-->· <!--[endif]-->Easy to install, configure and maintain

A solution which can reliably stream database transactions to a desired target enables that the effort is spent on data analysis rather than data acquisition. Also, when the solution is non-intrusive and minimally impacts the source database, it minimizes the need for additional resources and changes on the source database.

Oracle GoldenGate is a time tested and proven product for real-time, heterogeneous relational database replication. Oracle GoldenGate addresses the challenges listed above and is widely used by organizations for mission critical data replication among relational databases. Furthermore, GoldenGate moves transactional data in real-time to support timely operational business intelligence needs.

Oracle GoldenGate Integration Options for Big Data Analytics

There is a variety of integration options available with the Oracle GoldenGate product that facilitates delivering transactions on relational databases into non-relational targets.

Oracle GoldenGate provides pre-built adapters which integrate with Flat Files and Messaging Systems. Please refer to Oracle GoldenGate for Java - Administration Guide and Oracle GoldenGate for Flat Files -Administration Guide for more information.

Oracle GoldenGate also provides Java APIs and a framework for developing custom integrations to Java enabled targets. Using this capability, custom adapters or handlers can be developed to address specific requirements. In this blog post I’d like focus on Oracle GoldenGate Java APIs for developing custom integrations to big data systems.

As we mentioned earlier, 'big data systems' is an umbrella terminology used in general to describe a wide variety of technologies, each of which is used for a specific purpose. Among the various big data systems, Hadoop and its suite of technologies are widely adopted by various organizations for processing big data. The below diagram illustrates a general high level architecture for integrating with Hadoop.

<!--[if !vml]--><!--[endif]--> <!--[if !vml]--><!--[endif]-->

Custom Adapter

<!--[if !vml]--><!--[endif]--> <!--[if !vml]--><!--[endif]--> <!--[if !vml]--><!--[endif]--> <!--[if !vml]--><!--[endif]-->

Pump Parameter file

Adapter Properties file

<!--[if !vml]-->

You can implement custom adapter or handler for the big data system using Oracle GoldenGate's Java API. The custom adapter is deployed as an integral part of the Oracle GoldenGate Pump process. The Pump and the custom adapter are configured through the Pump parameter file and custom adapter's properties file respectively. Depending upon the requirements, the properties for the custom adapter will need to be determined and implemented.

The Pump process will execute the adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter will write the transactions into Hadoop.

Enabling the co-existence of big data systems with relational systems will benefit organizations to better serve customers and improve decision-making capabilities. Oracle GoldenGate, which has an excellent record of empowering IT on the various aspects of data management requirements, provides the capability to integrate with big data systems. In the upcoming blog posts, we will discuss in depth the implementation and the configuration of integrating Oracle GoldenGate with Hadoop technologies. 

Monday Sep 09, 2013

Boosting Oracle Applications with Oracle GoldenGate: Hear from Masterminds

In recent blogs we have mentioned new data integration solutions and resources for Oracle Applications. Oracle Data Integration products provide incremental value, the AppAdvantage, to Oracle Application users by maximizing performance and availability, and by enabling real-time insight.

Many Oracle Applications customers such as Herbalife and BT, have seen great results from GoldenGate’s solutions. Herbalife used GoldenGate for offloading Oracle E-Business Suite operational reporting to an ODS. As a result the company increased performance significantly while improving the business reports using real-time data. BT used GoldenGate for upgrading its Siebel CRM system to the latest version without impacting operations, with minimized risk, and improved system performance by 5%.

There are many more customers who achieved major performance, availability, and improved insight. This year at Oracle OpenWorld we decided to showcase GoldenGate’s strong offering for Oracle Applications in a customer panel:

A Panel of Masterminds: Perspectives from Oracle GoldenGate Customers

Tuesday, Sep 24, 3:45 PM - 4:45 PM - Moscone West – 2024

In this session industry leaders, Smuckers , Veolia Water, and Land O’Lakes, will talk about their use of GoldenGate for Oracle Applications including E-Business Suite and JD Edwards. The panelists will discuss the best practices and tips for using GoldenGate’s real-time data replication capabilities in offloading reporting to a reporting environment and migrating or upgrading their application environment.

If you attended previous OpenWorld’s customer panels within the Data Integration track, you know there will be great depth in the discussion. For example, last year we had two customer panels for GoldenGate. In one panel St. Jude Medical, Equifax, and Bank of America, presented how they achieved eliminated planned downtime for critical systems using GoldenGate, and in another panel Comcast, Raymond James, and Turk Telekom discussed operational reporting solutions using GoldenGate. 

Both sessions had great audience participation and I expect this year’s panels will be very interactive too.

Another related session on using GoldenGate for Oracle Applications is going to be delivered by the product management team. GoldenGate PM will be joined with PMs from Oracle  BI Applications and Oracle Coherence teams. The team will present latest  updates such as tight integration with Oracle Coherence, and solutions for Oracle BI Applications, E-Business Suite, Siebel CRM, ATG Web Commerce and more.

Powering Solutions for Oracle Applications with Oracle GoldenGate

Wednesday, Sep 25, 11:45 AM - 12:45 PM - Moscone West - 2024

 I hope those of you who will be at OpenWorld will join these sessions. If you will not be at OpenWorld, you can visit our new resource center to download free screencasts and white papers on our solutions for Oracle Applications.

Friday Aug 23, 2013

Ensuring Data Consistency with Oracle GoldenGate Veridata

Today’s 24/7/365, fast-paced world brought the need for high data availability and with that came redundant distributed copies of the data. In our complex IT environment, maintaining data consistency across these distributed systems is not a trivial task unfortunately. Data discrepancy occurs for many reasons, including database configuration differences between source and target, configuration errors during data migration, user errors, gaps in data replication or replication lags. Even when using products that replicate data reliably, such as Oracle GoldenGate, there remain potential causes of data discrepancy.

When data discrepancies occur, businesses can see its impact in the form of incorrect decision-making; failed service-level agreements; and ultimately, operational, financial, and legal risk.

Oracle GoldenGate Veridata helps companies avoid these risks and operate their redundant systems with high confidence knowing that their backup systems, reporting databases, multi-master replication environments have consistency. Oracle GoldenGate Veridata provides an easy-to-use yet powerful solution for identifying out-of-synch data before it negatively impacts the business.

Deployed together with the Oracle GoldenGate real-time data replication  product or separately, Oracle GoldenGate Veridata ensures data consistency is maintained across databases. The software enables periodic checks—as frequently as desired—between heterogeneous source and the target databases without taking either system offline.

Some of the benefits of the product include:

  • Reduced risk exposure. Knowing about data inconsistencies can help address issues before they impact business operations. Oracle GoldenGate Veridata identifies data discrepancies so that corrective measures can be quickly taken.
  • Improve IT Productivity. Oracle GoldenGate Veridata's automated data comparison capabilities reduces the time and resources required for data validation Graphical illustrations and detailed reports clearly convey the key results and specific data disparities so that IT can take immediate corrective action. If there are no inconsistencies, users can document the accuracy of their data for regulatory compliance reporting needs.
  • Continuous operation during validation. With its ability to compare in-flight transactions, Oracle GoldenGate Veridata supports databases that are in use without interrupting business operations. Combining a light footprint and groundbreaking technology, the application imposes minimal impact on technology infrastructure.

If your organization has strict requirements to keep redundant systems consistent, I highly recommend reading more about Oracle GoldenGate Veridata product via our new white paper: Ensuring Data Consistency with Oracle GoldenGate Veridata.

You can also find out best practices on installation and configuration in the Oracle GoldenGate Veridata Handbook on My Oracle Support (Knowledge ID 1583563.1.)

Monday Jul 01, 2013

Oracle GoldenGate 11g Certified Implementation Beta Exam Available

We have great news for Oracle Data Integration partners:  Oracle GoldenGate 11g Certified Implementation Beta Exam is now available. 

The Oracle GoldenGate 11g Certified Implementation Exam Essentials (1Z1-481) exam is designed for individuals who possess a strong foundation and expertise in selling and implementing Oracle Data Integration 11g solutions.

This certification covers topics such as:

  • Oracle GoldenGate 11g Overview
  • Architecture Overview, 
  • Configuring Oracle GoldenGate
  • Parameters, Mapping and Transformation Overview, 
  • Configuration Options, 
  • Managing and Monitoring Oracle GoldenGate 11g. 
This certification helps OPN members differentiate themselves in the marketplace through proven in-depth expertise and helps their partner company qualify for the Oracle Data Integration 11g Specialization Criteria. We recommend up-to-date training and field experience. OPN members earning this certification will be recognized as OPN Certified Specialists.

Request a discounted beta voucher today using the OPN Beta Certified Specialist Exam  Voucher Request Form
You can take the exam now at a near-by Pearson VUE testing center.

Friday Jun 28, 2013

Upgrading Oracle Siebel CRM Application Without Downtime

Oracle’s Siebel Customer Relationship Management (CRM) software helps organizations differentiate their businesses to achieve top- and bottom-line growth. Siebel CRM delivers comprehensive solutions that are tailored to more than 20 different industries. As Siebel CRM implementations have evolved into mission critical, operational business processes that must operate 24/7, companies are finding it increasingly difficult to afford the downtime typically required to perform an in-place upgrade. Without these upgrades, businesses loose out on critical new features and functionality.


With Oracle GoldenGate, customers don’t have to choose between upgrades and outages. Oracle GoldenGate allows Siebel CRM customers to perform upgrades with zero downtime. Now Siebel customers can always take advantages of the latest innovations in customer relationship management without having to worry about potential lost revenue due to downtime.


Oracle GoldenGate provides three different deployment models for Siebel CRM zero downtime upgrades that are designed to meet differing customer requirements. These range from a basic unidirectional model, which is designed to work out-of-the-box, to the most sophisticated active-active model for phased migrations.


If you have mission-critical Siebel CRM implementations I recommend that you watch the screencast below to learn how you can begin taking advantage of all the latest Siebel enhancements without having any downtime.

This screencast is also available on Oracle Media Network and Oracle's YouTube channel. For even more details I recommend reading the whitepaper Upgrading Siebel CRM with Zero Downtime .

Friday Jun 21, 2013

What Comes Next After You Decide on Using Oracle Exadata

As Oracle Exadata continues to expand its footprint for both transaction and analytical processing, moving existing systems to Exadata and feeding it with enterprise data on an ongoing basis have become important discussion topics for Exadata customers. Consolidation and migration is the first step of this powerful journey with Exadata, and I'd like to start there in today's blog post.   

The systems that benefit from Exadata's extreme performance and reliability are typically business-critical systems that carry major risks when it comes to migration. Any downtime or data loss can have significant impact to the business in terms of revenue generation, customer loyalty, and productivity. As Oracle GoldenGate user community knows well, GoldenGate's heteregenous, real-time, and bidirectional replication capabilities enable very strong zero downtime migration and consolidation solutions for major databases and platforms including Oracle, IBM DB2 (zOS, iSeries, and LUW), HP NonStop, SQL Server, Sybase ASE, MySQL, and Teradata.

We discussed GoldenGate's zero downtime migration to Exadata offering and best practices with our customer IQNavigator in a webcast that is now available on demand:

Zero-Downtime Migration to Oracle Exadata Using Oracle GoldenGate: A Customer Case Study

If you have not watched it, I highly recommend listening to the discussion, as it clearly explains there should be no concerns around causing business interruption when moving to Oracle Exadata using GoldenGate.  GoldenGate's failback option to the old environment is a great tool for minimizing risk and many organizations adopt that approach for their business-critical systems.  

In addition to migration to Oracle Exadata, customers use GoldenGate, and Oracle Data Integrator, with Exadata in a variety ways leveraging the natural fit between these technologies:

  • Active-active database synchronization across the globe for data distribution, continuous availability, and zero downtime maintenance purposes.
  • Real-time or near real-time data loading to data warehouse, or consolidated database, on Oracle Exadata from heterogeneous sources. Oracle Data Integrator plays the major role in this use case as it integrates with GoldenGate and loads data warehouse in near real-time after performing transformations within the Exadata machine. This use case will be another blog topic soon as it is a strong best practice for performing ETL/ E-LT for Exadata.
  • Moving change data from an OLTP application running on Exadata in real time, for downstream consumption by other systems including supporting service integration.

As additional resources on best practices for migrating to Exadata I'd like to point you to couple of great white papers: Zero-Downtime Migration to Oracle Exadata Using Oracle GoldenGate and Oracle GoldenGate on Exadata Database Machine.

Thursday Jun 13, 2013

Near Zero Downtime Upgrades for Oracle Communications Billing and Revenue Management

Billing application is one of the key systems for a communications company. It is critical for not only revenue generation but also for customer experience.  Oracle Communications Billing and Revenue Management (BRM) solutions offers a flexible and reliable solution that helps service providers maximize customer value and profitability, and enhance business agility. Oracle BRM supports new business models including cloud, machine to machine (M2M), and mobile virtual network operators (MVNO).  Many of the world's leading, and innovative service providers—including France Telecom, Vodafone, Swisscom, Savvis, Sirius XM Satellite Radio and Nintendo, rely on the BRM application for their critical billing operations.

When it comes to upgrading a major system, such as BRM, things may get a bit tricky. Until recently there were only two approaches to upgrading BRM systems: In-place upgrades and the parallel system run. The in-place upgrade approach means the current BRM system would be shut down and brought back up after it is upgraded to the latest release of BRM. In addition to downtime, it puts heavy risk on the business. The parallel system run approach is an attempt in having two BRM systems running live in parallel. Unfortunately, this approach could only keep a portion of the data in-sync between the two systems as not all data feeds could be applied to both databases. Both of these approaches to upgrade resulted in downtime, which is not desirable for a system that drives revenue and impacts customer experience directly. 

Oracle GoldenGate offers a third approach to upgrade BRM systems and fortunately this new solution delivers near zero downtime and minimized risk.  Using GoldenGate's real-time data replication capability IT teams can create a full copy of the new BRM system on the latest software release without impacting the current BRM system. End users can use the old version, while a parallel system can be upgraded to the newest version of the release, kept in synch in real time with the active database, and tested thoroughly without time pressure before end users switch over to the upgraded BRM instance. GoldenGate's failback option also mitigates the risk to the business during the upgrade process. Oracle GoldenGate’s unique software platform does not require any changes to the source BRM system and uses minimal system resources. The solution supports BRM clients running BRM 7.2 or higher.

This is a very powerful solution for BRM users and we decided to explain the solution in depth via a screencast to help BRM users understand the new method better. If you have a mission-critical BRM system I highly recommend watching the screencast below to learn the detailed implementation steps.

You can watch the screencast also on Oracle Media Network, or Oracle's YouTube account. We have a white paper Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management (BRM) Application that you can download for more information as well.

Monday Apr 29, 2013

Monitoring Oracle GoldenGate 11g Release 2

One of the key improvements in Oracle GoldenGate 11g Release 2 is in the monitoring area. This release brought  integration with Oracle Enterprise Manager 12c, as well as a new product dedicated for monitoring Oracle GoldenGate, called Oracle GoldenGate Monitor.  This new product is included in the Management Pack for Oracle GoldenGate and offers GUI-based solution for end-to-end, real-time visibility into Oracle GoldenGate implementations across the enterprise. GoldenGate Monitor increases IT productivity by offering consolidated information for multiple instances of Oracle GoldenGate and via its comprehensive e-mail, SNMP and CLI alert capabilities, among many other features.

We have a new screencast that presents Oracle GoldenGate  Monitor's features and architecture, along with a product demonstration. If you have a GoldenGate solution or plan to deploy one, I highly recommend watching this screencast here or via the embedded video below to learn about monitoring GoldenGate processes via this simple but powerful GUI-based product.


For more information on Oracle GoldenGate 11g Release 2, please check out our free resources.

Sunday Mar 03, 2013

Oracle GoldenGate's Source of Truth for Certification Matrix

This post contains information about Oracle GoldenGate's certification matrix.
[Read More]

Friday Feb 22, 2013

Türk Telekom Saves Big and Improves Performance with Real-Time Archiving Solution

Written by:

Serkan KARATAS, Database Division Manager, Türk Telekom

Sandrine Riley, Oracle Data Integration Product Management


Türk Telekom--a leading communication and convergence technology company in Turkey and the country’s most valuable brand for the last four years according to Brand Finance--provides fixed line communications and broadband internet services. The company, which was founded in 1994 by separating from Posta ve Telgraf Teşkilatı, the national post and telegraph directorate of Turkey, owns more than 14.3 million fixed access lines—nearly all fixed access lines in Turkey—and 7 million asymmetric, digital subscriber lines.

Challenges and Goals

Türk Telekom’s challenge came with growth in becoming a leader in their industry. Customer relationship, sales, marketing, and finance departments needed accurate, timely, and well reported data from the organization's core CRM, billing and payment systems. Unfortunately, the overall data latency, movement, and accuracy were not optimal. The reports needed by these departments were critical for business practices around debt tracking and collecting, detecting of revenue leakages, overall churn prevention, as well as revenue assurance controls. In addition, these reports needed to incorporate online real-time data as well as historical data. In response to the challenges it faced, Türk Telekom set out to build a solution where reports would be delivered on time with maximum performance, leveraging all the appropriate data for better data analysis relevant to the critical business processes. Initially, the team tried to serve all the reports from core OLTP systems, quickly realizing they suffered from performance issues. They needed to reduce the amount of data on the OLTP system, and have the ability to report leveraging the historical data, all at the same time. Additionally, unplanned downtime with the systems was becoming an issue, and the need for data archiving became apparent.

Solution and Results

Türk Telekom was familiar with GoldenGate, as they had used it for their real-time data warehouse, as well as for some zero downtime migrations. Pleased with their prior results, they embarked on a new project in using Oracle GoldenGate for online archiving. This may very well be the first implementation of its kind in the market. Oracle GoldenGate is used to replicate the data to the archive database with near real-time latency, with the ability to purge historical data from the core OLTP database without replicating the purge activity to the archive database. Unlike other archiving solutions, the archive database contains not only historical data but also hot data, and, the archiving processes run while the systems are online and in real-time – not with offline scheduled batches.

Development and testing took about six months, and once this solution was implemented, it showed big results, very quickly. Within 1 month’s time of the solution’s go-live, full return on investment was achieved. Türk Telekom has saved approximately 55 million US dollars in revenue during just the first year ! The company has decreased overall churn rate due to debt by 60%, and has drastically increased OLTP performance by 10X. The business now has immediate access to sales, marketing and financial reports – providing the appropriate departments data to better handle debt tracking, and revenue leakages for example. As initially promised, the reports generated combine historical and real-time data as desired for the business.

Türk Telekom also saw many technical improvements based on this solution as well. 65% percent of disk space was saved once the historical data was archived. Storage and back-up costs shrunk by 40%, and maintenance costs of the production database decreased by 50%. Overall, the systems are more flexible and perform better with the use of Oracle GoldenGate.

But that isn’t all. Türk Telekom is not finished using Oracle GoldenGate. Future projects include: archiving large log tables and transaction tables on the OLTP database for Siebel and Oracle SOA Suite applications in use, directing all application reports to the archive database and totally removing the workload on the OLTP database, as well as wanting to use the archive database as the source for GoldenGate replication to configure cascading replication from the archive database. While lots of work is ahead for the team at Türk Telekom – their success and desire for more of it is very motivating!

For more details, please see Turk Telecom’s Oracle OpenWorld 2012 presentation:

http://www.myexpospace.com/oracle2012/SessionFiles/CON8876_PDF_8876_0001.pdf

its customer snapshot:

http://www.oracle.com/us/corporate/customers/customersearch/turk-telekom-5-gg-ss-1872904.html

and podcast:

http://feedproxy.google.com/~r/OracleOfmRadio/~3/RIpUbVF1HhI/12755074_Turk_Telekom_101212.mp3

Thursday Jan 10, 2013

Maximum Availability with Oracle GoldenGate Webcast: On Demand Link & FAQ

Last month we had a live webcast on a hot topic: Maximum Availability with Oracle GoldenGate. The webcast presentation focused on how Oracle GoldenGate can help achieve maximum availability for Oracle Database and other major databases. This webcast was very useful in answering a frequently asked question around GoldenGate's role for Oracle Database Maximum Availability Architecture (MAA) and when to use Oracle GoldenGate vs Active Data Guard. Product experts, Joe Meeks from Oracle Database High Availability product management team and Nick Wagner from Oracle GoldenGate product management team, addressed this question for us. Joe and Nick presented GoldenGate’s key solutions within MAA, and when to use which product along with Integrated Capture and Automated Conflict Management features of GoldenGate 11gR2. If you missed this event I highly recommend watching it on demand via the following link.

Watch On Demand:Maximum Availability with Oracle GoldenGate.

We also recorded a short podcast on this topic. Oracle GoldenGate 11gR2 for Maximum Availability

In the live webcast we had a very interactive audience who submitted over 100 questions during the event. Joe and Nick answered the most commonly asked questions about GoldenGate’s use cases and technology. Here are a few of the questions that we did not have time to answer during the event, along with the answers from the PM team.

  • Can I use the target database as warehouse for select only. If I were to allow users to this target, are there steps to ensure both source and target databases remain in sync?

 Yes you can use the target for read only or select only operations. Yes, we can do bi-directional active-active replication if you want to allow users to make changes to the target system. We can replicate those back to the source.

  • Can Integrated Capture be run on Oracle Database Standard Edition?

Yes, but all of the features that are only supported by Integrated Capture are only available with Oracle Database Enterprise Edition.

  •  Is it possible to capture database transactions from a source database using GoldenGate and publish them as messages into   a JMS queue?

Yes, GoldenGate can capture from and deliver to JMS queue. Please see our page for Java Adapter for more info. 

  • Can GoldenGate be used in environments where a number of tables do not have primary keys?

Yes. In tables without primary keys, we will use all non-LOB columns in the where clause for updates and deletes and then use ROWNUM = 1.

  • Is there a webinar that shows how to set up Oracle GoldenGate?

 Oracle GoldenGate Youtube channels has many useful videos and labs including “Oracle GoldenGate Fundamentals and How Oracle GoldenGate Works” and “Oracle GoldenGate Deep Dive Hands-on Labs” in 3 part series. You can browse and watch them here. Please check out GoldenGate training sessions with Oracle University as well.

I highly recommend subscribing to GoldenGate’s Youtube channel along with joining the GoldenGate Facebook and Twitter communities to get regular updates on the product.

Thursday Dec 20, 2012

ODI 11g - Loading More Files Faster

This post shows you how to leverage more of your Oracle system, you can load more data and faster using these kind of best practices. The LKM here, lets you leverage the Oracle database's preprocessor to load MORE data faster. The KM also has the capability to leverage a regular expression to identify the file or files so that you can load much much more, it also sorts the files based on size to comply with the most efficient way of loading into Oracle via the external table. The preprocessor can basically do anything, but is commonly used to uncompress data on the fly - so you can eliminate transferring large uncompressed files around or needlessly uncompressing them! There are various creative blog posts around showing you what some customers have done - some even include the transfer in the preprocessing phase.

I posted the LKM on the java.net site here

You can see the LKM configured below, I have used the preprocessor and a regular expression to load all gzipped files in the directory; 

When the code is generated below you can see the PREPROCESSOR directive used, plus you can see that the file names are going to be generated from a Java API call where the regular expression and directory are passed.

My gunzipdb.bat script has the following definition, wrappering the gzip utility (note the use of the -d -c directives);

  • @echo off
  • c:\ext_tab_test\gzip -d -c %1

Check this creative use of the PREPROCESSOR directive here (External table to load compressed data residing on remote locations) which does some creative work in the preprocessing script. This could do pretty much anything you want including transforming from some complex file to project through an external table.

About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« July 2014
SunMonTueWedThuFriSat
  
4
5
6
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today