Thursday Mar 27, 2014

Interested in presenting and sharing your insights around Data Integration at OpenWorld?

You have been successful in making your organization run smoother. Faster. More cost-effectively. You have come up with the perfect solution to increase your staff retention, speed up your lead to sales pipeline, or minimize your supply management costs. We want to hear your story. Submit your proposal today and share your success at OpenWorld 2014.

Send us a proposal covering your Data Integration success. If it is selected, you will share your idea, experiences, and stories with Oracle customers, developers, and partners from around the world. You will also get a complimentary full pass to the conference.

Learn more about submitting your proposal.

Conference attendees want to hear it straight from you.

Don't wait—proposals must be submitted by April 15!

Tuesday Mar 25, 2014

Useful Tips on Oracle GoldenGate 12c Installation, Instantiation, and Setup

Written by Nick Wagner, Director of Product Management, Oracle Database High Availability

This is the first of 3 blog posts that I’m writing that will cover some of the questions I get from customers doing a first time implementation of Oracle GoldenGate. I’m not trying to reinvent the wheel, so to speak, so I’m going to reference a number of different My Oracle Support notes that go into more depth about a particular subject or process.

This first post will be about getting the environment set up and installation of Oracle GoldenGate. Also, for simplicity, this will assume a like to like replication environment as well as one-way replication. Future blogs will cover bi-directional replication and environments where the source and target objects have different structures.

Installation

Installation should be done on direct attached storage unless you are installing Oracle GoldenGate in a cluster or for failover, in that case the use of a dedicated NAS or SAN device is fine. Oracle GoldenGate is a quickly evolving product and we have a standard release and patching process. For example, in Oracle GoldenGate 11.2 and 12.1, it’s typically a 3 week patching cycle. So, always use the most recent version of Oracle GoldenGate so you have the most recent stable release. If you are using Oracle GoldenGate 11.2.1, you can just download the most recent release from My Oracle Support. If you are using Oracle GoldenGate 12.1.2, then you should download the installation from OTN or edelivery.oracle.com, and then download and upgrade to the most recent version from My Oracle Support and use Opatch to patch it to the latest release.

Instantiation

There are numerous articles on Oracle GoldenGate instantiation, the two most common are mentioned below.  Essentially, if you can do an Oracle hot backup that’s the easiest way to establish the target database. If that’s not possible, look at Oracle export/import or data pump, and if that can’t be used, you can use GoldenGate's own instantiation. In a situation where you are going to be doing transformation in the Delivery process (Replicat), then GoldenGate's own instantation, while being the slowest, is probably the best choice, as any transformation done in the Delivery can be applied during the initial load as well. It saves you from having to load the data, then transform it.

For like to like, the first is Oracle GoldenGate Best Practices: Instantiation from an Oracle Source Database [Article ID 1276058.1] this document is very comprehensive and covers Oracle to Oracle as well as heterogeneous options. And the other document: Oracle GoldenGate: Initial Load Techniques and References (Doc ID 1311707.1) can be used as well and has a few examples and would be ideal for targets where transformation is going to be used.

Initial Setup

The initial setup of the production database can be quite involved. However, the most important aspect is ensuring that the source database has the correct supplemental logging enabled. In the case where all (or a majority) of the tables are being replicated, it’s easier to configure supplemental logging at the schema level instead of the table level. Using the GGSCI command ADD SCHEMATRANDATA ensures that as the application changes, and new tables are added or keys changed on a table the underlying supplementally logged columns still adhere to those required by Oracle GoldenGate.

If there are tables without primary keys I would recommend that you review this article Supplemental logging – How to Handle Tables Without Primary Keys or Unique Indexes With Oracle GoldenGate [Article ID 1271578.1] that covers the way Oracle GoldenGate handles these types of tables and why it’s important to handle them correctly. If tables without keys aren’t managed properly, you could get anything from poor performance all the way to data corruption. The default way, may not always be the best way of handling these objects. Especially if there is a unique column that just doesn’t have a unique constraint on it.

Another simple way to reduce maintenance on Oracle GoldenGate is wildcarding. For example, if you have  Capture processes (also called Extract) running as a pump and pulling data from a Trail File, and all the data in the Trail File should be sent to the target, then use TABLE *.*; and that will instruct Capture to send all the data that it reads. That way, if a table is added to the Capture reading from the redo logs, you won’t need to make any changes to the Capture running as a pump is necessary. This same concept can even be used in the Delivery, where ASSUMETARGETDEFS is valid. Just use MAP *.*, TARGET *.*; and any table added to the Capture reading from the transaction logs will be sent all the way down to the target.

And that brings the first blog post to a close. I’ll be happy to answer any questions…

Tuesday Feb 18, 2014

Recap of Oracle GoldenGate 12c Webcast with Q&A

Simply amazing! That’s how I would summarize last week’s webcast for Oracle GoldenGate 12c.  It was a very interactive event with hundreds of live attendees and hundreds of great questions. In the presentation part my colleagues, Doug Reid and Joe deBuzna, went over the new features of Oracle GoldenGate 12c. They explained Oracle GoldenGate 12c key new features including:

  • Integrated Delivery for Oracle Database,
  • Coordinated Delivery for non-Oracle databases,
  • Support for Oracle Database 12c multitenant architecture,
  • Enhanced high availability via integration with Oracle Data Guard Fast-Start Failover,
  • Expanded heterogeneity, i.e. support for new databases and operating systems,
  • Improved security,
  • Low-downtime database migration solutions for Oracle E-Business Suite,
  • Integration with Oracle Coherence.

We also had a nice long and live Q&A section. In the previous Oracle GoldenGate webcasts, we could not respond to all audience questions in a 10-15 minute timeframe at the end of the presentation. This time we kept the presentation part short and left more than 30 minutes for Q&A. To our surprise, we could not answer even half of the questions we received. 

If you missed this great webcast discussing the new features of Oracle GoldenGate 12c,  and more than 30 minutes of Q&A with GoldenGate Product Management, you can still watch it on demand via the link below.

On Demand Webcast: Introducing Oracle GoldenGate 12c: Extreme Performance Simplified

On this blog post I would like to provide brief answers from our PM team  for some of the questions that we were not able to answer during the live webcast.

1) Does Oracle GoldenGate replicate DDL statements or DML for Oracle Database?

    Oracle GoldenGate replicates DML and DDL operations for Oracle Database and Teradata.

2) Where do we get more info on how to setup integration with Data Guard Fast-Start Failover (FSFO)?

     Please see the following blog posts or documents on My Oracle Support:

Best Practice - Oracle GoldenGate and Oracle Data Guard - Switchover/Fail-over Operations for GoldenGate    [My Oracle Support Article ID   1322547.1] 

Best Practice - Oracle GoldenGate 11gr2 integrated extract and Oracle Data Guard - Switchover/Fail-over Operations  [My Oracle Support Article ID 1436913.1] 

3) Does GoldenGate support SQL Server 2012 extraction? In the past only apply was supported.

Yes, starting with the new 12c release GoldenGate captures from SQL Server 2012 in addition to delivery capabilities.

4) Which RDBMS does GoldenGate 12c support?

GoldenGate supports all major RDBMS. For a full list of supported platforms please see Oracle GoldenGate certification matrix.

5) Could you provide some more details please on Integrated Delivery for dynamic parallel threads at Target side?

Please check out our white papers on Oracle GoldenGate 12c resource kit for more details on the new features, and how Oracle GoldenGate 12c works with Oracle Database. 

6) What is the best way to sync partial data (based on some selection criterion) from a table between databases?

 Please refer to the article: How To Resync A Single Table With Minimum Impact To Other Tables' Replication? [Article ID 966211.1]

7) How can GoldenGate be better than database trigger to push data into custom tables?

Triggers can cause high CPU overhead, in some cases almost double compared to reading from redo or transaction logs. In addition, they are intrusive to the application and cause management overhead as application changes. Oracle GoldenGate's log-based change data capture is not only low-impact in terms of CPU utilization, but also non-intrusive to the application with low maintenance requirements.

8) Are there any customers in the manufacturing industry using GoldenGate and for which application?

We have many references in manufacturing. In fact, SolarWorld USA was our guest speaker in the executive video webcast last November. You can watch the interview here. RIM Blackberry uses Oracle GoldenGate for multi-master replication between its global manufacturing systems. Here is another manufacturing customer story from AkzoNobel.

9) Does GoldenGate 12c support compressed objects for replication? Also does it supports BLOB/CLOB columns?

Yes, GoldenGate 12c and GoldenGate 11gR2 both support compressed objects. GoldenGate has been supporting BLOB/CLOB columns since version 10.

10) Is Oracle Database 11.2.0.4 mandatory to use GoldenGate 12c Integrated Delivery? Not earlier versions?

Yes. To use GoldenGate 12c’s Integrated Delivery, for the target environment Oracle Database 11.2.04 and above is required .

11) We have Oracle Streams implementation for more than 5 years. We would like to migrate to GoldenGate, however older version of GoldenGate were not supporting filtering individual transactions. Is it supported in GoldenGate 12c?

      Yes, it is supported in GoldenGate 12c.


In future blog posts I will continue to provide answers for common questions we received in the webcast. In the meanwhile I highly recommend watching the Introducing Oracle GoldenGate 12c: Extreme Performance Simplified webcast on demand.

Wednesday Feb 05, 2014

Introducing Oracle GoldenGate 12c: Extreme Performance Simplified

Oracle GoldenGate 12c was released last fall with a long list of new features that simplify configuration and increase flexibility, while delivering easy-to-use, advanced solutions with multi-fold performance gain.  We have been discussing these new features in various blog posts including:

· Advanced Replication for The Masses – Oracle GoldenGate 12c for the Oracle Database.

· GoldenGate 12c - What is Coordinated Delivery?

· GoldenGate 12c - Coordinated Delivery Example

· Oracle GoldenGate 12c - Announcing Support for Microsoft and IBM

And you can find new white papers about the 12c release in the Oracle GoldenGate resource kit.

Following the executive video webcast launching Oracle Data Integrator 12c and Oracle GoldenGate 12c in November 2013, we have set up another webcast for Oracle GoldenGate 12c where our product management team discusses the key new features in more depth and takes live questions from the audience.

If you would like to learn more about GoldenGate 12c I invite you to join us on Feb 12th in a webcast with product experts. You can register for this free event via the link below.

Live Webcast: Introducing Oracle GoldenGate 12c: Extreme Performance Simplified

February 12, 2014 -  10am PT/ 1pm ET

If you have missed our executive launch webcast I highly recommend that you watch it on demand via the link below. It talks about the tighter integration between Oracle Data Integrator 12c and Oracle GoldenGate and features customer and partner speakers from SolarWorld, BT, and Rittman Mead Consulting.

On-Demand Video Webcast: Introducing 12c for Oracle Data Integration

Wednesday Jan 22, 2014

Deep Dive Into Oracle Data Integrator Changed Data Capture Leveraging Oracle GoldenGate

Check out this blog post below from Christophe – first the details related to Oracle Data Integrator’s (ODI) Journalizing Knowledge Modules (JKMs) as well as a deeper dive into the particulars around the seamless out-of-the-box integration between Oracle Data Integrator (ODI) and Oracle GoldenGate.

http://www.ateam-oracle.com/understanding-the-odi-jkms-and-how-they-work-with-oracle-goldengate/

Happy reading!

Saturday Dec 28, 2013

GoldenGate 12c - Coordinated Delivery Example

This blog provides an example of setting up a coordinated replicat group using Oracle GoldenGate for MySQL.
[Read More]

Wednesday Dec 11, 2013

GoldenGate 12c - What is Coordinated Delivery?

In Oracle GoldenGate 12c, the parallel apply is improved with two features: Integrated Delivery and Coordinated Delivery. Different from the Integrated Delivery, Coordinated Delivery is not limited to Oracle Database.This blog discusses the new Coordinated Delivery feature by answering two questions:


  • Why do I need Coordinated Delivery?

  • What is Coordinated Delivery?
  • [Read More]

    Thursday Nov 21, 2013

    Recap of Oracle GoldenGate 12c and Oracle Data Integrator 12c Launch Webcast

    Last week we had a great video webcast for Oracle GoldenGate 12c and Oracle Data Integrator 12c. Our development executives, Brad Adelberg and Tim Hall,  talked about the new features and how the new release helps with delivering future-ready data integration solutions with extreme performance and high IT productivity. 

    If you missed the webcast you can watch it on demand via the following page:
    On-Demand Video Webcast: Introducing 12c for Oracle Data Integration

    In previous blogs we have talked about the new features of Oracle GoldenGate 12c and Oracle Data Integrator 12c, so I am not going to repeat them here. But worth repeating are what our customers and partners say about the new 12c release.

    SolarWorld’s Senior Database Administrator, Russ Toyama was in the studio with our executives for the launch webcast, and Russ discussed their GoldenGate implementation for SolarWorld's manufacturing process. SolarWorld is the largest U.S. solar panel manufacturer for more than 35 years and needed operational intelligence to continuously improve the quality of its products, while ensuring the systems operate with high performance and stability.

    SolarWorld uses GoldenGate to move data from multiple manufacturing databases into a single decision support system database in real time, freeing up the OLTP systems for transaction processing, which improves their performance and stability. The DSS database is very flexible in meeting reporting needs, and provides a comprehensive view of multiple manufacturing processes. This provides the traditional roles of reporting and engineering analysis to continuously improve product quality, yield and efficiency, and enables real time monitoring of the production manufacturing process. Using this real-time monitoring capability SolarWorld is able to detect the deviations from the norm right away, and take action to remedy or understand the situation. This not only improves production quality, but also improves cost management. The manufacturing process is a series of steps building upon the previous, so if there is an issue, it needs to be corrected as soon as possible to prevent waste and reduce manufacturing costs.

    For GoldenGate 12c we heard from Surren Partabh, CTO for Technology Services for BT. Surren explained the role of Oracle GoldenGate for their private cloud initiative and how they have improved customer experience and availability, while managing costs as well.  Surren highlighted that Oracle's data integration product family is one of the cornerstones for BT"s cloud migration project and enable to migrate to cloud simply, and in an agile manner. BT used GoldenGate to build a replication hub to help with the migration from legacy systems to the cloud with a reliable fallback strategy. Surren also commented on the tight integration between GoldenGate 12c and Oracle Data Integrator 12c,  saying that it is a "step in the right direction" as it simplifies the actual installation, configuration, management, and monitoring of solutions. You can watch the interview with BT's Surren Partabh here.

    In the launch webcast we also heard from Mark Rittman, CTO of Rittman Mead Consulting. Mark talked about the Oracle Data Integrator 12c new features in great depth, given that he was closely involved with the Beta testing program. Mark shared his opinion that the new flow-based design interface is the most critical feature of this 12c release and will bring major productivity gains for developers. He added that interoperability with Oracle Warehouse Builder for easier migration and tighter integration with Oracle GoldenGate are very valuable for customers. Mark also discussed the new release of Oracle BI Applications and how its use of Oracle Data Integrator for data movement and transformations simplifies life for Oracle BI Apps customers.  The complete interview with Mark Rittman is available for you as well.

    If you missed the launch webcast last week, I hope you take the time to watch it on demand and discover how Oracle has changed the data integration and replication technology space with Oracle Data Integrator 12c and Oracle GoldenGate 12c. For more information including white papers and podcasts you can also download free resources.

    Wednesday Oct 30, 2013

    Replicating between Cloud and On-Premises using Oracle GoldenGate

    Do you have applications running on the cloud that you need to connect with the on premises systems. The most likely answer to this question is an astounding YES!  If so, then you understand the importance of keep the data fresh at all times across the cloud and on-premises environments. This is also one of the key focus areas for the new GoldenGate 12c release which we announced couple of week ago via a press release.

    Most enterprises have spent years avoiding the data “silos” that inhibit productivity. For example, an enterprise which has adopted a CRM strategy could be relying on an on-premises based marketing application used for developing and nurturing leads. At the same time it could be using a SaaS based Sales application to create opportunities and quotes. The sales and the marketing teams which use these systems need to be able to access and share the data in a reliable and cohesive way. This example can be extended to other applications areas such as HR, Supply Chain, and Finance and the demands the users place on getting a consistent view of the data.

    When it comes to moving data in hybrid environments some of the key requirements include minimal latency, reliability and security:

    • Data must remain fresh. As data ages it becomes less relevant and less valuable—day-old data is often insufficient in today’s competitive landscape.
    • Reliability must be guaranteed despite system or connectivity issues that can occur between the cloud and on-premises instances.
    • Security is a key concern when replicating between cloud and on-premises instances. There are several options to consider when replicating between the cloud and on-premises instances.

    Option 1 – Secured network established between the cloud and on-premises

    A secured network is established between the cloud and on-premises which enables the applications (including replication software) running on the cloud and on-premises to have seamless connectivity to other applications irrespective of where they are physically located.

    Option 2 – Restricted network established between the cloud and on-premises

    A restricted network is established between the cloud and on-premises instances which enable certain ports (required by replication) be opened on both the cloud and on the on-premises instances and white lists the IP addresses of the cloud and on-premises instances.

    Option 3 – Restricted network access from on-premises and cloud through HTTP proxy

    This option can be considered when the ports required by the applications (including replication software) are not open and the cloud instance is not white listed on the on-premises instance. This option of tunneling through HTTP proxy may be only considered when proper security exceptions are obtained.

    Oracle GoldenGate

    Oracle GoldenGate is used for major Fortune 500 companies and other industry leaders worldwide to support mission-critical systems for data availability and integration. Oracle GoldenGate addresses the requirements for ensuring data consistency between cloud and on-premises instances, thus facilitating the business process to run effectively and reliably.

    The architecture diagram below illustrates the scenario where the cloud and the on-premises instance are connected using GoldenGate through a secured network

    In the above scenario, Oracle GoldenGate is installed and configured on both the cloud and the on-premises instances. On the cloud instance Oracle GoldenGate is installed and configured on the machine where the database instance can be accessed. Oracle GoldenGate can be configured for unidirectional or bi-directional replication between the cloud and on premises instances.

    The specific configuration details of Oracle GoldenGate processes will depend upon the option selected for establishing connectivity between the cloud and on-premises instances. The knowledge article (ID - 1588484.1) titled ' Replicating between Cloud and On-Premises using Oracle GoldenGate' discusses in detail the options for replicating between the cloud and on-premises instances. The article can be found on My Oracle Support.

    To learn more about Oracle GoldenGate 12c register for our launch webcast where we will go into these new features in more detail.   You may also want to download our white paper "Oracle GoldenGate 12c Release 1 New Features Overview"

    I would love to hear your requirements for replicating between on-premises and cloud instances, as well as your comments about the strategy discussed in the knowledge article to address your needs. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

    Thursday Oct 17, 2013

    Welcome Oracle Data Integration 12c: Simplified, Future-Ready Solutions with Extreme Performance

    The big day for the Oracle Data Integration team has finally arrived! It is my honor to introduce you to Oracle Data Integration 12c.

    Today we announced the general availability of 12c release for Oracle’s key data integration products: Oracle Data Integrator 12c and Oracle GoldenGate 12c. The new release delivers extreme performance, increase IT productivity, and simplify deployment, while helping IT organizations to keep pace with new data-oriented technology trends including cloud computing, big data analytics, real-time business intelligence. With the 12c release Oracle becomes the new leader in the data integration and replication technologies as no other vendor offers such a complete set of data integration capabilities for pervasive, continuous access to trusted data across Oracle platforms as well as third-party systems and applications.

    Oracle Data Integration 12c release addresses data-driven organizations’ critical and evolving data integration requirements under 3 key themes:

    • Future-Ready Solutions
    • Extreme Performance
    • Fast Time-to-Value

          There are many new features that support these key differentiators for Oracle Data Integrator 12c and for Oracle GoldenGate 12c. In this first 12c blog post, I will highlight only a few:·

    Future-Ready Solutions to Support Current and Emerging Initiatives: Oracle Data Integration offer robust and reliable solutions for key technology trends including cloud computing, big data analytics, real-time business intelligence and continuous data availability. Via the tight integration with Oracle’s database, middleware, and application offerings Oracle Data Integration will continue to support the new features and capabilities right away as these products evolve and provide advance features.

    E    Extreme Performance: Both GoldenGate and Data Integrator are known for their high performance. The new release widens the gap even further against competition. Oracle GoldenGate 12c’s Integrated Delivery feature enables higher throughput via a special application programming interface into Oracle Database. As mentioned in the press release, customers already report up to 5X higher performance compared to earlier versions of GoldenGate. Oracle Data Integrator 12c introduces parallelism that significantly increases its performance as well.

    Fast Time-to-Value via Higher IT Productivity and Simplified Solutions:  Oracle Data Integrator 12c’s new flow-based declarative UI brings superior developer productivity, ease of use, and ultimately fast time to market for end users.  It also gives the ability to seamlessly reuse mapping logic speeds development.Oracle GoldenGate 12c ‘s Integrated Delivery feature automatically optimally tunes the process, saving time while improving performance.

    This is just a quick glimpse into Oracle Data Integrator 12c and Oracle GoldenGate 12c. On November 12th we will reveal much more about the new release in our video webcast "Introducing 12c for Oracle Data Integration". Our customer and partner speakers, including SolarWorld, BT, Rittman Mead will join us in launching the new release. Please join us at this free event to learn more from our executives about the 12c release, hear our customers’ perspectives on the new features, and ask your questions to our experts in the live Q&A.

    Also, please continue to follow our blogs, tweets, and Facebook updates as we unveil more about the new features of the latest release.

    Monday Oct 14, 2013

    Streaming relational transactions to HBase

    Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems', and blog posts on "Streaming Relational Transactions to HDFS and Hive", in this blog post I focus on the architecture for streaming transactions into HBase.

    As shown in the diagram below, integrating database with HBase is accomplished by developing a custom handler using Oracle GoldenGate's Java API and HBase APIs.

     The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process.   The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file. The Pump process executes adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions to HBase.

    A sample implementation of the HBase adapter is provided on My Oracle Support (Knowledge ID - 1586211.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a corresponding HBase table. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

    The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

    As always, I would greatly appreciate if you can share your use case about integrating Oracle GoldenGate with your Big Data strategy and your feedback on using the custom handler for integrating relational database with your Big Data systems. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

    Wednesday Oct 09, 2013

    Streaming relational transactions to Hive

    Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems', and blog post on 'Streaming relational transactions to HDFS', in this blog post I will discuss the architecture for streaming relational transactions into Hive.

    Referring to the architecture diagram below, integrating database with Hive is accomplished by developing a custom handler using Oracle GoldenGate's Java API and Hadoop HDFS APIs.

    The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process. The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file.The Pump process executes adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions in a desired format, with the appropriate content to a file which is defined by the Hive DDL for the table.

    A sample implementation of the Hive adapter is provided on My Oracle Support (Knowledge ID - 1586188.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a corresponding Hive table. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

    The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

    It would be great if you could share your use case about leveraging Oracle GoldenGate in your Big Data strategy and your feedback on using the custom handler for integrating relational database with Hive. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

    Thursday Oct 03, 2013

    Streaming relational transactions to Hadoop- HDFS

    Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems'  in this blog post I will drill down on the architecture for streaming  relational transactions into HDFS. 

    As you can see in the architecture diagram below, you can integrate relational database with HDFS by developing a custom handler using Oracle GoldenGate's Java API and Hadoop HDFS APIs.

     The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process.   The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file.

    The Pump process executes the adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions in the desired format, with the appropriate content to a desired file on HDFS.

    A sample implementation of the HDFS adapter is provided on My Oracle Support (Knowledge ID - 1586210.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a file on HDFS. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

    The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

    Please share your use case about how you use Oracle GoldenGate in your Big Data strategy and your feedback on using the custom handler for integrating relational database with your Big Data systems. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

    Monday Sep 23, 2013

    At the End of the First Day of Data Integration Track at OpenWorld

    A long, but exciting Monday at Oracle OpenWorld 2013 ended for the Oracle Data Integration product management team. We had a full day with many key product strategy sessions.  Mark Hurd's keynote set the tone for the Data Integration sessions as well. Mark focused on big data analytics and the changing consumer expectations. Especially real-time insight is a key theme for Oracle overall and data integration products. In Mark Hurd's keynote we heard from key customers, such as Airbus and Thomson Reuters, how real-time analysis of operational data including machine data creates value, in some cases even saves lives. Thomas Kurian gave a deeper look into Oracle's big data and fast data solutions.

    Brad Adelberg, VP of Development, presented Oracle’s Data Integration product strategy based on these key trends that we heard in the keynotes. Brad talked about how Oracle's data integration products address the new data integration requirements that evolved with cloud computing, big data, and changing consumer expectations and how they set the key themes in our products’ road map. Brad explained why and how fast-time to value, high-performance and future-ready solutions are the top focus areas for product development. If you were not able to attend OpenWorld or this session I recommend reading the white paper : Five New Data Integration Requirements and How to Meet them with Oracle Data Integration, which provides an in-depth look into how Oracle addresses the new trends in the DI market.

    Following Brad’s session, Nick Wagner provided in depth review of Oracle GoldenGate’s latest features and roadmap. Nick discussed how Oracle GoldenGate’s tight integration with Oracle Database sets the product apart from the competition. We also heard that heterogeneity of the product is still a major focus for GoldenGate’s development and there will be more news on that front when there is a major release.

    After GoldenGate’s product strategy session, Denis Gray from the PM team presented Oracle Data Integrator’s product strategy session, talking about the latest and greatest on ODI. Another good session was delivered by long-time GoldenGate users, Comcast.  Jason Hurd and Amit Patel of Comcast talked about the various use cases they deploy Oracle GoldenGate throughout their enterprise,from database upgrades, feeding reporting systems,  to active-active database synchronization.  The Comcast team shared many good tips on how to use GoldenGate for both zero downtime upgrades and active-active replication with conflict management requirement.

    We ended the day with a wonderful award ceremony for Oracle Excellence Awards for Oracle Fusion Middleware Innovation. The ceremony was held in the Yerba Buena Center for the Arts. Congratulations to Royal Bank of Scotland and Yalumba Wine Company, the winners in the Data Integration category. You can find more information on the award and the winners in our previous blog post: 2013 Oracle Excellence Awards for Fusion Middleware Innovation…

    The demo pods for data integration products had a busy day as well. You can see from the photos below our experts stayed busy talking to attendees, yet they stopped a moment to smile to my camera.

    If you have any questions about sessions you’ve missed or general questions about our products you can find us in Moscone South on both sides of the expo hall within Oracle Database and Oracle Fusion Middleware demo pods. Also, don’t forget to add the following sessions to your schedule for Tuesday.

    A Panel of Masterminds: Perspectives from Oracle GoldenGate Customers (CON8898)

    Tuesday, September 24, 2013 at 3:45PM-4:45PM PT

    Moscone West – 2024

    Optimize Oracle Active Data Guard/Oracle GoldenGate: Oracle Database 12c Global Data Services (CON8704)

    Tuesday, September 24, 2013 at 5:15PM-6:15PM PT

    Moscone South – 200

    Combining Oracle Data Integrator and Oracle GoldenGate (HOL10144)

    Tuesday, September 24, 2013 at 3:45PM-4:45PM PT

    Marriott Marquis – Nob Hill AB

    For a full overview of the Data Integration track please check out Focus On Data Integration.

    2013 Oracle Excellence Awards for Fusion Middleware Innovation… and the Winners for Data Integration are…

    Every year at OpenWorld, Oracle announces the winners to its most prestigious awards in Middleware, the Oracle Excellence Awards for Fusion Middleware Innovation. This year, we celebrated in style at the Lam Research Theater. It felt like a night at the Oscars!