X

Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

Streaming relational transactions to HBase

Guest Author

Following the introductory blog post on the topic – ' Stream
your transactions into Big Data Systems', and blog posts on "Streaming Relational Transactions to HDFS and Hive", in this blog post I focus on the
architecture for streaming transactions into HBase.

As shown in the diagram below, integrating database with HBase is accomplished by
developing a custom handler using Oracle GoldenGate's Java API and HBase
APIs.

 The custom handler is deployed as an integral part of the
Oracle GoldenGate Pump process.   The
Pump process and the custom adapter are configured through the Pump parameter
file and custom adapter's properties file. The Pump process executes adapter in its address space. The Pump reads
the Trail File created by the Oracle GoldenGate Capture process and passes the
transactions to the adapter. Based on the configuration, the adapter writes the
transactions to HBase.

A sample implementation of the HBase adapter is provided on
My Oracle Support (Knowledge ID - 1586211.1). This is provided to illustrate the capability and to assist in the
adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration
and the code required for replicating database transactions on an example table
to a corresponding HBase table. The
instructions for configuring Oracle GoldenGate, compiling and running the
sample implementation are also provided.

The sample code and configuration may be extended to develop
custom solutions, however, please note that Oracle will not provide support for
the code and the configuration illustrated in the knowledge base paper.

As always, I would greatly appreciate if you can share your use case
about integrating Oracle GoldenGate with your Big Data strategy and your
feedback on using the custom handler for integrating relational database with
your Big Data systems. Please post your comments
in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

Join the discussion

Comments ( 5 )
  • DCL Tuesday, October 15, 2013

    Thanks for the sharing of this article on Streaming Relational Transactions to HDFS and Hive


  • guest Tuesday, April 15, 2014

    Hi Ananth

    Thanks for sharing across such a valuable information even I am looking for the same solution.

    It would be a great help if you could provide me with HBASE adapeter link or code or any information around it.

    Thanks

    Haider


  • guest Thursday, July 10, 2014

    Hey Anath,

    Nice post. Did you try this?


  • Ananth Tiru Friday, July 11, 2014

    Yes, I Did. Ananth


  • guest Friday, July 11, 2014

    Hey Ananth,

    I have the following use case, please give your inputs if hbase adapter would be the right choice or not.

    1) Getting the data(initial load/ Changed data) using the Golden gate CDC tool from RDBMS to the HBase.

    2) For now, we are using the JMS/Flat file application adapters as the target and from there some ETL tool to HBase, which is giving us Disk I/O problems. The application adapters is hosted on some linux file system server sitting between RDBMS and Hbase basically acting as a mid tier zone.

    3) We are trying to integrate/ combine the application adapters with HBase API which could read the trail files directly and load the data into HBase. We are facing lot of problems in developing this code.

    Please provide your inputs/ solutions.

    Thanks


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha