Streaming relational transactions to HBase

Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems', and blog posts on "Streaming Relational Transactions to HDFS and Hive", in this blog post I focus on the architecture for streaming transactions into HBase.

As shown in the diagram below, integrating database with HBase is accomplished by developing a custom handler using Oracle GoldenGate's Java API and HBase APIs.

 The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process.   The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file. The Pump process executes adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions to HBase.

A sample implementation of the HBase adapter is provided on My Oracle Support (Knowledge ID - 1586211.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a corresponding HBase table. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

As always, I would greatly appreciate if you can share your use case about integrating Oracle GoldenGate with your Big Data strategy and your feedback on using the custom handler for integrating relational database with your Big Data systems. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

Comments:

Thanks for the sharing of this article on Streaming Relational Transactions to HDFS and Hive

Posted by DCL on October 14, 2013 at 11:19 PM PDT #

Hi Ananth

Thanks for sharing across such a valuable information even I am looking for the same solution.
It would be a great help if you could provide me with HBASE adapeter link or code or any information around it.

Thanks
Haider

Posted by guest on April 15, 2014 at 07:53 AM PDT #

Hey Anath,

Nice post. Did you try this?

Posted by guest on July 10, 2014 at 08:30 AM PDT #

Yes, I Did. Ananth

Posted by Ananth Tiru on July 10, 2014 at 11:47 PM PDT #

Hey Ananth,

I have the following use case, please give your inputs if hbase adapter would be the right choice or not.

1) Getting the data(initial load/ Changed data) using the Golden gate CDC tool from RDBMS to the HBase.
2) For now, we are using the JMS/Flat file application adapters as the target and from there some ETL tool to HBase, which is giving us Disk I/O problems. The application adapters is hosted on some linux file system server sitting between RDBMS and Hbase basically acting as a mid tier zone.
3) We are trying to integrate/ combine the application adapters with HBase API which could read the trail files directly and load the data into HBase. We are facing lot of problems in developing this code.

Please provide your inputs/ solutions.

Thanks

Posted by guest on July 11, 2014 at 10:50 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« July 2015
SunMonTueWedThuFriSat
   
3
4
5
6
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
       
Today