Sunday Apr 17, 2016

Oracle GoldenGate Adapter for ElasticSearch

Oracle GoldenGate can now integrate with ElasticSearch using the GoldenGate Adapter for ElasticSearch. GoldenGate Adapter for ElasticSearch can run on the latest Java based Replicat available with Oracle GoldenGate for Big Data 12.2. It uses high performance native Java driver to ElasticSearch. It can handle automatic DDL replication and supports inserts, updates and deletes into ElasticSearch. 

This project has converted to an open-source project for increased contribution from the Java community. The downloadable bundle contains source code, binary (jar) files, sample configuration files and instructions to use (readme.txt). You can download the bundle from URL https://java.net/projects/oracledi/downloads/directory/GoldenGate/Oracle%20GoldenGate%20Adapter%20for%20ElasticSearch

 

Compatibility Matrix:

  • GoldenGate for Big Data: 12.2.0.1.0 (All platforms)
  • ElasticSearch Server : 2.0.0 and above
  • ElasticSearch Java Driver: 2.2.1

Friday Feb 19, 2016

Oracle GoldenGate Adapter for MongoDB

Oracle GoldenGate can now integrate with Mongo DB using the GoldenGate Adapter for MongoDB. GoldenGate Adapter for MongoDB can run on the latest Java based Replicat available with Oracle GoldenGate for Big Data 12.2. It uses high performance native Java driver to MongoDB. It can handle automatic DDL replication and supports inserts, updates and deletes into MongoDB.

This project has converted to an open-source project for increased contribution from the Java community. The downloadable bundle contains source code, binary (jar) files, sample configuration files and instructions to use (readme.txt). You can download the bundle from URL https://java.net/projects/oracledi/downloads/directory/GoldenGate/MongoDB%20Adapter

Compatibility Matrix:

  • GoldenGate for Big Data: 12.2.0.1.0 (All platforms)
  • MongoDB Server : 2.4, 2.6, 3.0 3.2
  • MongoDB Java Driver: 3.2.2


Wednesday Oct 07, 2015

More on Leveraging Oracle Data Integrator (ODI) for Cloud Applications

Take a look at this week’s A-Team Blog post: Need to Integrate your Cloud Applications with On-Premise Systems… What about ODI? This blog is tightly coupled to the recent blog post: A Universal Cloud Applications Adapter for ODI.

First you learned about the simplicity of leveraging Oracle Data Integrator (ODI) with all emerging technologies in the world of cloud computing. Now read about which Cloud JDBC drivers will allow you to expand your data integration initiatives to include PaaS and SaaS connectivity.

For more A-Team reads on ODI, browse through the A-Team Chronicles.

Tuesday Aug 04, 2015

Simplicity in Leveraging Oracle Data Integrator for Cloud Applications

Check out last week’s A-Team Blog post… A Universal Cloud Applications Adapter for ODI

Learn about the simplicity of leveraging Oracle Data Integrator (ODI) with all emerging technologies in the world of cloud computing!

For more A-Team reads on ODI, browse through the A-Team Chronicles.

Wednesday Oct 01, 2014

Streaming Relational Transactions to Flume using Oracle GoldenGate

In prior articles, we have introduced architectures for streaming transactions from Oracle GoldenGate to HDFS, Hive, and HBase. In this article we are adding to this list by presenting a solution for streaming transactions into the Flume service. 

Apache Flume is a distributed, fault tolerant, and available service for efficiently collecting, aggregating, and moving large amounts of streaming data into HDFS. It can be configured into a variety of distribution mechanisms, such as delivery to multiple clusters, or rolling of HDFS files based on time or size. 

As shown in the diagram below, streaming database transactions to Flume is accomplished by developing a custom handler using Oracle GoldenGate's Java API and Flume's Avro RPC APIs.

The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process.   The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file. The Pump process executes the adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. The adapter then writes the transactions to a Flume Avro RPC source at the given host/port defined in the configuration file. The Flume Agent streams the data to the final destination; in the supplied example Flume writes into an HDFS directory for subsequent consumption by Hive. 

A sample implementation of the Flume Adapter for Oracle GoldenGate is provided at the My Oracle Support site as Knowledge Base article 1926867.1

Monday Oct 14, 2013

Streaming relational transactions to HBase

Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems', and blog posts on "Streaming Relational Transactions to HDFS and Hive", in this blog post I focus on the architecture for streaming transactions into HBase.

As shown in the diagram below, integrating database with HBase is accomplished by developing a custom handler using Oracle GoldenGate's Java API and HBase APIs.

 The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process.   The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file. The Pump process executes adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions to HBase.

A sample implementation of the HBase adapter is provided on My Oracle Support (Knowledge ID - 1586211.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a corresponding HBase table. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

As always, I would greatly appreciate if you can share your use case about integrating Oracle GoldenGate with your Big Data strategy and your feedback on using the custom handler for integrating relational database with your Big Data systems. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

Wednesday Oct 09, 2013

Streaming relational transactions to Hive

Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems', and blog post on 'Streaming relational transactions to HDFS', in this blog post I will discuss the architecture for streaming relational transactions into Hive.

Referring to the architecture diagram below, integrating database with Hive is accomplished by developing a custom handler using Oracle GoldenGate's Java API and Hadoop HDFS APIs.

The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process. The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file.The Pump process executes adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions in a desired format, with the appropriate content to a file which is defined by the Hive DDL for the table.

A sample implementation of the Hive adapter is provided on My Oracle Support (Knowledge ID - 1586188.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a corresponding Hive table. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

It would be great if you could share your use case about leveraging Oracle GoldenGate in your Big Data strategy and your feedback on using the custom handler for integrating relational database with Hive. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

Thursday Oct 03, 2013

Streaming relational transactions to Hadoop- HDFS

Following the introductory blog post on the topic – ' Stream your transactions into Big Data Systems'  in this blog post I will drill down on the architecture for streaming  relational transactions into HDFS. 

As you can see in the architecture diagram below, you can integrate relational database with HDFS by developing a custom handler using Oracle GoldenGate's Java API and Hadoop HDFS APIs.

 The custom handler is deployed as an integral part of the Oracle GoldenGate Pump process.   The Pump process and the custom adapter are configured through the Pump parameter file and custom adapter's properties file.

The Pump process executes the adapter in its address space. The Pump reads the Trail File created by the Oracle GoldenGate Capture process and passes the transactions to the adapter. Based on the configuration, the adapter writes the transactions in the desired format, with the appropriate content to a desired file on HDFS.

A sample implementation of the HDFS adapter is provided on My Oracle Support (Knowledge ID - 1586210.1). This is provided to illustrate the capability and to assist in the adoption of the Oracle GoldenGate Java API in developing custom solutions. The sample implementation illustrates the configuration and the code required for replicating database transactions on an example table to a file on HDFS. The instructions for configuring Oracle GoldenGate, compiling and running the sample implementation are also provided.

The sample code and configuration may be extended to develop custom solutions, however, please note that Oracle will not provide support for the code and the configuration illustrated in the knowledge base paper.

Please share your use case about how you use Oracle GoldenGate in your Big Data strategy and your feedback on using the custom handler for integrating relational database with your Big Data systems. Please post your comments in this blog or in the Oracle GoldenGate public forum - https://forums.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/goldengate

About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« May 2016
SunMonTueWedThuFriSat
1
2
4
5
6
7
8
9
10
11
13
14
15
17
18
19
21
22
23
24
25
26
27
28
29
30
31
    
       
Today