Unlocking the Power of GoldenGate 23ai: A Guide to Creating Data Streams on OCI GoldenGate

July 21, 2024 | 6 minute read
Madhu Kumar S
Senior Solution Engineer
Text Size 100%:

Harnessing the supremacy of Oracle GoldenGate for seamless data integration, Oracle GoldenGate 23ai introduces Data Streams, a new feature that offers application developers and data scientists an approach to access real-time transactional data captured by Oracle GoldenGate. This technology streamlines data ingestion by eliminating the need for intermediary systems and facilitates seamless integration with existing workflows.

Oracle GoldenGate Data Streams utilizes the AsyncAPI specification for defining asynchronous APIs. This approach enables applications to efficiently subscribe to data streams using a Publish or Subscribe model. Updates are received as soon as changes are committed in the source database, minimizing latency and simplifying application development. Additionally, Oracle GoldenGate Data Streams enable users to specify their preferred data format, such as JSON, for seamless integration with existing tools and frameworks within their development environment.
Add Prerequisites.

  • GoldenGate Data Streams is available with Oracle GoldenGate 23ai. Oracle GoldenGate 23ai can be either provisioned as managed deployment within OCI GoldenGate or installed on-premises on a Linux host.
  • Make sure the Extract process is running and capturing data into trail files. If not, please follow steps outlined here.
  • Node.js shines for consuming data streams thanks to its asynchronous and event-driven nature. The Node.js application subscribes to the GoldenGate Data Stream’s Async API, allowing it to receive real-time updates as data changes occur. This empowers  to process these updates efficiently and react swiftly to evolving information within your application. For this blog, we are going to use NodeJS application  that subscribes to GoldenGate Data Streams .

 

Architecture :

data-stream-arch-trail-files

Follow the steps to create a Data Stream:

A. Creating a Data Streams on OCI console.

  1. Login to OCI GoldenGate with username and password.

    ogg home page

     
  2. Data streams are created from the Distribution Service. Switch to Distribution Service to begin creating a Data Stream process.


    data stream creation pages 2

     
  3. On the Data Stream Information page, enter a Data Stream process name in the Name box and add a description for it. Click Next

    data stream creation pages 3
    .
  4. On the Source Options page, provide the values for options shown below:
     
    • Enter the Trail Name:

      data stream creation pages trail name
       
    • Choose the “Quality of Service” and click “Next”.

      data stream creation pages 5
       
  5. The Filtering Options page, the options for include and exclude filtering rules are available, as shown in the following image:

    data stream creation pages 6
     
  6. Click Create Data Stream. You will be returned to the Distribution Service home page where the Data Stream is listed.

    data stream creation pages 6
     
  7. Data Stream “ociggdatastream”  created successfully.

    data stream created successfully

B. Consume the Data Streams by generating Node.js client-side code with @asyncapi/generator.

  @asyncapi/generator, a code generator, requires Node.js and npm packages of the versions listed below.

  • Node.js v18.12.0+
  • npm v8.19.0+
  1. Clone the websocket-client-template from below the github.
Copied to Clipboard
Error: Could not Copy
Copied to Clipboard
Error: Could not Copy
git clone https://github.com/oracle-samples/websocket-client-template.git


Github Download page
 

  1. Install dependencies and the AsyncAPI Generator via npm.
npm install

 

npm install output
 

npm install -g @asyncapi/generator


async downloader
 

  1. Create  a YAML file by copying the YAML content from the Data Streams page to generate the Node.js client code.
     

yaml content copy

  1.  Run the generation by passing arguments (Server Name and Authorization).
ag test/<yaml-file-name>.yaml . -o output -p server=<Deployment Console URL>  -p authorization=basic

Argument : 

-o : output directory   Ex:output

-p server : Server Name Ex: XXXXX.deployment.goldengate.YYYY.oci.oraclecloud.com

-p authorization : Authorization type. Ex : basic/digest/certificate

Example :

ag command

 5. Go to generated output folder, and install needed packages for client

Copied to Clipboard
Error: Could not Copy
Copied to Clipboard
Error: Could not Copy
cd output

npm install

Note : when using Reverse Proxy, Kindly update below parameters in client.js code.

  • Update the Service URL :

serviceURL = 'ws:// XXXXX.deployment.goldengate.YYYY.oci.oraclecloud.com /services /v2/stream/<data stream name>;

to

serviceURL = 'wss:// XXXXX.deployment.goldengate.YYYY.oci.oraclecloud.com /services/distsrvr/v2/stream/<data stream name>;

      6. Start the Node..js client.

node client.js


node js command

To learn more about Oracle GoldenGate Data Streams, Please refer the listed links :

Oracle GoldenGate Data Streams

Async API

Madhu Kumar S

Senior Solution Engineer

As a GoldenGate specialist, I have spent years mastering the intricacies of Oracle GoldenGate and honing my expertise in real-time data integration and replication. I possess an in-depth understanding of the architecture, components, and capabilities of GoldenGate, allowing me to design and implement robust and efficient data integration solutions.

Show more

Previous Post

Oracle GoldenGate for Distributed Applications and Analytics 23ai Articles and new features

Anuradha Chepuri | 5 min read

Next Post


AsyncAPI Code Generator for GoldenGate Data Streams

Deniz Sendil | 6 min read