Harnessing the supremacy of Oracle GoldenGate for seamless data integration, Oracle GoldenGate 23ai introduces Data Streams, a new feature that offers application developers and data scientists an approach to access real-time transactional data captured by Oracle GoldenGate. This technology streamlines data ingestion by eliminating the need for intermediary systems and facilitates seamless integration with existing workflows.
Oracle GoldenGate Data Streams utilizes the AsyncAPI specification for defining asynchronous APIs. This approach enables applications to efficiently subscribe to data streams using a Publish or Subscribe model. Updates are received as soon as changes are committed in the source database, minimizing latency and simplifying application development. Additionally, Oracle GoldenGate Data Streams enable users to specify their preferred data format, such as JSON, for seamless integration with existing tools and frameworks within their development environment.
Add Prerequisites.
- GoldenGate Data Streams is available with Oracle GoldenGate 23ai. Oracle GoldenGate 23ai can be either provisioned as managed deployment within OCI GoldenGate or installed on-premises on a Linux host.
- Make sure the Extract process is running and capturing data into trail files. If not, please follow steps outlined here.
- Node.js shines for consuming data streams thanks to its asynchronous and event-driven nature. The Node.js application subscribes to the GoldenGate Data Stream’s Async API, allowing it to receive real-time updates as data changes occur. This empowers to process these updates efficiently and react swiftly to evolving information within your application. For this blog, we are going to use NodeJS application that subscribes to GoldenGate Data Streams .
Architecture :
![data-stream-arch-trail-files](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTF54922D88BBD4CD097C1BB35F352CC08/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
Follow the steps to create a Data Stream:
A. Creating a Data Streams on OCI console.
- Login to OCI GoldenGate with username and password.
![ogg home page](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTA55E22FDAD76428E86D7F1CD8C1C46FD/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Data streams are created from the Distribution Service. Switch to Distribution Service to begin creating a Data Stream process.
![data stream creation pages 2](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT2BC4C6B7361644D89CD65324A922635F/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- On the Data Stream Information page, enter a Data Stream process name in the Name box and add a description for it. Click Next.
![data stream creation pages 3](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT0FF651B561A54ABE906E675FF05614A3/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
.
- On the Source Options page, provide the values for options shown below:
- Enter the Trail Name:
![data stream creation pages trail name](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTC29A94A7D03345608E859F4E74806346/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Choose the “Quality of Service” and click “Next”.
![data stream creation pages 5](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTAB6B093916CD4EAAA362F0BE5CFE73AC/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- The Filtering Options page, the options for include and exclude filtering rules are available, as shown in the following image:
![data stream creation pages 6](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT36733B3DEF284821A9CCB6854277FD83/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Click Create Data Stream. You will be returned to the Distribution Service home page where the Data Stream is listed.
![data stream creation pages 6](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTE8C792872CA84CE3A8CA28728A4E5F37/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Data Stream “ociggdatastream” created successfully.
![data stream created successfully](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTF57D103CD0FE427AA6C619D1A0162B7E/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
B. Consume the Data Streams by generating Node.js client-side code with @asyncapi/generator.
@asyncapi/generator, a code generator, requires Node.js and npm packages of the versions listed below.
- Node.js v18.12.0+
- npm v8.19.0+
- Clone the websocket-client-template from below the github.
Copied to Clipboard
Error: Could not Copy
Copied to Clipboard
Error: Could not Copy
git clone https://github.com/oracle-samples/websocket-client-template.git
![Github Download page](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT1CD88F23FEDE4CDDB64C0F4F3C724F75/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Install dependencies and the AsyncAPI Generator via npm.
npm install
![npm install output](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT03BB5EE1666F46C5863E8D9ABABB720F/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
npm install -g @asyncapi/generator
![async downloader](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT254F39CB89DB4A53BFDFBB07ACD8B3B1/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Create a YAML file by copying the YAML content from the Data Streams page to generate the Node.js client code.
![yaml content copy](https://blogs.oracle.com/content/published/api/v1.1/assets/CONTE8ADB26DD6704C388C9618B5F7EF33EB/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
- Run the generation by passing arguments (Server Name and Authorization).
ag test/<yaml-file-name>.yaml . -o output -p server=<Deployment Console URL> -p authorization=basic
Argument :
-o : output directory Ex:output
-p server : Server Name Ex: XXXXX.deployment.goldengate.YYYY.oci.oraclecloud.com
-p authorization : Authorization type. Ex : basic/digest/certificate
Example :
![ag command](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT86BE33B28FD54323BFCABD90361FFB98/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
5. Go to generated output folder, and install needed packages for client
Copied to Clipboard
Error: Could not Copy
Copied to Clipboard
Error: Could not Copy
cd output
npm install
Note : when using Reverse Proxy, Kindly update below parameters in client.js code.
serviceURL = 'ws:// XXXXX.deployment.goldengate.YYYY.oci.oraclecloud.com /services /v2/stream/<data stream name>;
to
serviceURL = 'wss:// XXXXX.deployment.goldengate.YYYY.oci.oraclecloud.com /services/distsrvr/v2/stream/<data stream name>;
6. Start the Node..js client.
node client.js
![node js command](https://blogs.oracle.com/content/published/api/v1.1/assets/CONT95171732BBF34AAEAB9C7B5273419EA8/Medium?cb=_cache_0905&format=jpg&channelToken=8b43304987cf4ca092ab7a83133d013d)
To learn more about Oracle GoldenGate Data Streams, Please refer the listed links :
Oracle GoldenGate Data Streams
Async API
Madhu Kumar S
Senior Solution Engineer
As a GoldenGate specialist, I have spent years mastering the intricacies of Oracle GoldenGate and honing my expertise in real-time data integration and replication. I possess an in-depth understanding of the architecture, components, and capabilities of GoldenGate, allowing me to design and implement robust and efficient data integration solutions.
Show more