AsyncAPI Code Generator for GoldenGate Data Streams

July 22, 2024 | 6 minute read
Deniz Sendil
Senior Principal Product Manager
Text Size 100%:

In May 2024, we announced the availability of GoldenGate Data Streams with GoldenGate 23ai. GoldenGate Data Streams is a brand new publish-subscribe REST interface to your JSON formatted data events.

GoldenGate Data Streams leverages AsyncAPI and enables applications to efficiently subscribe to data streams using a Publish / Subscribe model. GoldenGate Data Streams is an AsyncAPI based asynchronous messaging platform which enables loosely coupled architectures where systems can communicate without being tightly integrated, without any 3rd part messaging system dependency. GoldenGate Data Streams is a language-agnostic solution, ensuring seamless integration with various client applications irrespective of the programming language used. To get more details about GoldenGate Data Streams, you can listen to an architectural overview of Data Streams and the AsyncAPI implementation.

Now, we are happy to announce the availability of GoldenGate Data Streams Code Generator. GoldenGate Data Streams uses a configuration file (which is the AsyncAPI definition for Data Streaming) for generating client codes where you can test GoldenGate Data Streams connections and apply your business logic. GoldenGate Data Streams Code Generator packages client libraries for various programming languages and provides efficiency to developers. GoldenGate Data Streams Code Generator also ensures that clients are consistent with GoldenGate Data Streams definitions. In summary, GoldenGate Data Streams Code Genretor increases developer productivity, code quality and provides consistency with GoldenGate Data Streams AsyncAPI definitions.

In this blog, I'll go through GoldenGate Data Streams Code Generator installation steps. For this blog, I'm using a GoldenGate 23ai instance running on a compute instance and I'll configure my code generator to run on a compute instance as well. I'll be using an existing trail file created by an extract process. I'll use python client from code generator for receiving the messages. 

I recommend the following installations before you run the GoldenGate Data Streams code generator:

  • Node.js v18.12.0 or higher -- needed to run code generator and node.js client
  • npm v8.19.0 or higher -- needed to run code generator
  • python 3.8 (and pip) or higher -- needed for python client
  • JDK 21 or higher (for java client) -- needed for java client
  • the_silver_searcher (ag) -- needed to run code generator

Let's start with creating a GoldenGate Data Stream.

Creating a GoldenGate Data Stream:

  1. In GoldenGate interface, navigate to Distribution Service and click on Data Streams. Click "Add Data Stream".
     

    GoldenGate Data Streams

     

  2. Provide a Name and click Next.

    GoldenGate Data Streams

     

     

  3. In Source Options, provide your source Trail Name, select Quality of Service and click Next

    GoldenGate Data Streams

     

     

  4. For this blog, we skip filters. You can click Create Data Stream.

After Data Stream is created, you can review the AsyncAPI configuration file which provides all the details about server, channel, schema and object definitions. In the next step, we will be using this definition file to run the code generator. 

 

GoldenGate Data Streams

Now, we are ready to run the GoldenGate Data Streams Code Generator. 

Running Code Generator:

Before running the code generator, please make sure that your environment covers the technical requirements. You can refer to websocket-client-template for technical details. At the date of this blog, node.js v18.12.0+ (node -v for linux) and  npm v8.19.0+ (npm -v for linux) is required. 

I also setup npm install global position to prevent possible permission issues.

mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
export PATH=~/.npm-global/bin:$PATH

Now, we can continue with installation. 

  1. Copy the gitbuh library and cd into websocket-client-template directory.

    git clone https://github.com/oracle-samples/websocket-client-template.git
    cd websocket-client-template
     

    GoldenGate Data Streams

     

     

  2. Create the yaml configuration file with your GoldenGate Data Stream definition file in websocket-client-template directory. You can just create a file and copy/ paste the content from your GoldenGate Data Stream configuration file.
     

    GoldenGate Data Streams


     
  3. When in websocket-client-template directory, install dependencies and the AsyncAPI Generator. Installing dependencies and the Async Generator should be completed without any errors. 

    npm install -g @asyncapi/generator
    npm install

    If you encounter any conflict error, you can use --force option (npm install --force)
     
  4. Run the code generator with the asyncapi definition document you created in step two. 

    ag test/streaming.yaml . -o output -p server=<GoldenGate_Host> -p language=all -p authorization=basic 

    Make sure running this in websocket-client-template folder. You may need to replace '.' with absolute path. You can check websocket-client-template for other configuration details.
     
  5. Once installation is successfully complete, you should see a message as below. You can go into output directory where the client connection codes are created. Currently, python and node.js clients are available. 
     

    GoldenGate Data Streams

     

     

  6. While running python client, you may need to upgrade your python version and install modules depending on your existing python configuration. 
     
  7. When client.py is executed, it will ask for username, password (username and password you're using for data streams. If needed, you can create a new user from User Administration) and beginning point (just hit enter for default setting as earliest). 
  8. If connection is successful, client will show the messages published to GoldenGate Data Streams. 
     

    GoldenGate Data Streams

     

As you can see, it is very easy to run GoldenGate Data Streams Code Generator for testing GoldenGate Data Streams connections. Once basic connection issuccessful, you can also implement your own logic in the code for further integrations with other applications. 

In this blog, I covered how to create GoldenGate Data Streams Async API Code Generator and executed the python client (using ws protocol) that fetched messages from my on-prem GoldenGate Data Streams channel. 

As noted above, node.js and python client are currently available. We're working on Java client and please stay tuned for the announcement. If you're interested in configuring GoldenGate Data Streams Async API Code Generator node.js with OCI GoldenGate, you can check GoldenGate 23ai : Creating Data Streams on OCI GoldenGate blogpost published by Madhu Kumar S. Also, you can check Connecting to OCI GoldenGate Data Streams API Endpoints for connecting to OCI GoldenGate Data Streams with a python client using secure web socket (wss) protocol. For more details, you can check the product documentation

Deniz Sendil

Senior Principal Product Manager

Deniz Sendil is a Senior Principle Product Manager in Oracle GoldenGate product group with a focus on GoldenGate for Big Data. Deniz has over 17 years of experience in data management, analytics and data integration domains. 


Previous Post

Unlocking the Power of GoldenGate 23ai: A Guide to Creating Data Streams on OCI GoldenGate

Madhu Kumar S | 6 min read

Next Post


Configure OCI Dataflow with ADW-PE in another tenancy in the same region

Mario Miola | 6 min read