Connecting to OCI GoldenGate Data Streams API Endpoints

May 2, 2024 | 5 minute read
Deniz Sendil
Senior Principal Product Manager
Text Size 100%:

Back in Oracle Cloud World 2023, GoldenGate Data Streams was announced as a new GoldenGate feature. It is a brand new publish-subscribe REST interface to your JSON formatted data events and now, GoldenGate Data Streams is available with OCI GoldenGate 23ai.

GoldenGate Data Streams leverages AsyncAPI and enables applications to efficiently subscribe to data streams using a Publish / Subscribe model. With the most recent OCI GoldenGate 23ai deployment versions, you can create Data Streams and subscribe to these streams from your clients. GoldenGate Data Streams is a language-agnostic solution, ensuring seamless integration with various client applications irrespective of the programming language used. To get more details about GoldenGate Data Streams, you can listen to an architectural overview of Data Streams and the AsyncAPI implementation.

GoldenGate Data Streams uses the trail files as source; so an extract process is necessary. GoldenGate Data Streams is available in all GoldenGate deployment types which enables users to publish messages to Data Streams from various sources like Oracle, MS SQL, MongoDB and many more. You can refer to GoldenGate certification matrix to see the list of all supported technologies.

With OCI GoldenGate Data Streams, direct access to the data in user specified format is enabled through a dedicated secured websocket (wss) channel that follows a secure and simple streaming protocol. You can use self-signed certificates or CA signed certificates between your clients and Data Stream for a secured websocket connection.

In this blog post, I'll share a step-by-step guidance to set up an OCI GoldenData Data Strem and configure a python client.

Before we begin, a few notes regarding this blog:

  • I'm using Oracle Database 23ai deployment type. You can check Julien's blog to see how to create an OCI GoldenGate 23ai deployment.
  • I'm using an existing extract process and I already created source operations. Extract is out of scope of this blog. You can refer to OCI GoldenGate Document to get more details about extracts.
  • I'm using an OCI compute instance to host my python client. This is a very simple client that subscribes to OCI GoldenGata Data Streams and writes the payloads into a log file.

Let's begin!

Creating self-signed certificates and adding a truststore certificate to OCI GoldenGate

To connect to an OCI GoldenGate Data Streams from a client, you need to create self signed certificates.

  1. In your client machine, create a new directory for certificates:
    mkdir -p ~/selfsigned-certs
    cd ~/selfsigned-certs
  2. Create CA's private key and note the password:
    openssl genrsa -aes256 -out ca-key.pem 4096
  3. Create a public certificate:
    openssl req -new -x509 -sha256 -days 365 -key ca-key.pem -out ca.pem -passin pass:<your_private_key_pw>
  4. To view certificate details:
    openssl x509 -in ca.pem -text
  5. Create client's private key:
    openssl genrsa -out client-key.pem 4096
  6. Create a Certificate Signing Request (CSR). Ensure the Common Name (CN) client's hostname where the certificate will be used:
    openssl req -new -sha256 -subj "/CN=hostname" -key client-key.pem -out client.csr
  7. Create the Client Certificate using the CA with a configuration file. In the configuration file, set DNS and IP based on your client details:
    echo "subjectAltName=DNS:*.oracle.com,IP:<client_public/private_ip" > extfile.cnf
    openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem -out client-key.pem -extfile extfile.cnf -CAcreateserial -passin pass:<your_private_key_pw>
  8. Verify client certificate:
    openssl verify -CAfile ca.pem -verbose client-key.pem

If you're configuring for a client on OCI compute instance, copy the CA certificate (ca.pem) to your local. To upload the certificate to OCI GoldenGate;

  1. Locate Truststore Certificates under Deployment Details/ Resources and click Add Truststore Certificate.

    OCI GoldenGate

     

  2. Provide a name for your key, select your key (ca.pem) and clik Add

    OCI GoldenGate

  3. Make sure that State is active after adding the certificate.

    OCI GoldenGate

     

Creating OCI GoldenGate Data Stream

  1. In OCI GoldenGate Console, go to Distribution Service/ Data Streams.

    OCI GoldenGate

     

  2. Provide a Name for your Data Stream and click Next.
     
  3. Provide Source Options:
    Trail Name:
    Name of the source trail file
    Encoding: JSON
    Quality of Service: Exactly Once
    Cloud Events Format: Check if you publish in CloudEvents format. 

    OCI GoldenGate

  4. In Filtering Options, you can set Rule Actions to include/ exclude object types/ object names. 
  5. Click Create Data Stream.

    OCI GoldenGate

     

When you click on the Data Stream, you can see the AsyncAPI yaml document that describes your data streaming API specs.

OCI GoldenGate

Our Data Stream is now ready for the client.

Configuring the python client to pull messages from OCI GoldenGate Data Stream:

If you want, you can use the simple python client I created for this blog. A few notes if you want to use it;

  • You need to update the Data Stream "uri" and path to client key file in the client.
  • Client will subscribe to OCI GoldenGate Data Stream (AsyncAPI endpoint), read the messages in the trail file and writes into websocket_messages.log file. You can stop the application to confirm the payloads.
  • Make sure that client is added to your VCN's security list.

This is a very simple OCI GoldenGate Data Streams client-side configuration with python. OCI GoldenGate Data Streams api endpoints are programming language agnositic and you can configure your own client in a similar way. In real world applications, much complicated dataStream payloads can be used during the handshake phase of the streaming protocol to configure the data streaming behavior. You can specify filtering rules, encoding format, and bufferSize along with the required data source trail name.

For more details about OCI GoldenGate Data Streams, you can refer to following documents:

Deniz Sendil

Senior Principal Product Manager

Deniz Sendil is a Senior Principle Product Manager in Oracle GoldenGate product group with a focus on GoldenGate for Big Data. Deniz has over 17 years of experience in data management, analytics and data integration domains. 


Previous Post

Improve GoldenGate Observability with StatsD and Telegraf

Alex Lima | 11 min read

Next Post


Oracle GoldenGate 23ai and Oracle Database 23ai Vectors

Alex Lima | 5 min read