Introduction

Oracle Cloud Infrastructure (OCI) GoldenGate is a managed service providing a real-time data mesh platform that uses replication to keep data highly available and enable real-time analysis.

OCI GoldenGate (OCI GG) is based on a Microservices Architectures that provides REST-enabled services as part of the Oracle GoldenGate environment. The REST-enabled services provide remote configuration, administration, and monitoring through its user interface.

The Oracle GoldenGate Stream Analytics (GGSA) is a complete solution platform for building applications to filter, correlate, and process events in real-time. With flexible deployment options of stand-alone Spark or Hadoop-YARN, it proves to be a versatile, high-performance event processing engine.

Why Kafka?

Apache Kafka is an open-source distributed event streaming platform used by thousands of customers. In several projects, it is required to mash up transactional data with Kafka processes. Oracle GoldenGate Stream Analytics provides the capabilities to do it. In this example, we will use GoldenGate and GoldenGate Stream Analytics to load data from an Oracle Database into Kafka.

Pre-requisites:

Before you get started, you must register your GGSA instance with OCI GoldenGate as a database. I know that GGSA is not a database, but we need to tell OCI GoldenGate how to establish a secure connection. OCI GG only allows connections through ports 443, 1521, and 1522.
This registration has some particularities. We must provide the FQDN and private IP as listed in the compute details of GGSA.

Example for GGSA instance:

 

 

 

 

 

 

You need to provide GoldenGate Stream Analytics hostname and the port 443. Don’t forget to tick “Network Connectivity via Private Endpoint” and add the private IP of your GGSA instance.

Create GGSA Connection 1

Create GGSA Connection 2

Then assign this Connection to your OCI GoldenGate deployment.

Assign GGSA Connection to OCI GoldenGate Deployment

Go to your GGSA instance and edit /u01/app/osa/osa-base/etc/osa-oci.properties:

  • Change instance.ip to full FQDN as listed in the compute details
  • Add the Kafka hostname to /etc/hosts

In OCI GG, create your Extract for the schema/table you want to load into Kafka. To get more information on how to create an Extract, click on this link.

Oracle GoldenGate Stream Analytics

1 – Setting up connections

Let’s start by creating the connection from GoldenGate Stream Analytics to OCI GG.

Log in to GGSA and go to Catalog. Create New Item → Connection → GoldenGate:

You must specify the OCI GG hostname, the port (443), and the username and password for the OCI GG deployment.

Now, create a connection for Kafka. In this example, we are using Kafka with SSL enabled. For Kafka without SSL, use port 9092 and don’t select any checkbox on security type:

2 – Creating GoldenGate Change Data Capture (CDC)

At this point, the only step missing is to create the Change Data Capture (CDC). For OCI GoldenGate, you need to copy and paste the deployment name in the Deployments field. Even so, it doesn’t show up after you paste it. It will be saved.

Now, start the Change Data Capture (CDC).

The CDC will create a distribution path in the background from OCI GoldenGate to Oracle GoldenGate Stream Analytics, and it will generate all configuration files for Kafka and a replicat.

Go to your OCI GoldenGate, and a distribution path should have been created:

If you go to your /dirprm directory, you will see that 3 new files were created with the GG Change Data Capture prefix defined in the previous step.

If you are not using SSL on Kafka, please skip to “3 – Loading data into Kafka”.
If you check your Kafka Server log, you will see that it contains some errors related to SSL:

That means your Kafka server is not accepting the connection from GoldenGate Stream Analytics. There are some missing configurations. Upload a valid Keystore and Truststore to GoldenGate Stream Analytics. In my case, I will use this directory to /u01/app/osa/tmp/ store both files.

Go to go to your /dirprm directory and edit the following file: <your_GG_Change Data_Capture>_kafka_producer.properties
kafka_producer.properties file 
Stop and start the CDC and now you shouldn’t see any error on Kafka Server.

3 – Loading data into Kafka

If you list your Kafka topics, you will notice that there is a new topic:
bin/kafka-topics.sh –list –bootstrap-server <host>:9093 –command-config /home/opc/kafka/kafka_2.13-3.0.0/config/client_ssl.properties

If it is not listed, please make an insert/update/delete on your source table, and it will show up next time you list your topics.

It has been created as a topic with the concatenation of the CDC name and the extract parameter.

To see the replication from Oracle DB to Kafka in real time, open a terminal shell and run the Kafka consumer:
bin/kafka-console-consumer.sh –bootstrap-server <kafka broker>:<port> –topic <topic_name> –from-beginning **–consumer.config /home/opc/kafka/kafka_2.13-3.0.0/config/client_ssl.properties** Only for Kafka SSL, remove this part if you are using port 9092

Now, insert a record on your source database:

Conclusion

As we have seen, we can easily use OCI GoldenGate and GoldenGate Streaming Analytics together to load data into Kafka!

To learn more about GoldenGate and GoldenGate Stream Analytics, check out all the blogs related to GoldenGate, the Oracle Live Labs for Oracle GoldenGate, and its documentation. Go to Oracle Cloud Free tier, where you can access a wide range of Oracle Cloud services for 30 days, including Databases, Analytics, Compute, and Container Engine for Kubernetes.