How to Replicate data to Azure Event Hub using Oracle GoldenGate for Big Data

You need to follow the installation and configuration steps as listed below:

1. Spin-up a Linux virtual machine(VM) on Azure Cloud.
2. Download the GoldenGate for Big Data 19.1.0.0.5 Binaries and transfer it to the Azure VM.
   https://www.oracle.com/middleware/technologies/goldengate-downloads.html
   

   

3. Oracle GoldenGate for Big Data requires that you install Oracle Java 8 Java Runtime Environment (JRE) at a minimum. So, Download Java 1.8.202
   https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.htm
   
   Download the GoldenGate for Big Data installers Linux x64.tar.gz file from OTN or E-delivery or Oracle Support Portal 
   

   

4. Download kafka_2.12-2.2.1 from https://kafka.apache.org/downloads  
   

   

5. Extract/Unzip GoldenGate for Big data,Java1.8 and kafka_2.12-2.2.1 into separate directories.
   

   

   

   

6. Configure        JAVA_HOME   and      LD_LIBRARY_PATH
   export JAVA_HOME=<Path_To_Java1.8_Directory>/jdk1.8.0_202
   export LD_LIBRARY_PATH<Path_To_Java1.8_Directory>/jre1.8.0_202/lib/amd64/server:$JAVA_HOME/lib
    
7. Traverse to the GoldenGate for Big Data directory and login into the ggsci prompt
   Edit the parameter file of manager and specify the manager PORT, Save and close it
   Edit params mgr
   PORT 7809
   Start mgr

   

   Now, we need to set up the target on Azure.
    
8. From the Azure portal, Navigate to All Services and select Event Hubs and follow the steps mentioned in the below link to setup an Event Hub Namespace https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create. And make sure you select the Standard option under Pricing Tier as Basic option doesn’t allow Kafka to be enabled for the Event Hub.
   

   

   
    
9. Go Back to the GoldenGate directory, Traverse to AdapterExamplesàbig-data-àKafka
   And copy sample rkafka.prm, custom_producer.properties and kafka.props file to GoldenGate dirprm directory.
   

   

10. Open the rkafka.prm file  and change the MAP/TARGET schema name according to the schema name at the source database. For testing, we are using the sample config trail and parameter file
    REPLICAT rkafka
    — Trail file for this example is located in “AdapterExamples/trail” directory
    — Command to add REPLICAT
    — add replicat rkafka, exttrail AdapterExamples/trail/tr
    TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
    REPORTCOUNT EVERY 1 MINUTES, RATE
    GROUPTRANSOPS 10000
    MAP QASOURCE.*, TARGET QASOURCE.*;
     
11. Add the replicat using the following command-
    add replicat rkafka, exttrail <Trail_file_path_from_Source>
    For testing, you can use the sample trail file that comes with the GoldenGate binaries-
    add replicat rkafka, exttrail AdapterExamples/trail/tr
     
12. Open the Kafka props file and change the gg.classpath to the correct kafka  installation directory path under #Sample gg.classpath for Apache Kafka.
    gg.handlerlist = kafkahandler
    gg.handler.kafkahandler.type=kafka
    gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
    #The following resolves the topic name using the short table name
    gg.handler.kafkahandler.topicMappingTemplate=${tableName}
    #The following selects the message key using the concatenated primary key
    gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
    gg.handler.kafkahandler.format=json
    #gg.handler.kafkahandler.SchemaTopicName=mySchemaTopic
    gg.handler.kafkahandler.BlockingSend =false
    gg.handler.kafkahandler.includeTokens=false
    gg.handler.kafkahandler.mode=op
    #gg.handler.kafkahandler.metaHeadersTemplate=${alltokens}
    gg.handler.kafkahandler.transactionsEnabled=false
    auto.create.topics.enable=TRUE
    goldengate.userexit.writers=javawrite
    javawriter.stats.display=TRUE
    javawriter.stats.full=TRUE
    gg.log=log4j
    gg.log.level=INFO
    gg.report.time=30sec
    #Sample gg.classpath for Apache Kafka
    gg.classpath=dirprm/:/home/oracle/kafkalibs/kafka_2.12-2.2.1/libs/*
    javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

13. Make changes to custom_producer.properties file, by adding your EventHUB connect string and password  needed to connect to Azure Event hub

bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093

security.protocol=SASL_SSL

sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule  required username=”$ConnectionString” password=”Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX”;

For more information: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs

14. Login into your GoldenGate instance and start the Replicat process-

./ggsci

Start rkafka

15. Now check the Statistics for the Replicat by using the stats command-

16. Note that the topics are created automatically based on the table name in Azure Event Hub, as we have set gg.handler.kafkahandler.topicMappingTemplate=${tableName} .

17. Check the stats/Incoming Messages on the Azure Portal (Event Hub)-

Content contributed by Mr. Shrinidhi Kulkarni, Senior Solution Engineer, Solution Engineering NA Technology, Oracle