X

Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

Try Oracle Cloud Platform For Free

How to Replicate data to Azure Event Hub using Oracle GoldenGate for Big Data

Thomas Vengal
Director, Product Management

You need to follow the installation and configuration steps as listed below:

  1. Spin-up a Linux virtual machine(VM) on Azure Cloud.
  2. Download the GoldenGate for Big Data 19.1.0.0.5 Binaries and transfer it to the Azure VM.
    https://www.oracle.com/middleware/technologies/goldengate-downloads.html
  3. Oracle GoldenGate for Big Data requires that you install Oracle Java 8 Java Runtime Environment (JRE) at a minimum. So, Download Java 1.8.202
    https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.htm

    Download the GoldenGate for Big Data installers Linux x64.tar.gz file from OTN or E-delivery or Oracle Support Portal 
  4. Download kafka_2.12-2.2.1 from https://kafka.apache.org/downloads  
  5. Extract/Unzip GoldenGate for Big data,Java1.8 and kafka_2.12-2.2.1 into separate directories.
  6. Configure        JAVA_HOME   and      LD_LIBRARY_PATH
    export JAVA_HOME=<Path_To_Java1.8_Directory>/jdk1.8.0_202
    export LD_LIBRARY_PATH<Path_To_Java1.8_Directory>/jre1.8.0_202/lib/amd64/server:$JAVA_HOME/lib
     
  7. Traverse to the GoldenGate for Big Data directory and login into the ggsci prompt
    Edit the parameter file of manager and specify the manager PORT, Save and close it
    Edit params mgr
    PORT 7809
    Start mgrNow, we need to set up the target on Azure.
     
  8. From the Azure portal, Navigate to All Services and select Event Hubs and follow the steps mentioned in the below link to setup an Event Hub Namespace https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create. And make sure you select the Standard option under Pricing Tier as Basic option doesn’t allow Kafka to be enabled for the Event Hub.

     
  9. Go Back to the GoldenGate directory, Traverse to AdapterExamplesàbig-data-àKafka
    And copy sample rkafka.prm, custom_producer.properties and kafka.props file to GoldenGate dirprm directory.
  10. Open the rkafka.prm file  and change the MAP/TARGET schema name according to the schema name at the source database. For testing, we are using the sample config trail and parameter file
    REPLICAT rkafka
    -- Trail file for this example is located in "AdapterExamples/trail" directory
    -- Command to add REPLICAT
    -- add replicat rkafka, exttrail AdapterExamples/trail/tr
    TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
    REPORTCOUNT EVERY 1 MINUTES, RATE
    GROUPTRANSOPS 10000
    MAP QASOURCE.*, TARGET QASOURCE.*;
     
  11. Add the replicat using the following command-
    add replicat rkafka, exttrail <Trail_file_path_from_Source>
    For testing, you can use the sample trail file that comes with the GoldenGate binaries-
    add replicat rkafka, exttrail AdapterExamples/trail/tr
     
  12. Open the Kafka props file and change the gg.classpath to the correct kafka  installation directory path under #Sample gg.classpath for Apache Kafka.
    gg.handlerlist = kafkahandler
    gg.handler.kafkahandler.type=kafka
    gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
    #The following resolves the topic name using the short table name
    gg.handler.kafkahandler.topicMappingTemplate=${tableName}
    #The following selects the message key using the concatenated primary key
    gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
    gg.handler.kafkahandler.format=json
    #gg.handler.kafkahandler.SchemaTopicName=mySchemaTopic
    gg.handler.kafkahandler.BlockingSend =false
    gg.handler.kafkahandler.includeTokens=false
    gg.handler.kafkahandler.mode=op
    #gg.handler.kafkahandler.metaHeadersTemplate=${alltokens}
    gg.handler.kafkahandler.transactionsEnabled=false
    auto.create.topics.enable=TRUE
    goldengate.userexit.writers=javawrite
    javawriter.stats.display=TRUE
    j    avawriter.stats.full=TRUE
    gg.log=log4j
    gg.log.level=INFO
    gg.report.time=30sec
    #Sample gg.classpath for Apache Kafka
    gg.classpath=dirprm/:/home/oracle/kafkalibs/kafka_2.12-2.2.1/libs/*
    javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

13. Make changes to custom_producer.properties file, by adding your EventHUB connect string and password  needed to connect to Azure Event hub

bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093

security.protocol=SASL_SSL

sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule  required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";

For more information: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs

 

14. Login into your GoldenGate instance and start the Replicat process-

./ggsci

Start rkafka

15. Now check the Statistics for the Replicat by using the stats command-

 

16. Note that the topics are created automatically based on the table name in Azure Event Hub, as we have set gg.handler.kafkahandler.topicMappingTemplate=${tableName} .

 

17. Check the stats/Incoming Messages on the Azure Portal (Event Hub)-

Content contributed by Mr. Shrinidhi Kulkarni, Senior Solution Engineer, Solution Engineering NA Technology, Oracle

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.

Recent Content