Oracle GoldenGate 23ai now supports the public preview of integration into Microsoft Fabric via Open Mirroring.  This powerful combination unlocks real-time data integration, continuously synchronizing data across your hybrid and multicloud environments into mirrored databases in the Microsoft Fabric, enabling data to be always AI and analytics-ready.

Open Mirroring in Microsoft Fabric integration is delivered through the GoldenGate for Distributed Applications and Analytics 23ai (GG for DAA) (23.6) product.

Fabric Conenctivity

Learn more about other existing Azure integrations in our documentation.

Open Mirroring is designed to enable Data ISVs to extend Mirroring in Microsoft Fabric based on open Delta Lake table format. This capability enables Oracle GoldenGate 23ai to write change data directly into a Mirrored Database in Microsoft Fabric based on the open mirroring public APIs and approach.  To learn more about Open Mirroring in Microsoft Fabric, please visit Introducing Open Mirroring in Microsoft Fabric.

In this blog post, we will dive deeper into the implementation aspects of this integration and show how to configure the GoldenGate integration.

Configuration Steps:

  • Understand the reference architecture
  • Prerequisites
  • Step 1:  Login into the GG for DAA deployment
  • Step 2:  Create GG for DAA Replicat
  • Step 3:  Check the Replicat status and statistics
  • Step 4:  Verify replicated data in Mirrored Database Landing zone
  • Step 5:  Verify Open Mirroring status in Microsoft Fabric
  • Step 6:  Preview data in Mirrored Database Delta tables

Reference Architecture

The following diagram illustrates this reference architecture for an Oracle GoldenGate implementation hosted in Microsoft Azure for integration with Mirrored Database in Microsoft Fabric.

Fabric Reference Architecture


Prerequisistes

  • Install Oracle GoldenGate for Distributed Applications and Analytics 23ai (GG for DAA).
  • Deployment Configuration: To add a GG for DAA 23ai deployment, execute Oracle GoldenGate Configuration Assistant (oggca) program to start the wizard in Interactive or silent mode.Follow the steps outlined here to add a deployment using Oracle GoldenGate Configuration Assistant wizard. GG for DAA install home includes a sample trail file for replication.
  • Oracle GoldenGate extract (Oracle, Non Oracle or Big Data) capturing data from Source system is up and running. And the trails are being sent to GG for DAA Deployment.
  • Set GoldenGate Environment Variables: Configure the environment to find the JRE in the PATH, and the JVM shared library, using the appropriate environmental variable for your system. On Linux, set JAVA_HOME and LD_LIBRARY_PATH to include the directory containing the JVM shared library as follows:
    export JAVA_HOME=<ggdaa_install_home>/jdk
    export PATH=$JAVA_HOME/bin:$PATH
    export LD_LIBRARY_PATH=$JAVA_HOME/lib/server:$LD_LIBRARY_PATH
  • Create Mirrored Database in Microsoft Fabric: The Mirrored Database in Microsoft Fabric can either be created using publicly available Rest API or through the Microsoft Fabric console.
  • Download Dependencies: Traverse to the GG for DAA install home directory . Run the Dependency Downloader to download the Microsoft Fabric dependencies.To run the dependency downloader script:
    • Use a Unix terminal interface navigate to the following directory: {GGforDAA install}/opt/DependencyDownloader.
    • Execute the following to run the scripts: ./{the dependency script} {version of the dependencies to download}
      ./onelake.sh 12.20.0
    • Dependency libraries get downloaded to the following directory: {GGforDAAinstall}/opt/DependencyDownloader/dependencies/onelake_12.20.0
  • Create Microsoft Entra ID app for Authentication: GG for DAA Replicat uses the Microsoft Entra ID app for authentication with Microsoft Fabric. Configure it as follows:
    • Sign in to the Azure portal.
    • Search for and select Microsoft Entra ID.
    • From the Manage list, select App registration.
    • Select New registration.
    • Fill in the required information:
      • Name – Enter a name for your application.
      • Supported account types – Select the accounts you want your app to supported.
      • (Optional) Redirect URI – Select Web under “Select a Platform” dropdown.
    • Select Register.
    • Select the App Registration you created to access Microsoft Fabric.
    • Note down the Application (client) ID, Directory (tenant) ID displayed for the App.
    • Under Manage, select Certificates & secrets, click on New client secret to create a new secret and save the value securely for next steps. This generated key value string is available only once when the secret is created. If you do not know the generated secret, then create another secret making sure you capture the generated value string.
    • Traverse to the Fabric Workspace, and click on Manage Access.
       

      Fabric_Screen1

       

    • Click on Add people or Groups.Type the App name created in the previous steps and select the App. Assign the App Contributer role and click Add.
       

      Fabric_Screen2


Create and Configure the GG for DAA Replicat for Mirrored Database in Microsoft Fabric

Open the Service Manager login page in a web browser and log in to the Service Manager with your Oracle GoldenGate administrator user credentials. If logging in for the first time, you have to log in with the administrator account user credentials, created when adding your deployment with Oracle GoldenGate Configuration Assistant wizard.
 

GGMA23ai_UI

 

Step 1:  Login into the GoldenGate for Distributed Applications and Analytics deployment

From the Deployments list, click on the Administration Service of the GoldenGate for Distributed Applications and Analytics deployment. This opens the Administration Service login page. Log in to the Administration Service using the same credentials, which you used to log in to the Service Manager. The Administration Service Overview page is displayed.
 

GGMA23ai_AdminUI

Step 2:  Create GoldenGate for Distributed Applications and Analytics Replicat

To add a Replicat, click the + adjacent to “Replicats” label.
 

GGMA23ai_AddReplicat

  There are two Replicat types:

  • Classic Replicat is a single threaded process.
  • Coordinated Replicat is a multithreaded process that applies transactions in parallel.

Select the Classic Replicat as Replicat Type and click Next.
 

GGMA_ReplicatUI

Enter the Replicat options and click Next:

  • Trail Name: Name of the required trail file (if using sample trail, provide as tr)
  • Subdirectory: Enter GG_HOME/opt/AdapterExamples/trail/ if using the sample trail.
  • Target: Mirrored Database in Microsoft Fabric
     

GGMA23ai_ReplicatDetails

Leave Managed Options as default and click Next.

Enter Parameter File details and click Next. In the Parameter File, you can specify source to target mapping. If you’re using the sample trail file (tr) provide as followed: MAP QASOURCE.*, TARGET <your_schema_name>.*;
 

GGMA_ReplicatMapping

In the Properties file, update the properties marked as TODO:
 

gg.target=fabric_mirrored_database
#TODO: Edit the Fabric workspace name.
gg.eventhandler.onelake.workspace=<workspace-name>
#TODO: Edit the Fabric mirror name.
gg.eventhandler.onelake.mirror=<mirror-name>
#TODO: Edit the tenant ID of the application.
gg.eventhandler.onelake.tenantId=<azure-tenant-id>
#TODO: Edit the client ID of the application.
gg.eventhandler.onelake.clientId=<azure-client-id>
#TODO: Edit the client secret for the authentication.
gg.eventhandler.onelake.clientSecret=<azure-client-secret>
#TODO: Edit the classpath to include Hadoop, Parquet, and Azure DataLake SDK dependencies.
gg.classpath=$THIRD_PARTY_DIR/hadoop/*:$THIRD_PARTY_DIR/parquet/*:$THIRD_PARTY_DIR/onelake/*
#TODO: Edit the proxy configuration.
#jvm.bootoptions=-Dhttps.proxyHost=some-proxy-address.com -Dhttps.proxyPort=80 -Djava.net.useSystemProxies=true

Provide path to dependency jar files that you downloaded as part of Dependencies. Click Create and Run.
 

GGMA_ReplicatProps

 

Step 3:  Check the Replicat status and statistics

Check the Running status to ensure that the Replicat process is running.

GGMA_Replicat

Click on the Replicat name and click on the statistics tab in the left menu.
 

GGMA_ReplicatStats

 

Step 4:  Verify replicated data in Mirrored Database Landing zone

To verify that the data has been replicated to Mirrored Database, go to your Microsoft Fabric workspace. Click on the Mirrored Database name.
 

Fabric_ReplicatMirrorDB

 

Click on the View tab at the top. Then click on Replication status option to check the Status and number of Rows replicated by Microsoft Fabric Mirroring to the delta tables.
 

Fabric_mirrorview

The GoldenGate replicat process replicates data in parquet format to the OneLake Landing zone of the Mirrored Database. Alternatively, you can verify that the data has landed into the Mirrored database Landing zone successfully by expanding the Landing zone option in UI to see the schema name along with table names being replicated.
 

Fabric_MirrorLanding

 

Step 5:  Verify Open Mirroring status in Microsoft Fabric

Changes may take up to 60 seconds to appear under the Replication Status tab. Use the Refresh button to get the latest data.
 

Fabric_landingRefresh

 

Step 6:  Preview data in Mirrored Database Delta tables

Once the data is replicated successfully, preview the data for the replicated tables by selecting a table shown under Tables dropdown.
 

Fabric_MirorDeltaTable

 

Additional Oracle GoldenGate Open Mirroring Resources

Additional Popular Oracle GoldenGate Resources