OCI GoldenGate now supports Open Mirroring in Microsoft Fabric

OCI GoldenGate now supports the public preview of integration into Microsoft Fabric via Open Mirroring. This powerful combination provides real-time data integration, continuously synchronizing data across your hybrid and multicloud environments into mirrored databases in the Microsoft Fabric, which helps enable up-to-date data for AI and analytics.

Learn more about other existing Azure integrations in our documentation.

In this blog post, we will dive deeper into the implementation aspects of this integration and show how to configure OCI GoldenGate’s Big Data deployment for data replication to a Mirrored Database in Microsoft Fabric.

Configuration Steps:

• Understand the reference architecture
• Prerequisites
• Step 1:   Login into OCI GoldenGate console
• Step 2:  Create/Upgrade Big Data deployment
• Step 3:  Create and test OCI GoldenGate connection for Mirrored Database
• Step 4:  Create Big Data Replicat for Mirrored Database
• Step 5:  Check the Replicat status and statistics
• Step 6:  Verify replicated data in Mirrored Database Landing zone
• Step 7:  Preview data in Mirrored Database Delta tables

Reference Architecture

The following diagram illustrates this reference architecture for OCI GoldenGate integration with Mirrored Database in Microsoft Fabric within Azure cloud.

Prerequisistes

• OCI GoldenGate needs a secure, low-latency interconnection between Oracle Cloud and Microsoft Azure.

  • Oracle Interconnect for Azure and a site-to-site VPN provide a path for data going from OCI to Azure. Data flows from the dynamic routing gateway on the OCI HUB VCN to the Site-to-Site VPN and Oracle Interconnect for Azure. The data from both sources flows through a virtual network gateway on the Azure VNet that includes the private link and endpoint. From there, it flows to the Azure resources.

  • Learn about interconnecting Oracle Cloud with Microsoft Azure.

• Create a Big Data deployment 23ai (v23.6+) in OCI GoldenGate
  Note: You can also upgrade your current Big Data deployment to v23.6+.

• Ensure that the GoldenGate extract (Oracle, Non Oracle or Big Data) is capturing data from the source system and the trails are being sent to the Big Data deployment.
  Note: You can use the sample trail file within the Big Data deployment for testing the replication.

• Create a Mirrored Database in Microsoft Fabric: The Mirrored Database in Microsoft Fabric can either be created using publicly available Rest API or through the Microsoft Fabric console.

• Within Microsoft Fabric Tenant settings, enable the option to allow an external app to use a service principal as an authentication method for access to Microsoft Fabric resources. Follow the steps outlined here.

• Within Microsoft Fabric Tenant settings, also enable the option to allow apps running outside of Fabric to access data via OneLake. Follow the steps outlined here.

• Create Microsoft Entra ID app for Authentication: OCI GoldenGate Big Data Replicat uses the Microsoft Entra ID app for authentication with Microsoft Fabric. Configure it as following

  • Sign in to the Azure portal.

  • Search for and select Microsoft Entra ID.

  • From the Manage list, select App registration.

  • Select New registration.

  • Fill in the required information:

    • Name – Enter a name for your application.

    • Supported account types – Select the accounts you want your app to supported.

    • (Optional) Redirect URI – Select Web under “Select a Platform” dropdown.

  • Select Register.

  • Select the App Registration you created to access Microsoft Fabric.

  • Note down the Application (client) ID, Directory (tenant) ID displayed for the App.

  • Under Manage, select Certificates & secrets, click on New client secret to create a new secret and save the value securely for next steps. This generated key value string is available only once when the secret is created. If you do not know the generated secret, then create another secret making sure you capture the generated value string.

  • Traverse to the Fabric Workspace, and click on Manage Access.

• Click on Add people or Groups.Type the App name created in the previous steps and select the App. Assign the App Contributer role and click Add.

Create and test OCI GoldenGate connection for Mirrored Database in Microsoft Fabric

To create an OCI GoldenGate connection for Mirrored Database in Microsoft Fabric:

• From the OCI GoldenGate Overview page, click Connections.
• On the Connections page, click Create Connection.
• On the Create Connection page, complete the fields as follows:
  • For Name, enter a name for the connection.
  • (Optional) For Description, enter a description that helps you distinguish this connection from others.
  • For Compartment, select the compartment in which to create the connection.
  • From the Type dropdown, select Microsoft Fabric Mirror.
  • For Azure tenant id, enter the Azure tenant ID of the application.
  • For Client id, enter the client ID of the application.
  • For Client secret, enter the client secret of the application for authentication.
  • Click Create. 

• After the connection is created, it appears in the Connections list. Be sure to assign the connection to a Big Data deployment to use it as a target.

• After you assign the connection to the deployment, test the connectivity to check that the deployment can connect to Microsoft Fabric. From the Action (three dots) menu for the connection you want to test, select Test connection.

• Some connections perform both tests, while others, such as Big Data connection types, don’t currently support application-level tests. As long as your connection passes network-level connectivity tests, you can use your connection and start creating Replicat processes.

Create and Configure the Big Data Replicat for Mirrored Database in Microsoft Fabric

Step 1:  Login into the Big Data deployment

• Log in to the OCI GoldenGate deployment console with the GoldenGate username and password specified while creating the deployment. The Administration Service Overview page is displayed.

Step 2:  Create a Replicat for Mirrored Database in Microsoft Fabric 

• To add a Replicat, click the + adjacent to “Replicats” label.

• There are two Replicat types:
  • Classic Replicat is a single threaded process.
  • Coordinated Replicat is a multithreaded process that applies transactions in parallel.
• Select the Classic Replicat as Replicat Type and click Next.

• Enter the Replicat options and click Next:
  • Trail Name: Name of the required trail file (if using sample trail, provide as tr)
  • Subdirectory: Enter /u01/ogg/opt/AdapterExamples/trail if using the sample trail.
  • Target: Mirrored Database in Microsoft Fabric

• Leave Managed Options as default and click Next.
  Enter Parameter File details and click Next. In the Parameter File, you can specify source to target mapping. If you’re using the sample trail file (tr) provide as followed: MAP QASOURCE.*, TARGET <your_schema_name>.*;

• In the Properties file, update the properties marked as TODO:

• Click Create and Run.

Step 3:  Check the Replicat status and statistics

• Check the Running status to ensure that the Replicat process is running.

• Click on the Replicat name and click on the statistics tab in the left menu.

Step 4:  Verify replicated data in Mirrored Database Landing zone

• To verify that the data has been replicated to Mirrored Database, go to your Microsoft Fabric workspace. Click on the Mirrored Database name.

• Under Monitor replication section, check the Status and number of Rows replicated by Microsoft Fabric Mirroring to the delta tables.

• Changes may take up to 60 secs to appear under the Monitor replication tab. Use the refresh button to get the latest data.

Step 6:  Preview data in Mirrored Database Delta tables

• Click on “Query in T-SQL” option to preview the data.

• Once the data is replicated successfully, preview the data for the replicated tables by selecting a table shown under Tables dropdown.You can also click on “New SQL Query” option to select the data using T-SQL query.

• The OCI GoldenGate Replicat process replicates the data in delta parquet format to the OneLake LandingZone of the Mirrored Database. Alternatively, you can also verify that the data has landed into the Mirrored database LandingZone successfully using Azure Storage Explorer.

Additional Resouces

• OCI GoldenGate Release Notes
• OCI GoldenGate documentation
• GoldenGate for Distributed Applications and Analytics 23.6 support for Microsoft Fabric
• Open Mirroring in Microsoft Fabric
• Microsoft Blog and Announcement for Open Mirroring in Microsoft Fabric
• Oracle Cloudworld Announcement