Oracle Cloud Infrastructure GoldenGate is a fully managed, native cloud service that moves data in real-time, at scale. OCI GoldenGate processes data as it moves from one or more data management systems to target databases. You can also design, run, orchestrate, and monitor data replication tasks without having to allocate or manage any compute environments. With its real-time data replication capability, OCI GoldenGate plays a key role in real-time data ingestion and no downtime migration use cases.
With recently released MongoDB capture functionality, OCI GoldenGate can be used for extracting data from MongoDB databases. MongoDB extract has been available with on-prem GoldenGate for almost a year and now OCI GoldenGate makes it faster, cheaper and better.
You need a MongoDB 3.6 or higher if you want to capture from MongoDB. Please note that we have identified some issues with MongoDB 5.5 capture and that’s an exception as of today. OCI GoldenGate captures Insert, Update and Delete operations from source MongoDB. DDL operations like Create Collection, Drop Collection, Rename Collection are not supported as of today. You can manage extract behaviour with two different parameters. If you use NO_ABEND_ON_DDL in your MongoDB extract, extract process skips DDL operations and continue processing the next operation. If you use ABEND_ON_DDL, the extract will abend in case of a DDL.
OCI GoldenGate supports two different extract modes for MongoDB:
- Initial Load Extract
- Initial Load Extract scans the full MongoDB database/ collection and writes them into a file format that can be used by initial load replicat process for different targets.
- Change Data Capture (CDC) Extract
- CDC Extract captures supported operations from MongoDB database/ collection and writes them into a trail file which can be used to replicate the captured data to a supported target. OCI GoldenGate reads the changed data from MongoDB oplog files and because of that, you need a replicaset configured in your MongoDB environment. If not, OCI GoldenGate change data capture will fail.
A short summary: MongoDB 3.6 or higher is supported, DDL operations are not supported today and replica set configuration is required for MongoDB capture. If you’re testing OCI GoldenGate MongoDB capture functionality, you can configure a single member replica set and use for testing. MongoDB reccomends minimum 3-member replica sets for real-file scenarios. To create a mongodb instance with a single replica set:
- Install MongoDB on a compute node
- In /etc/mongod.conf, update BindIp as 0.0.0.0 (this will allow access from clients)
- Add port 27017 to your VNC’s ingress rules
- Also check if you need to configure firewall setting on your node
- Create a directroy for data files (or use the default location)
- Start mongod with: bin/mongod –bind_ip <hostname/ ip> –port 27017 –replSet rs0 –dbpath ../data/d1/
- In a new terminal, connect with mongosh: mongosh <hostname/ ip>:27017
- Create user name & password for MongoDB:
- Initiate a replica set: rs.initiate()
- Check replica set’s status: rs.status()
Creating OCI GoldenGate MongoDB Connection
Click on Create Connection in OCI GoldenGate. Provide a Name, select a Compartment and select Type as MongoDB. Click Next.
Provide the Connection String without Username and Password. For example, if you use “mongodb://test:test@118.38.134.167:27017” for connecting to MongoDB instance from a client, you need to provide the connection string as mongodb://118.38.134.167:27017. Then, provide the Username and Password in the dedicated fields. Click Create.
MongoDB capture uses Oracle GoldenGate credential store to manage user IDs and their encrypted passwords (together known as credentials) that are used by Oracle GoldenGate processes to interact with the MongoDB database. The credential store eliminates the need to specify usernames and clear-text passwords in the Oracle GoldenGate parameter files.
MongoDB Capture uses connection URI to connect to a MongoDB deployment. Authentication and aecurity is passed as query string as part of connection URI. You can check security and authentication document for more details.
When connection is available, assign it to your OCI GoldenGate Big Data deployment.
Create MongoDB Initial Load Extract:
Log into OCI GoldenGate console and click Add Extract.
Select Source as MongoDB and Extract Type as Initial Load Extract. Click Next.
Enter Process Name, select the assigned connection in Connection_Alias, keep Source as Trail, enter two character Trail Name and click Next.
Update source database/ collection mapping with TABLE source.*; First part represents the database and second part represents the collection. For this blog, I’ll run the initial load from all collections in a database called dstest. I’ll update as TABLE dstest.*;
Click Create and Run.
To see the results, click Extract Details/ Report and scroll down in the report.
If your Initial Load Extract runs successfully, you’ll see the statistics at the end of the report file.
Create MongoDB Change Data Capture Extract:
Log into OCI GoldenGate console and click Add Extract.
Select Source as MongoDB and Extract Type as Change Data Capture Extract. Click Next.
Enter Process Name, select the assigned connection in Connection_Alias, leave Begin as Now, enter two-character Trail Name and click Next.
Update source database/ collection mapping with TABLE source.*; First part represents the database and second part represents the collection. For this blog, I’ll run extract from all collections in a database called dstest. I’ll update as TABLE dstest.*;
Click Create and Run.
You’ll see your extract in a running status with a green check. Go to Extract Actions, Details, Statistics to observe extract statistics.
