Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

Synchronize Data between Source and Target in 2 Clicks!

The recently launched Data Integration Platform Cloud (DIPC) provides capabilities for various data integration requirements covering data transformation, integration, replication and governance. DIPC introduces the concept of Elevated Tasks to hide the complexity of underlying data integration processes used for achieving an end-to-end use case. Synchronize Data is the first such elevated task that allows you to synchronize a source schema with a target schema with no effort. Synchronize Data allows you to keep data in your target database schema in sync with a production database so that the target database can be used for real time Business Intelligence and reporting without affecting production system performance.

Let us quickly understand the challenges in implementing such a data synchronization solution. The entire process can be achieved in two steps. First perform initial load of the existing source data to target and then configure a replication process to replicate all the ongoing transactions to the target. Let’s understand  the steps required if you were implementing it using Oracle Data Integrator (ODI) and Oracle GoldenGate (OGG).

  1. Create ODI mappings for each of the tables in the source schema
  2. Create and run an ODI procedure to retrieve the System Change Number (SCN) from the database. The data up to this SCN will be loaded by initial load in ODI and all transactions after this SCN will be replicated by OGG
  3. Run OGG extract process to capture transactions from the source database
  4. Run the ODI mappings created in step 1 to perform initial load up to the SCN
  5. Run OGG pump to push trail files
  6. Run the replicat process to start applying transactions from SCN checkpoint

You notice that there are a number of intricate steps involved here that must be performed in the right order, across different products and requires a handshake between ODI and OGG. To implement all of this, you would require deep understanding of both ODI and OGG and it may take several days if not weeks to achieve the process end to end. Additionally, monitoring the progress of each of the steps and getting consolidated statistics will be another challenge.

With the new Synchronized Data task in DIPC, this entire operation is now done with few clicks without worrying about the complexity of the underlying steps. All you need to do is create a Synchronize Data task, which entails having to specify the source and target schema, and run it.  DIPC takes care of creating the appropriate ODI scenarios, retrieving SCN in ODI, passing SCN  from ODI to OGG, and initializing and running relevant OGG processes – OGG extract, OGG pump, OGG replicat. DIPC also provides central monitoring capability so that you can view the ongoing progress of each of the steps and their statistics.

Let us go through the steps for synchronizing data to see how easily you can do it in DIPC.

  1. First go to the DIPC home page and click on the create Synchronize Data Task

  1. On the Task creation screen enter the source and target information and click “Save and Run”. DIPC will save the task and kick off the execution. As part of execution DIPC will perform following operations
    1. Create ODI scenario to create tables in target schema and perform initial load
    2. Retrieve the System Changes Number (SCN) from the database
    3. Run OGG extract process to capture transactions from the source database
    4. Run the ODI scenario to perform initial load up to the SCN
    5. Run OGG pump to push trail files
    6. Pass SCN retrieved by ODI to GG Replicat process
    7. Start Replicat process to apply transactions from SCN

  1. Congratulations! You have created and executed the Synchronize Data Task. You can see the corresponding Job status on the Jobs page

  1. Click on the Job to see status and statistics of different steps. It provides you individual process level and consolidated statistics on inserts, updates, duration and lag.

  1. You can also view details on the underlying process that is executed for each step

As shown above, DIPC has drastically simplified the data synchronization use case between two databases. Now, anybody can implement such an end-to-end scenario without needing the deep expertise previously required or the  juggle of multiple underlying products.

Stay tuned for upcoming blogs on other exciting features introduced in DIPC. Meanwhile, check the product blogs to get more information: Data Integration Platform Cloud and Getting a Data Integration Platform Cloud (DIPC) Trial Instance.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.