We recently released a new deployment type in OCI GoldenGate called Data Transforms. This new deployment type adds batch ETL/ELT processing capabilities to the OCI GoldenGate unified data fabric platform.
We will focus on creating Data Flows in this fourth technical article in the Get Started with OCI GoldenGate Data Transforms series.
What is a Data Flow?
A Data Flow in Data Transforms defines the flow of data and its transformations using a comprehensive library of built-in transformations:
- Data Transform: ETL operators (Join, Aggregate, Filter, …)
- Data Preparation: Collection of data preparation functions (Cleansing, Substitution, Lead, Lag, …)
- Machine Learning: Regression, classification, and anomaly detection algorithms
- Text: Regular expression and string manipulation functions
- Oracle Spatial and Graph: Access to SDO geometry and geocoding libraries
Create a Data Flow
You can create a Project in Data Transforms before creating a Data Flow. Refer to Get Started with OCI GoldenGate Data Transforms – Create a Data Load for more information.
In your Project, click Data Flows under Resources and click Create Data Flow.
Enter a Name, optionally a Description, and click Create. The Data Flow Editor opens up:
Click Add a Schema in the Data Entities panel on the left to import Data Entities from the Connections you have already defined. Select a Connection and a Schema, then click Import.
The Import Data Entities panel appears. You can specify a Mask to filter the objects that will be imported into Data Transforms and configure the available options.
Note: You can also import Data Entities from the Data Entities panel.
Click Start to kick off the metadata import process. Repeat the same steps to import the target metadata.
Once imported, the Data Entities appear in the Data Flow Editor.
You can then drag and drop Data Entities and transformation components into the Data Flow Editor canvas to define the flow of the data and its transformations. Here is an example of a Data Flow:
Click on the target Data Entity (TRG_CUSTOMER above) and open Column Mapping to map columns and define transformations.
The integration strategy (Append, Merge/Incremental Update, etc.) is set in the Options panel along with some options:
Refer to Data Flows in the Data Transforms documentation for more information.
Click Save (
) to save the Data Flow configuration.
You can also Validate the Data Flow and generate a preview of its code before executing it.
Data Flows can be executed individually, scheduled, or added into Workflows to orchestrate the overall data pipeline. The next article in the Get Started with OCI GoldenGate Data Transforms series focuses on creating Workflows to orchestrate the execution of your Cloud Data Pipelines.
Next Steps
Try it out! OCI GoldenGate Data Transforms is available today. Read the documentation to explore more features and benefits. You can also go through the OCI GoldenGate Data Transforms quickstart to familiarize yourself with the service capabilities.
More resources: