Today, we will learn about Parameters in Oracle Cloud Infrastructure Data Integration. Parameters enable users to externalize and override configuration values at runtime or while creating Data Flows and Tasks so that the same Data Flow designs can be reused with different configurations (different source or target locations, different filters, or join conditions, etc.). To further understand Parameters let me walk you through an example. We will be using a Data Flow with two sources, a join condition, and a target table. This Data Flow is moving data from a development environment to a testing environment. Later on, we will change the Parameter values to read data from the production environment. Below are the steps describing how parameters can be assigned in a Data Flow.
Pre-Requisites – Based on the previous blogs on Oracle Cloud Infrastructure Data Integration and Oracle Documentation, you have set up Workspace, Projects, Applications, and understood the creation and execution of a Data Flow using Integration Task. Below are the links for the reference:
– Workspace in Oracle Cloud Infrastructure (OCI) Data Integration
– Data Asset in Oracle Cloud Infrastructure (OCI) Data Integration
– Project Setup in Oracle Cloud Infrastructure (OCI) Data Integration
– Data Flow overview in Oracle Cloud Infrastructure (OCI) Data Integration
–
Understanding Parameters
Parameters need to be created in a Data Flow. The created Parameters can be configured or modified during the Task creation or while running the Task. In the below example, Data Flow with name “DF_Loading_Data_into_ADW” has the following operators:
- Source Operators (COUNTRIES and REGIONS) – Data Entities are in Autonomous Data Warehouse (ADW) development environment
- Join Operator (JOIN_CNTRY_RGN)
- Target Operator (COUNTRIES_RGN) – Target is in Autonomous Data Warehouse (ADW) testing environment
Now, what if you want to reuse the same Data Flow logic with production schema? In our example, we have two tables for Regions and Countries in the Autonomous Data Warehouse (ADW) production environment (in Germany) that we would like to use instead of the ADW development environment (in Ashburn) while configuring/running the Task. This scenario is useful in validating the Data Flow and identifying any functional issues in the early phases of testing. The other use case is reusing the same Data Flow logic when processing like-shaped data.
In the Data Flow “DF_Loading_Data_into_ADW” we will assign Parameters to the Source Operators Countries and Regions. To add a new Parameter, Click on Source Operator, next to the Data Asset field click on “Assign Parameter” then click Add New Parameter. In the Add New Parameter panel, provide the Parameter name and Data Asset that needs to be parameterized. By default, the Data Asset “DEV_ADW_ASHBURN” pointing to the development environment is selected.
In the below screenshot, Parameters are assigned to Data Asset, Connection, and Schema. We have not assigned any Parameter for Data Entity since the Data Entity would be having the same name in the production environment.
Similarly for Regions
In addition, a Parameter may be required for the Join Operator when the source Data Entity names are the same but the Data Entity structure might be different in terms of column name or additional columns in different environments (i.e. in this case development and production environment). During Task configuration or runtime, join conditions can be re-defined based on the environments.
All the Parameters assigned can be seen in the corresponding Data Flow by clicking on view -→ Parameters -→ config. The below screenshot represents a list of Parameters created in the Data Flow “DF_Loading_Data_into_ADW”. Also, under the view tab, we can check where the Parameters are being used.
Assigned Parameters in the Data Flow can be Deleted by selecting them or by clicking on the Action three-dot icon.
Parameters can be configured in the Integration Task. The Integration Task enables us to run the Data Flows that are already created under Projects/Folders. With the help of Oracle Cloud Infrastructure Data Integration Tasks, We can create multiple Tasks with distinct configurations for the same Data Flow. To understand more about Integration Tasks, refer to – Integration Tasks in Oracle Cloud Infrastructure (OCI) Data Integration.
These assigned Parameters can be changed in the Integration Task by clicking on “Edit” in Configure Parameters. If Parameters are not configured in the Data Flow then the “Configure Parameters” block is not visible while creating the Integration Task. In this example, the Integration Task with the name “IntegrationTask_ADW_Production” has been created.
These Parameters can be also changed during runtime. To run the Task, first, it needs to be published in an Application, and from Application, you can run the corresponding Integration Task. Here two Integration Tasks have been created with the name “IntegrationTask_ADW_Production” with Production ADW (in Germany) as a source and “IntegrationTask_ADW_Development” with Development ADW (in Ashburn) as a source.
A new screen appears which will ask to configure or select the existing Parameters
