Welcome back to this series of blog posts. This is part 2 of the 3 part blog series. In part 1, we have seen an OCI-DI OIC integration flow and OIC integration implementation steps. In this blog, we will look into the OCI-DI data flow Implementation steps.
OCI DI data flow Implementation:
Sign in to OCI console. Click on the hamburger menu on your top left corner. Navigate as Analytics & AI à Data Integration and then click on the OCI DI workspace you created.
Create a Data Asset for REST:
Refer to the blog https://blogs.oracle.com/dataintegration/post/extracting-rest-data-using-oracle-cloud-infrastructure-data-integration-oci-di for detailed information about creating a new data asset for REST. I have followed this blog to create a new data asset for REST connection to OIC integration. Special Thanks to GuruDixit Chepuri
Provide an OIC integration base endpoint URL and upload a manifest file. You have noted down this URL and saved the manifest file in the previous steps (previous post).
Add a default connection.
Select the authentication type as Basic Authentication and Use vault secret OCID. Provide the OIC username and vault secret OCID. I have created a vault secret for OIC password in the same OCI tenancy. You can also use the Use password option for which you need to enter the password in the text box.
If you remember, we have used security policy as OAuth 2.0 or Basic Authentication in the AllPurposeRestTrigger OIC connection (refer to OIC integration implementation steps in the previous post). Any requests to OIC integration are authenticated first using Basic Authentication security policy.
I have created two other new data assets. One for Object Storage and another for Oracle ADW. Refer to these blogs for the detailed steps:
Create an OCI-DI data flow.
Refer to the interactive Data Integration Designer Tour (https://docs.cloud.oracle.com/en-us/data-integration/dataflow-tour/index.html) for a hands-on introduction to data flows.
Oracle documentation: https://docs.oracle.com/en-us/iaas/data-integration/using/data-flows.htm
OCI-DI Data flow – Get and Load OIC Suppliers
GET_SUPPLIERS source action uses Get Suppliers – OIC REST data asset and is configured to use auto-populated base OIC integration endpoint URL, relative resource URI and operation. Relative resource URI and operation are fetched from the manifest file and are available to choose from the dropdown.
Fields that are exposed through OIC integration are visible under Attributes tab.
CSV_ID_SOURCE source action uses OCI Object storage data asset and is configured to read the supplier ids csv file from OCI Object storage bucket.
Corresponding expression operators are used to convert the supplier id field data type from VARCHAR to LONG.
In Look up operator, Supplier Ids of supplier Id file are looked into the supplier Id of supplier JSON data API response (OIC integration response), and then the matched supplier ids and records are finally loaded into Oracle ADW table.
ADW_TARGET target action uses Oracle ADW data asset and is configured to insert the records into SUPPLIERS_TARGET table.
Create an integration task Get and Load OIC Supplier task for this data flow and publish the task to an application.
In the next (or last) part of this blog series, we will run the integration and see the results.