Calling an Oracle Integration Cloud integration from OCI Data Integration Data flow using a Generic REST connection

In this blog, I will show you how we can call an Oracle Integration Cloud (OIC) integration from OCI Data Integration (DI) Data flow using a Generic REST connection.

This is part 1 of the 3 part blog series.

Introduction

REST APIs provide a flexible, lightweight way to integrate applications, and have emerged as the most common method for connecting components in micro services architectures. An API, or application-programming interface, is a set of rules that define how applications or devices can connect to and communicate with each other. A REST API is an API that conforms to the design principles of the REST, or representational state transfer architectural style. For this reason, REST APIs are sometimes referred to RESTful APIs.

OCI DI is a fully managed, multi-tenant service that helps data engineers and ETL developers with common extract, transform, and load (ETL) tasks such as ingesting data from a variety of data assets; cleansing, transforming, and reshaping that data; and efficiently loading it to target data assets.

With OIC, you have the power to integrate your cloud and on-premises applications, automate business processes, gain insight into your business processes, develop visual applications, use an SFTP-compliant file server to store and retrieve files, and exchange business documents with a B2B trading partner.

There are many reasons why we have to take an OIC integration route from OCI DI flow. Few of them listed below:

1. If your applications/systems the response you are expecting from them are through REST APIs and are backed by advance security policies like OAuth, API key etc.
2. Those APIs are expecting some kind of custom headers in the request and responses.
3. You have a requirement to do an orchestration and enrichment by calling multiple applications/systems and then send back the consolidated response.
4. You have to do some complex transformations on the JSON or XML data that you received after making multiple calls as part of an orchestration.
5. You wanted to leverage broad range of OIC adapters to connect to your end systems or applications.
6. Split huge chunk of data file using OCI DI first and process each smaller chunk of data files in OIC.
7. Convert Parquet, Avro and Excel files to CSV using OCI DI and process converted CSV file in OIC

In this blog, I am taking a use case of loading supplier’s data into Oracle ADW.

Integration flow diagram:

The supplier id file is uploaded into OCI Object storage bucket from external sources. This file is read by OCI DI data flow when the flow is kicked off. In this data flow, we have two sources, one is to read the supplier Id file and another is to call or invoke an OIC integration using Generic REST connection. OIC integration is exposed as a REST API and when its triggered, it receives a request from OCI DI data flow, maps the request parameters, standard and custom headers, calls an external API (here taking an example of Get Suppliers from Oracle ERP Cloud), maps the suppliers data response that it received from an external API call to an integration response and then sends the integration response to an OCI DI data flow.

Corresponding expression operators are defined in OCI DI data flow for converting the varchar data type to long data type for supplier id field. This is required for easy look-ups. In Look up operator, Supplier Ids of supplier Id file are looked into the supplier Id of supplier JSON data API response, and then the matched supplier ids and records are finally loaded into Oracle ADW table.

OIC integration implementation

Sign in to OIC console.

Create connections for External Suppliers REST API and trigger for OIC integration:

Click on the hamburger menu on your top left corner. Navigate as Integrations à Connections

Refer to the oracle documentation https://docs.oracle.com/en/cloud/paas/integration-cloud/integrations-user/create-connections.html for how to create connections, test and save the connections in OIC.

REST ERP VIS Connection (for External Suppliers REST API call)

Select security policy based on how your External REST API endpoint is secured. Here I have selected the Basic Authentication option and provided a username and password.

You can select other security policies like OAuth, API Key etc. based on your requirement.

AllPurposeRestTrigger (for to trigger an OIC integration). You can select Security policy as OAuth 2.0 or Basic Authentication. This means an OIC integration using this connection as a trigger will be authenticated through either OAuth 2.0 or Basic Authentication security mechanism.

Create an integration. In OIC console, navigate as Integrations à Integrations. 

Refer to the oracle documentation https://docs.oracle.com/en/cloud/paas/integration-cloud/integrations-user/create-integrations.html for how to create integrations, save and activate integrations in OIC.

Get Suppliers (with App Driven Orchestration style)

First drag and drop an AllPurposeRestTrigger connection to create a trigger action.

In trigger action, you configure the relative resource URI, REST method and response payload of an integration.

Here, the response payload used is same as external Get Suppliers REST API. It may be different in case of any rich orchestrations or enrichments needed in the integration flow.

Next, create an invoke action to an External Get Suppliers REST API by dragging and dropping the connection you created in the previous steps.

Configure the invoke action with an endpoint name, optional description, relative resource URI, REST method, template or query parameters, standard headers or custom headers, request/response payloads of an external Get Suppliers REST API.

Response Payload:

Data mappers are auto created once trigger and invoke actions are created.

Map integration request to an external Get Suppliers REST API request.

Map external Get Suppliers REST API response to an integration response.

Save and activate the integration.

Get the OIC base URL and manifest file:

Hover on to the activated integration and then click on Run. Then click on the Metadata URL.

Note down the URL under Endpoint URL. This is your base OIC integration endpoint URL.

Click the URL under Open API.

Save the Open API specs as a manifest file.

OIC integration base endpoint URL and manifest file is needed in OCI DI REST data asset connection (next post).

In the next post (part 2) of this blog series, we will look into the OCI-DI data flow implementation steps.