Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

Data Loader Task in Oracle Cloud Infrastructure (OCI) Data Integration

Product Manager

Today, let's explore the Data Loader Task in Oracle Cloud Infrastructure Data Integration.  The Data Loader Task enables the use-cases where one to one loading of data is required between two systems with minimal/no transformations. This task is required for Data Preparation, Data Migration, or for loading diverse data into Data Lakes like Oracle Object Storage.

With the Data Loader Task, the entire process is now done with few clicks without worrying about the complexity of the underlying steps. All the user needs to do is create a Data Loader Task, which entails having to specify the source and target with the option to specify transformations, publish and run it. Let us consider an example where data is loaded from Oracle Database to Autonomous Data Warehouse (ADW). Here in this use case, the following are the steps performed to load table data from Oracle Database to ADW. 

  1. Before Creating the Data Loader Task all the required Data Assets are configured within the Workspace. Here, in this use case, Oracle Database and ADW are configured.  For more understanding on Data Assets refer to Data Asset in Oracle Cloud Infrastructure Data Integration.

  2. To create a Data Loader Task - Log into your Workspace, navigate to Projects, open your Project, go to Tasks, and click on Create Task then select Data Loader.

  3. Provide the name and Description related to Data Loader Task. The identifier is auto-populated and can be changed only during creation.

  4. Select the applicable Source in the Data Loader Task by providing the Connection, Schema, and Data Entity.

  5. Transformations are optional in the Data Loader Task. By clicking "Configure" next to Configure Transformation you're brought to the interactive Data Xplorer, where you can view a data sampling from the source Data Asset, and apply transformations to the attributes or data values. 

  6. For this use case, we have applied one of the transformation rules:
    • EMPL_NAME is in Proper Case and will be transformed into Upper Case. In the Data tab click on three dots next to EMPL_NAME to select the "Change Case" transformation rule then select Upper in Type and click Apply. The Data tab will refresh and a new attribute containing the transformed data will appear (E_UPPER_NAME).

  7. The summary of the transformations can be seen by clicking the icon beside Data Entity. After all the validations are done, click on "OK".

  8. In the main screen of the Data Loader Task, the number of transformations applied can be seen.

  9. Select the Target in the Data Loader Task similar to the Source:

  10. Once the required Source, Transformations, and Target are configured then click on "Save and Close" to create the Data Loader Task. Once it is created it can be seen under Projects/Folder.

  11. For any Task, i.e. Data Loader or Integration to run in Oracle Cloud Infrastructure Data Integration, it needs to be published in an Application and then the corresponding Task can be executed. An Application is a container for published tasks and their dependencies. Data Integration includes one default Application that you can publish tasks to. You can create additional Applications if needed. Click the Actions (three dots) menu for the task and select Publish then pick your Application.

  12. Once the Task is published, go to the Application --→ "Default Application" within the Workspace and under Patches the published tasks can be seen. By clicking on "View Details" for a particular patch (three dots) menu the task name appears.

  13. Go to the Applications and then Run the Task
  14. After running the task the executed jobs can be seen under Task Runs with the number of processed records, Duration, Executed By, Execution Date, etc.

As shown above, the Data Loader Task has drastically simplified the loading of the data between two systems. Now, users can implement such an end-to-end scenario without needing deep data integration expertise.

Have you seen some of our other blogs?  Check out all the blogs related to Oracle Cloud Infrastructure Data Integration - https://blogs.oracle.com/dataintegration/oracle-cloud-infrastructure-data-integration. To learn more, check out some Oracle Cloud Infrastructure Data Integration Tutorials and Documentation.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.