X

Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

ODI - Integration with Oracle Storage Cloud Service

Oracle Storage Cloud Service provides a low cost, reliable,
secure, and scalable object-storage solution for storing unstructured data and
accessing it anytime from anywhere. It provides means to stage data in the
cloud from where it can be consumed by many other cloud services such as Oracle Business
Intelligence Cloud Service (BICS)
and Oracle Big Data Preparation Cloud
Service
. These cloud services directly ingest a file from Oracle Storage Cloud
Service. Therefore integration with Oracle Storage Cloud Service will be very
useful to orchestrate end to end integration flows using Oracle Data Integrator.
The below screenshot shows the Big Data Preparation Cloud Service that can
ingest source files from the Oracle Storage Cloud Service.

Oracle Storage Cloud Service exposes RESTful
web services
as the primary way for accessing it. There is a java
library
available that provides wrapping Java APIs for these RESTful web
services and can be used for programmatic access to Oracle Storage Cloud.

In the below sections we will go through how to create an Oracle
Data Integrator open tool for Oracle Storage Cloud that can upload or
download a file to the Oracle Storage Cloud. It will be developed using the
Oracle Data Integrator open
tool development framework
and utilizes the Oracle Storage Cloud Java APIs.
Such open tool can be used directly in Oracle Data Integrator packages as part
of the integration process orchestration.

Developing Open Tool
for Oracle Storage Cloud Service

Oracle Data Integrator provides the flexibility to develop
an open tool for performing custom tasks and it can be easily developed with
very little programming knowledge. The Developing
Open Tools
section in Developers guide provides details on implementing the
open tool. Let’s revisit the steps to
create an open tool for Oracle Storage Cloud.

  • Extract the Java library's classes and add them to your Java application's class path. This will allow
    you to use the Oracle Storage Cloud Services Java APIs in your program implementing
    the ODI open tool.
  • Create a
    16x16 image and a 32x32 pixel image for ODI to use as open tool icon in ODI
    Studio.
  • Write a java class extending OpenToolAbstract
public class SimpleMessageBox extends oracle.odi.sdk.opentools.OpenToolAbstract {
  • Define all required open tools parameters
private static final IOpenToolParameter[] mParameters = new IOpenToolParameter[] {
     new OpenToolParameter(“-URL”, "URL", " Storage Cloud URL", true),
     new OpenToolParameter(“-SERVICE”, "Service Name", "Service Name", true),
     ...
};
  • Implement the getter function getParameters() to retrieve the
    parameters
public IOpenToolParameter[] getParameters()
{
      return mParameters;
} 
  • Implement
    functions to return information about your Open Tool:
public String getDescription() {return " Open Tool for Storage Cloud Service" ;}
public String getVersion() { return "v1.0"; }
public String getProvider() { return "Oracle"; }
  • Implement the getSyntax() method which determines the name of the Open Tool as it
    is displayed in the Oracle Data Integrator graphical interface, and also the
    initial values of the parameter.
public String getSyntax()
{
      return "StorageCloudService \"-URL=<url>\" \"-Service=<service name>\" ...";
    } 
  • Implement getIcon()
    method to return paths to the two images created in earlier steps
public String getIcon(int pIconType)
{
      switch (pIconType)
      {
             case IOpenTool.SMALL_ICON:
               return  "images/StorageCloud_16.png";
             case IOpenTool.BIG_ICON:
              return “images/StorageCloud_32.gif";
             default:
              return "images/StorageCloud_16.png";
     }
} 
  • Finally, implement execute()method
    encapsulating the actual operation performed by the Open Tool.
public void execute() throws Exception
{  ...  }
  • In execute() method get the storage cloud connection as follows
{C}{C}
CloudStorageConfig csConfig = new CloudStorageConfig();
csConfig.setServiceName(serviceName).setUsername(user).setPassword(password.toCharArray()).setServiceUrl(url);              
CloudStorage cConnection = CloudStorageFactory.getStorage(csConfig); 
  • Use the above cloud connection to perform
    operations on Storage Cloud objects
List<Container> cntrList  = myConnection.listContainers();
...
Container cntr = myConnection.createContainer(“ContainerName”);
myConnection.storeObject(“ContainerName”, “fileName”, fileType, fis);

 

Reference OpenTool
for Oracle Storage Cloud

The reference implementation of the open tool for Oracle
Storage Cloud Service is available on OTN: Storage Cloud Service OpenTool. It contains a couple of open tools – OracleStorageCloudPut
and OracleStorageCloudGet – to
upload and download files from Oracle Storage Cloud service. It provides
following features

  • Create container if it does not exist during
    upload
  • Upload a single file
  • Upload all the files within a folder
  • Upload the files satisfying given Java regular
    expression as file name
  • Download single file
  • Download a set of files satisfying given file
    prefix
  • Retry downloading file on intermittent failure.
  • Option to overwrite or append to an existing
    local file

The screenshot below shows the OracleStorageCloudGet tool used in a package. Notice the addition
of two new open tools in the Toolbox list. You can hard code the parameter
values directly or pass them using variable as shown in below example where the password is
passed using a variable.

Note: In order to use the referenced Jar files you will need
to download the Oracle
Storage Cloud Service Java libraries
and add them in your ODI Studio or Agent
class path.

The source code of the open tools is also available inside
the open tool jar. So you can customize the open tool further to make it more
robust as per you requirement.

Conclusion

Oracle Data Integrator’s open tool framework can be
leveraged to quickly get access to the Oracle Storage Cloud Service, which is
gradually becoming an essential part for integrating on premise data to many
cloud services. The reference implementation of an open tool for Oracle Storage
Cloud is now available on the Data Integration project on OTN: Storage Cloud Service OpenTool which can be used and modified as per
your integration needs.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.