Thursday Oct 08, 2015

Featuring Big Data Preparation Cloud Service and other Cloud Data Integration Sessions at Oracle OpenWorld 2015

Oracle OpenWorld is almost upon us! We are excited to be sharing with you some previews of what will be seen and discussed in just a few weeks in San Francisco!

One of the highlights is Oracle’s new Cloud Based Data Preparation Solution, Oracle Big Data Preparation Cloud Service, also known as BDP.  This new service will revolutionize the process of importing, preparing and publishing your complex business data and getting it ready for use allowing you to spend more time analyzing data rather than preparing data for analysis. Users are guided through the process with intuitive recommendation driven interfaces. The system also provides various ways to automate and operationalize the entire data preparation pipeline via the built in scheduler or via a rich set of RESTful API’s.

During OpenWorld, Oracle’s Luis Rivas, alongside Blue Cloud Innovations’ Vinay Kumar and Pythian’s Alex Gorbachev will discuss and demonstrate how big data promises many game-changing capabilities if tackled efficiently! You will discover how Oracle Big Data Preparation Cloud Service takes “noisy” data from a broad variety of sources in many different formats, both structured and unstructured, and uses sophisticated and unique blend of machine learning and Natural Language Processing based on a vast set of linked open reference data that provide a powerful way to ingest, prepare, enrich, and publish it into useful data streams, ready for further discovery, analysis, and reporting. Don’t miss it:

CON9615 Solving the “Dirty Secret” of Big Data with Oracle Big Data Preparation Cloud Service

Tuesday, Oct 27, 5:15 p.m. | Moscone South—310

Curious to find out more about BDP before the conference? Take a look here and view a short video: Chalk Talk: Oracle Big Data Preparation Cloud Service!

Since we are on the topic of Data Integration and the Cloud – I will also take a quick moment to remind everyone about Oracle Data Integrator’s (ODI) integration relative to the Oracle Storage Cloud Service as well for example. But that’s not all – here is a view into the Data Integration sessions that relate to the Cloud – in chronological order:

CON3506 Into the Cloud and Back with Oracle Data Integrator 12c

Monday, Oct 26, 5:15 p.m. | Moscone West—2022


CON9614 Oracle Data Integration Solutions: the Foundation for Cloud Integration

Wednesday, Oct 28, 11:00 a.m. | Moscone South—274


CON9717 Accelerate Cloud Onboarding Using Oracle GoldenGate Cloud Service

Wednesday, Oct 28, 3:00 p.m. | Moscone West—2022


CON9595 Cloud Data Quality: Lessons Learned from Oracle’s Journey to the Sales Cloud

Thursday, Oct 29, 12:00 p.m. | Moscone West—2022


CON9612 Oracle Enterprise Metadata Management and the Cloud

Thursday, Oct 29, 1:15 p.m. | Marriott Marquis—Salon 4/5/6

We hope you will join a few! Don’t forget to view the Focus on Data Integration – for a full review of Data Integration Sessions during OpenWorld. See you there!

Wednesday Oct 07, 2015

More on Leveraging Oracle Data Integrator (ODI) for Cloud Applications

Take a look at this week’s A-Team Blog post: Need to Integrate your Cloud Applications with On-Premise Systems… What about ODI? This blog is tightly coupled to the recent blog post: A Universal Cloud Applications Adapter for ODI.

First you learned about the simplicity of leveraging Oracle Data Integrator (ODI) with all emerging technologies in the world of cloud computing. Now read about which Cloud JDBC drivers will allow you to expand your data integration initiatives to include PaaS and SaaS connectivity.

For more A-Team reads on ODI, browse through the A-Team Chronicles.

Tuesday Aug 04, 2015

Simplicity in Leveraging Oracle Data Integrator for Cloud Applications

Check out last week’s A-Team Blog post… A Universal Cloud Applications Adapter for ODI

Learn about the simplicity of leveraging Oracle Data Integrator (ODI) with all emerging technologies in the world of cloud computing!

For more A-Team reads on ODI, browse through the A-Team Chronicles.

Wednesday Jul 08, 2015

ODI KMs for Business Intelligence Cloud Service

In this article we will learn how to leverage Oracle Data Integrator’s extensible Knowledge Modules (KM) framework to create knowledge modules to load data into the Oracle Business Intelligence Cloud Service (BICS). The following instructions are targeted for BICS instances using Schema Cloud Service, if your BICS instance uses Database as a Service then you can directly load data into DBaaS tables as described in the blog post ODI 12c and DBaaS in the Oracle Public Cloud. More details on implementing Knowledge Modules can be found in Knowledge Module Developer Guide.

BICS exposes REST APIs that allows programmatically creating, managing, and loading schemas, tables, and data into Oracle BI Cloud Service. We will invoke these REST APIs using the Jersey client libraries providing wrapper implementation for invoking RESTful web services. The sample implementation of it is available on : RKM and IKM for Oracle BI Cloud Service
[Read More]

Tuesday Jul 07, 2015

Chalk Talk Video: Kick-Start Big Data Integration with Oracle

Next in the series for Oracle Data Integration chalk talk videos, we speak to Oracle Data Integrator (ODI) for big data. ODI allows you to become a big data developer without learning to code Java and Map Reduce! ODI generates the code and optimizes it with support for Hive, Spark, Oozie, and Pig.

View this video to learn more: Chalk Talk: Kick-Start Big Data Integration with Oracle.

For additional information on Oracle Data Integrator, visit the ODI homepage and the ODI for Big Data page. This blog can be very handy also: Announcing Oracle Data Integrator for Big Data.

Wednesday Jul 01, 2015

ODI - Integration with Oracle Storage Cloud Service

Oracle Data Integrator’s open tool framework can be leveraged to quickly get access to the Oracle Storage Cloud Service, which is gradually becoming an essential part for integrating on premise data to many cloud services. The reference implementation of an open tool for Oracle Storage Cloud is now available on the Data Integration project on ODI OpenTool for Oracle Storage Cloud which can be used and modified as per your integration needs. [Read More]

Tuesday Jun 09, 2015

Oracle Data Integrator Journalizing Knowledge Module for GoldenGate Integrated Replicat Blog from the A-Team

As always, useful content from the A-Team…

Check out the most recent blog about how to modify the out-of-the-box Journalizing Knowledge Module for GoldenGate to support the Integrated Replicat apply mode.

An Oracle Data Integrator Journalizing Knowledge Module for GoldenGate Integrated Replicat


Wednesday May 13, 2015

Looking for Cutting-Edge Data Integration: 2015 Excellence Awards

It is nomination time!!!

This year's Oracle Fusion Middleware Excellence Awards will honor customers and partners who are creatively using various products across Oracle Fusion Middleware. Think you have something unique and innovative with Oracle Data Integration products?

We'd love to hear from you! Please submit today in the Big Data and Analytics category.

The deadline for the nomination is July 31, 2015. Win a free pass to Oracle OpenWorld 2015!!

Let’s reminisce a little…

For details on the 2014 Data Integration Winners: NET Serviços and Griffith University, check out this blog post.

For details on the 2013 Data Integration Winners: Royal Bank of Scotland’s Market and International Banking and The Yalumba Wine Company, check out this blog post.

For details on the 2012 Data Integration Winners: Raymond James Financial and Morrisons Supermarkets, check out this blog post.

We hope to honor you!

Click here to submit your nomination today. And just a reminder: the deadline to submit a nomination is 5pm Pacific Time on July 31, 2015.

Tuesday May 12, 2015

ODI 12c - Improving Usability of KM recipes

This post is all about reducing user errors and improving usability surrounding definition of Knowledge Modules and their usage. Knowledge Modules are all about encapsulating recipes - every great cookbook has lots of recipes and some are based on common techniques and ingredients. ODI Knowledge Modules are data integration recipes - they define how to access, transform and store information based on the directions in the KM. There are a few usability improvements in the recent release around both the KM definition and usage of the KM that make for an improved user experience. I've seen many KMs over the years where its many things to many people and there are a bundle of options that expose all facets for every path possible in the KM - the user has to read the description and follow the instructions.

The first improvement I'll mention is the KM (and procedure) option type of 'Choice'. Not exactly rocket science here I know, but an addition that greatly helps usage of a KM that may do more than one thing. Let's take the example of a KM that can make different .....pizzas. In the past you would have an option field which was a string based value where the user would type either margerita or pepperoni to drive a path within the KM implementation, users of the KM would have to know that those were the accepted option values and they'd have to type it in properly (otherwise things would go wrong). So now the options can be specified as the 'Choice' type, see below where in the IKM we capture the recipe type as a choice.

The choices can be defined in the default value field, below the recipe is going to either create margherita pizza or pepperoni- these are the only two choices and the default is margherita;

Then I can define all the rest of the options, let's say the pizza needs flour, oil, salt, yeast and pepperoni needs... pepperoni of course and margherita needs tomatoes and basil - so some of the options are applicable to both types and some are only applicable to the specific one. Prior to this release when the KM is used you would see all of these option values and you'd be reading the description 'only set basil if you are making margherita' and so on. Another feature has been added to improve this area. Below you can see all of the options....

One column was snipped out of the image - the condition expression. This is a groovy expression to determine whether the option is shown. So now we can say only display basil when margherita pizza is the recipe type or only display pepperoni when pepperoni is the recipe type. We see below the options only applicable to the recipe type are displayed - anything common has no condition expression.

The groovy snippet must return a string. The string must be of the format show=true|false

When you see the KM assigned in the mapping it becomes a little clearer. Below you can see the choice box, the user is constrained to pick one of those types;

 When margherita is selected above remember some options were for margherita and some were for pepperoni, we see a subset of options;

Above you can see tomatoes and basic, if you change the type to pepperoni the above options are hidden and pepperoni is displayed as below;

This helps guide the user into configuration options that are more applicable to a path within the KM. One of the other visual enhancements is the ability to group options together. We can add all of the options above into a group named 'Ingredients' that helps visually group related options together;

 Then when this is used you see the options related to ingredients from where the KM is assigned.

You can see how these help improve the usability of KMs in ODI and help reduce errors by further specializing how data is entered and related in the configuration options of the KM. The tasks within the KM can retrieve the option values and perform condition code based on those values. There are some other areas around this but that's all for now. The functionality described here is available in the

Friday Apr 10, 2015

This Week's A-Team Blog Speaks to Automating Changes after Upgrading ODI or Migrating from Oracle Warehouse Builder

The A-Team not only provides great content, they are humorous too!

Check out this week’s post, the title says it all: Getting Groovy with Oracle Data Integrator: Automating Changes after Upgrading ODI or Migrating from Oracle Warehouse Builder

The article covers various scripts written in Groovy and leverage the ODI SDK that assist in automating massive changes to one’s repository. These initially came to be as a result of customer desire in enhancing their environment in their effort to move from Oracle Warehouse Builder (ODI) to Oracle Data Integrator (ODI), but in the end came the realization that these scripts could be used by any ODI user.

Happy reading!

Thursday Mar 26, 2015

Oracle Big Data Lite 4.1.0 is available with more on Oracle GoldenGate and Oracle Data Integrator

Oracle's big data team has announced the newest Oracle Big Data Lite Virtual Machine 4.1.0.  This newest Big Data Lite Virtual Machine contains great improvements from a data integration perspective with inclusion of the recently released Oracle GoldenGate for Big Data.  You will see this in an improved demonstration that highlights inserts, updates, and deletes into Hive using Oracle GoldenGate for Big Data with Oracle Data Integrator performing a merge of the new operations into a consolidated table.

Big Data Lite is a pre-built environment which includes many of the key capabilities for Oracle's big data platform.   The components have been configured to work together in this Virtual Machine, providing a simple way to get started in a big data environment.  The components include Oracle Database, Cloudera Distribution including Apache Hadoop, Oracle Data Integrator, Oracle GoldenGate amongst others. 

Big Data Lite also contains hands-on labs and demonstrations to help you get started using the system.  Tame Big Data with Oracle Data Integration is a hands-on lab that teaches you how to design Hadoop data integration using Oracle Data Integrator and Oracle GoldenGate. 

                Start here to learn more!  Enjoy!

Thursday Feb 19, 2015

Hive, Pig, Spark - Choose your Big Data Language with Oracle Data Integrator

The strength of Oracle Data Integrator (ODI) has always been the separation of logical design and physical implementation. Users can define a logical transformation flow that maps any sources to targets without being concerned what exact mechanisms would be used to realize such a job. In fact, ODI doesn’t have its own transformation engine but instead outsources all work to the native mechanisms of the underlying platforms, may it be relational databases, data warehouse appliances, or Hadoop clusters.

In the case of Big Data this philosophy of ODI gains even more importance. New Hadoop projects are incubated and released on a constant basis and introduce exciting new capabilities; the combined brain trust of the big data community conceives new technology that outdoes any proprietary ETL engine. ODI’s ability to separate your design from the implementation enables you to pick the ideal environment for your use case; and if the Hadoop landscape evolves, it is easy to retool an existing mapping with a new physical implementation. This way you don’t have to tie yourself to one language that is hyped this year, but might be legacy in the next.

ODI enables the generation from logical design into executed code through physical designs and Knowledge Modules. You can even define multiple physical designs for different languages based on the same logical design. For example, you could choose Hive as your transformation platform, and ODI would generate Hive SQL as the execution language. You could also pick Pig, and the generated code would be Pig Latin. If you choose Spark, ODI will generate PySpark code, which is Python with Spark APIs. Knowledge Modules will orchestrate the generation of code for the different languages and can be further configured to optimize the execution of the different implementation, for example parallelism in Pig or in-memory caching for Spark.

The example below shows an ODI mapping that reads from a log file in HDFS, registered in HCatalog. It gets filtered, aggregated, and then joined with another table, before being written into another HCatalog-based table. ODI can generate code for Hive, Pig, or Spark based on the Knowledge Modules chosen. 

 ODI provides developer productivity and can future-proof your investment by overcoming the need to manually code Hadoop transformations to a particular language.  You can logically design your mapping and then choose the implementation that best suits your use case.

Friday Jan 09, 2015

ODI 12c - Mapping SDK Overview

In this post I'll show some of the high level concepts in the physical design and the SDKs that go with it. To do this I'll cover some of the logical design area so that it all makes sense. The conceptual model for logical mapping in ODI 12c is shown below (it's quite a change from 11g), the model below allows us to build arbitrary flows. Each entity below you can find in the 12c SDK. Many of these have specialized classes - for example MapComponent has specializations for the many mapping components available from the designer - these classes may have specific business logic or specialized behavior. You can use the strongly typed, highly specialized classes like DatastoreComponent or you can write applications in a generic manner using the conceptual high level SDK - this is the technique I used in the mapping builder here.

The heart of the SDK for this area of the model can be found here;

If you need to see these types in action, take the mapping illustration below as an example, I have annotated the different items within the mapper. The connector points are shown in the property inspector, they are not shown in the graphical design. Some components have many input or output connector points (for example set component has many input connector points). Some components are simple expression based components (such as join and filter) we call these selector components, other components project a specific shape, we call those projectors - that's just how we classify them. 

In 12c we clearly separated the physical design from the logical, in 11g much of this was blended together. In separating them we also allow many physical designs for a logical mapping design. We also had to change the physical SDK and model so that we could support multiple targets and arbitrary flows. 11g was fairly rigid - if you look at the 'limitations' sections of the KMs you can see some of that. KMs are assigned on map physical nodes in the physical design, there are some helper methods on execution unit so you can set/get KMs.

The heart of the SDK for this logical mapping area of the model can be found here;

If we use the logical mapping shown earlier and look at the physical design we have for it, we can annotate the items below so you can envisage how each of the classes above is used in the design;

The MapPhysicalDesign class has all of the physical related information such as the ODI Optimization Context and Default Staging Location (there also exists a Staging Location Hint on the logical mapping design) - these are items that existed in ODI 11g and are carried forward.

To take an example if I want to change the LKMs or IKMs set on all physical designs, one approach would be to iterate through all of the nodes in a physical design and you can check whether an LKM or an IKM is assigned for that node - this then let;s you do all sorts - from get the current setting, to setting it with a new value. The snippet below gives a small illustration using groovy of the methods from the ODI SDK;

  1.         PhysicalDesignList=map.getPhysicalDesigns()
  2.          for (pd in PhysicalDesignList){
  3.             PhysicalNodesList=pd.getPhysicalNodes()
  4.             for (pn in PhysicalNodesList){
  5.                 if (pn.isLKMNode()){
  6.                     CurrentLKMName=pn.getLKMName()
  7. ...
  8.                          pn.setLKM(myLKM) 
  9.                 }else if (pn.isIKMNode()){
  10.                     CurrentIKMName=pn.getIKMName()
  11. ...
  12.                      pn.setIKM(myIKM)

There are many other methods within the SDK to do all sorts of useful stuff - first example is the getAllAPNodes method on a MapPhysicalDesign. This gives all of the nodes in a design which will have LKMs assigned - so you can quickly set or check. The second example is the getTargetNodes method on MapPhysicalDesign - this is handy to get all target nodes to set IKMs on, Final example is to find an AP node in the physical design for a logical component in your design - use the method findNode to achieve this.

Hopefully there are some useful pointers here, worth being aware of the ODI blog on Mapping SDK the ins and outs which provides an overview and cross reference to the primary ODI objects and the underpinning SDKs. If there are any other specific questions let us know.

Wednesday Dec 17, 2014

Oracle Partition Exchange Blog from the ODI A-Team

More great information from the ODI A-Team!

Check out the A-Team’s most recent blog about the Oracle Partition Exchange – it does come in two parts:

Using Oracle Partition Exchange with ODI

Configuring ODI with Oracle Partition Exchange

The knowledge module is on Java.Net, and it is called “IKM Oracle Partition Exchange Load”.  To search for it, enter “PEL” or “exchange” in the Search option of Java.Net.

A sample ODI 12.1.3 Repository is available as well.  The ODI sample repository has great examples of how to perform both initial and incremental data upload operations with Oracle Partition Exchange.  This repository will help users to understand how to use Oracle Partition Exchange with ODI.

Happy reading!

Wednesday Dec 10, 2014

Oracle Enterprise Metadata Management is now available!

As a quick refresher, Metadata Management is essential to solve a wide variety of critical business and technical challenges which include how report figures are calculated, understanding the impact of changes to data upstream, providing reports in a business friendly way in the browser and providing reporting capabilities on the entire metadata of an enterprise for analysis and improvement. Oracle Enterprise Metadata Management is built to solve all these pressing needs for customers in a lightweight browser-based interface. Today, we announce the availability of Oracle Enterprise Metadata Management as we continue to enhance this offering.

With Oracle Enterprise Metadata Management, you will find business glossary updates, updates for a better experience to the user interface as well as improved and new metadata harvesting bridges including Oracle SQL Server Data Modeler, Microsoft SQL Server Integration Services, SAP Sybase PowerDesigner, Tableau and more. There are also new dedicated web pages for tracing data lineage and impact! At a more granular level you will also find new customizable action menus per repository object type for more personalization. For a full read on new features, please read here. Additionally, view here for the certification matrix details.

Download Oracle Enterprise Metadata Management!


Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality


« October 2015