Monday Feb 24, 2014

Highlighting Oracle Data Integrator 12c (ODI12c)

Towards the last two months of 2013 we highlighted several features of ODI12c's various features with full length blogs for each of the features. This was so popular that we bring you a one stop shop where you can browse through the various entries at your convenience. This is a great page to bookmark, even if we say it ourselves if you are using or thinking of using ODI12c.

Blog Title

Blog Description

Kicking off the ODI12c Blog Series

Talks about ODI12c themes and features at a high level shedding light on the new releases focus areas.

ODI 12c's Mapping Designer - Combining Flow Based and Expression Based Mapping

Talks about ODI's new declarative designer with the familiar flow based designer.

Big Data Matters with ODI12c

Talks about ODI12c enterprise solutions for the movement, translation and transformation of information and data heterogeneously and in Big Data Environments.

ODI 12c - Aggregating Data

Look at the aggregation component that was introduced in ODI 12c for composing data with relational like operations such as sum, average and so forth.

ODI 12c - Parallel Table Load

Looks at the ODI 12c capability of parallel table load from the aspect of the mapping developer and the knowledge module developer - two quite different viewpoints.

In-Session Parallelism in ODI12c

Discusses the new in-session parallelism, the intelligence to concurrently execute part of the mappings that are independent of each other, introduced in the ODI12c release.

ODI 12c - Mapping SDK the ins and outs

Talks about the ODI 12c SDK that provides a mechanism to accelerate data integration development using patterns and the APIs in the SDK

ODI 12c - XML improvements

Explains ODI support to advanced XML Schema constructs including union, list, substitution groups, mixed content, and annotations.

ODI 12c - Components and LKMs/IKMs

Illustrates capabilities of ODI 12c's knowledge module framework in combination with the new component based mapper.

Welcome Oracle Management Pack for Oracle Data Integrator! Let’s maximize the value of your Oracle Data Integrator investments!

To help you make the most of Oracle Data Integrator, and to deliver a superior ownership experience in an effort to minimize systems management costs, Oracle recently released Oracle Management Pack for Oracle Data Integrator.

ODI 12c - Mapping SDK Auto Mapping

If you want to properly leverage the 12c release the new mapping designer and SDK is the way forward.

ODI 12c - Table Functions, Parallel Unload to File and More

Helps you integrate an existing table function implementation into a flow.

And below is a list of “how to” and “hands on” blogs about ODI 12c and to get started.

ODI 12c - Getting up and running fast

A quick A-B-C to show you how to quickly get up and running with ODI 12c, from getting the software to creating a repository via wizard or the command line, then installing an agent for running load plans and the like.

Time to Get Started with Oracle Data Integrator 12c!

We would like to highlight for you a great place to begin your journey with ODI.  Here you will find the Getting Started section for ODI,

ODI 12c - Slowly Changing Dimensions

Helps setup a slowly changing dimension load in ODI 12c, everything from defining the metadata on the datastore to loading the data.

ODI 12c - Temporal Data Loading

The temporal validity feature in 12c of the Oracle Database is a great feature for any time based data. If you are thinking dimensional data that varies over time.... the temporal validity capabilities of 12c are a great fit, worth checking it out.

ODI 12.1.2 Demo on the Oracle BigDataLite Virtual Machine

Oracle's big data team has just announced the Oracle BigDataLite Virtual Machine, a pre-built environment to get you started on an environment reflecting the core software of Oracle's Big Data Appliance 2.4. BigDataLite is a VirtualBox VM that contains a fully configured Cloudera Hadoop distribution CDH 4.5, an Oracle DB 12c, Oracle's Big Data Connectors, Oracle Data Integrator 12.1.2, and other software.

Webcast - Oracle Data Integrator 12c and Oracle Warehouse Builder

If you missed the recent Oracle Data Integrator 12c and Oracle Warehouse builder live webcast. You can catch up on the events and connect with us with your feedback here. Here we discuss customer examples,ODI12c new features, Big Data compatibility, Oracle Warehouse Builder Migration Utility and Support and live Q and A among other topics.  

 For more information on Oracle Data Integrator visit the ODI Resource Center.

Thursday Jan 30, 2014

Webcast - Oracle Data Integrator 12c and Oracle Warehouse Builder

If you missed the recent Oracle Data Integrator 12c and Oracle Warehouse builder live webcast do not worry. You can still catch up on the events and connect with us with your feedback here. Here we discuss customer examples,ODI12c new features, Big Data compatibility, Oracle Warehouse Builder Migration Utility and Support and live Q and A among other topics.  

Register to listen. 

ODI12c and OWB Webcast.

Here are some questions from the audiences that were answered following the webcast.

Question

Response

Is the management pack free to ODI customers?

While the Management Pack for ODI works to consolidate and manage your ODI infrastructure it is a separately licensed product.  

- You can learn more about the Management Pack for ODI in this Data Sheet

Also visit the Oracle Management Pack for ODI homepage for more details.

Where can we download ODI 12c?

The ODI 12c can be downloaded in various ways including Virtual Machines and complete downloads from the Oracle Technology Network.  

This page also gives the latest on trainings, patches and other happenings around Oracle Data Integrator so keep this page bookmarked to follow the latest on ODI.

Thank you.

Happy to help, please let me know if you need any other assistance. Thank you for your participation.

Could you expand on how the time reduction is effected through parallelism?

As you might have suspected the time reductions depends on various factors including database tunings, network capabilities and code efficiencies.

ODI’s parallelism offers parallel threading and intelligent identification of parallelizable components.

This blog on the in-session parallelism functionality explains the workings and logic behind the process.

What has Oracle done to ensure any applications written today using existing KM's will work going forward for any ODI upgrades?

While we do our best to ensure consistency across the versions with Knowledge Modules - any Knowledge Modules you have changed would require additional testing to make sure everything works as it should.

Customization allows your Knowledge Modules to work most efficiently but also means changes from the out of the box KMs provided. Unfortunately there is no simple/easy answer outside of further testing.

Could you tell more about integrations to Workday? (real-time)

Workday integration is currently provided through some of our partners , mostly with customized Knowledge Modules.  For help in contacting such a partner, please reach out to your Oracle Data Integration Sales Manager.

Does Oracle Business Intelligence Application 11.1.1.7.1 offer a choice to use ODI or a third party DAC?

Oracle BI Apps version 11.1.1.7.1 which was released in May of 2013 was built on Oracle Data Integrator and fully functions only on ODI.

Is ODI 12c Generally Available?

Yes, ODI 12c has been generally available since October of 2013. For more information on the product please go to the ODI homepage.


You can also download the product here

Can you provide high level differences between Oracle Data Integrator and Golden Gate ?

At a high level, ODI is bulk data movement and transformation - known often as ETL/ELT. 

OGG is a high-volume, low latency replication/synchronization product.  For more information - please view the various datasheets, whitepaper on the two products that can both be found here.

What versions of E-Business Suite does ODI 12c require?

The certification matrix gives an overview of the compatible technologies and standards for ODI. 

You can also find more information in the E business documentation

Do you support connection, extraction and upload from QuickBase and into Quickbase in ODI 12c ?

For Intuit QuickBase there is an ODBC driver which might work in conjunction with the generic SQL KMs shipped with ODI. 

More information can be found here: http://qunect.com/. 

Please note that QuickBase is not certified with ODI. For a full set of certified products refer to the certification matrix.

Is it correct that there is no Oracle Warehouse Builder in 12c ?

That is correct. From a packaging perspective, OWB is not shipped with Oracle DB 12c, however, previous OWB releases can connect to Oracle DB 12c.  

The Statement of Direction around ODI and OWB explains the reasons and direction forward for both the products.

Great Webinar - would love to see more of these

Great! The feedback has been great for all our audiences and we shall definitely find ways to bring ODI to you. Meanwhile we invite you to follow our blogs and facebook channels.

Wednesday Jan 29, 2014

ODI 12.1.2 Demo on the Oracle BigDataLite Virtual Machine

Oracle's big data team has just announced the Oracle BigDataLite Virtual Machine, a pre-built environment to get you started on an environment reflecting the core software of Oracle's Big Data Appliance 2.4. BigDataLite is a VirtualBox VM that contains a fully configured Cloudera Hadoop distribution CDH 4.5, an Oracle DB 12c, Oracle's Big Data Connectors, Oracle Data Integrator 12.1.2, and other software.

You can use this environment to see ODI 12c in action integrating big data with Oracle DB using ODI's declarative graphical design, efficient EL-T loads, and Knowledge Modules designed to optimize big data integration. 

The sample data contained in BigDataLite represents the fictional Oracle MoviePlex on-line movie streaming company. The ODI sample performs the following two steps:

  • Pre-process application logs within Hadoop: All user activity on the MoviePlex web site is gathered on HDFS in Avro format. ODI is reading these logs through Hive and processes activities by aggregating, filtering, joining and unioning the records in an ODI flow-based mapping. All processing is performed inside Hive map-reduce jobs controlled by ODI, and the resulting data is stored in a staging table within Hive.
  • Loading user activity data from Hadoop into Oracle: The previously pre-processed data is loaded from Hadoop into an Oracle 12c database, where this data can be used as basis for Business Intelligence reports. ODI is using the Oracle Loader for Hadoop (OLH) connector, which executes distributed Map-Reduce processes to load data in parallel from Hadoop into Oracle. ODI is transparently configuring and invoking this connector through the Hive-to-Oracle Knowledge Module.

Both steps are orchestrated and executed through an ODI Package workflow. 

Demo Instructions

Please follow these steps to execute the ODI demo in BigDataLite:

  1. Download and install BigDataLite. Please follow the instructions in the Deployment Guide at the download page
  2. Start the VM and log in as user oracle, password welcome1.
  3. Start the Oracle Database 12c by double-clicking the icon on the desktop.


  4. Start ODI 12.1.2 by clicking the icon on the toolbar.


  5. Press Connect To Repository... on the ODI Studio window. 


  6. Press OK in the ODI Login dialog.


  7. Switch to the Designer tab, open the Projects accordion and expand the projects tree to Movie > First Folder > Mappings. Double-click on the mapping Transform Hive Avro to Hive Staging.


  8. Review the mapping that transforms source Avro data by aggregating, joining, and unioning data within Hive. You can also review the mapping Load Hive Staging to Oracle the same way. 

    (Click image for full size)

  9. In the Projects accordion expand the projects tree to Movie > First Folder > Packages. Double-click on the package Process Movie Data.


  10. The Package workflow for Process Movie Data opens. You can review the package.


  11. Press the Run icon on the toolbar. Press OK for the Run and Information: Session started dialogs. 




  12. You can follow the progress of the load by switching to the Operator tab and expanding All Executions and the upmost Process Movie Data entry. You can refresh the display by pressing the refresh button or setting Auto-Refresh. 


  13. Depending on the environment, the load can take 5-15 minutes. When the load is complete, the execution will show all green checkboxes. You can traverse the operator log and double-click entries to explore statistics and executed commands. 

This demo shows only some of the ODI big data capabilities. You can find more information about ODI's big data capabilities at:


Monday Jan 06, 2014

Welcome Oracle Management Pack for Oracle Data Integrator! Let’s maximize the value of your Oracle Data Integrator investments!

To help you make the most of Oracle Data Integrator, and to deliver a superior ownership experience in an effort to minimize systems management costs, Oracle recently released Oracle Management Pack for Oracle Data Integrator. This new product leverages Oracle Enterprise Manager Cloud Control's advanced management capabilities to provide an integrated and top-down solution for your Oracle Data Integrator environments. Management Pack for Oracle Data Integrator supports both 11g (11.1.1.7.0 and higher) and 12c versions of Oracle Data Integrator.

Management Pack for Oracle Data Integrator provides a consolidated view of your entire Oracle Data Integrator infrastructure. This enables users to monitor and manage all their components centrally from Oracle Enterprise Manager Cloud Control.

Performance Monitoring and Management

Management Pack for Oracle Data Integrator streamlines the monitoring of the health, performance, and availability of each and all components of an Oracle Data Integrator environment – this includes Master and Work Repositories, Standalone and JEE agents, as well as source and target Data Servers.


Figure 1 – Management Pack for Oracle Data Integrator - Dashboard

Through an easy-to-use graphical interface, administrators and operators are able to quickly asses the status of their Oracle Data Integrator environments. The Dashboard page provides a summary of the health of each Oracle Data Integrator components, and clearly highlights potential issues with links for drill-down such that users can easily further investigate and assess more details.


Figure 2 – Management Pack for Oracle Data Integrator– Repositories Page

The Repositories page clearly offers key database statistics and performance metrics, where users can easily track the growth of the repository tablespace and purge the Oracle Data Integrator logs when needed – one of Oracle Data Integrator’s best practices!


Figure 3 – Management Pack for Oracle Data Integrator– Database Execution Details Page

The Load plan Executions/Sessions page allows developers and operators to view the entire Oracle Data Integrator Sessions activities and review detailed execution statistics; such as the overall duration of processes, or the number of inserts, updates, deletes performed. Users can click to access database details on Oracle, or to view a relevant entire SQL statement generated by Oracle Data Integrator.


Figure 4 - Management Pack for Oracle Data Integrator– Agents Details Page

The Agents Page provides the list of Oracle Data Integrator Agents and the ability to drill down into each one. Recently triggered alerts or warnings can also be viewed. Overall metrics for agent performance, with quick links to session details are provided.

Configuration Management

To ensure that the configurations of all critical Oracle Data Integrator components in production environments are consistent with staging or test environments, Management Pack for Oracle Data Integrator provides Configuration Snapshots to save working configurations into the Management Repository or into an external XML file. These snapshots can then be compared with the active configuration in staging or test environments, or against any historical snapshot. Configuration Comparison is the next logical step which thus allows you to ensure the consistency of configurations in your application environment. Going one step further, Configuration History keeps track of all configuration changes, which can help in diagnosing performance problems that may be related to configuration changes along the way.

Service Level Management

Management Pack for Oracle Data Integrator allows for a proactive method to monitor Oracle Data Integrator environments at all levels. Many out of the box metrics are automatically collected for each Oracle Data Integrator component (repositories, agents, sources, targets…) which then provides users the ability to set up alerts based on critical or warning thresholds. Notifications can be provided to users in various ways through emails or Simple Network Management Protocol (SNMP) options.

Historical Analysis and Reporting

While real-time metrics certainly come in handy, there is also something to be said for historical collection of data! Management Pack for Oracle Data Integrator stores collected metrics and configurations data in a central repository –allowing users to thus analyze the metrics in various incremental views that include; last 24 hours, last 7 days, last 31 days. In addition, customizable service and system dashboard functionality allows users to create reports on various services and systems for service level availability, usage, performance, and business indicators – all in an effort to facilitate strategic trend analysis and reporting.

In a nutshell, Oracle Management Pack for Oracle Data Integrator benefits include: reducing the costs associated with monitoring the overall Oracle Data Integrator infrastructure and providing visibility into complex Oracle Data Integrator deployments across the enterprise, minimizing troubleshooting and performance tuning efforts as well as improving the overall ability to efficiently monitor Service Level Agreement compliance using powerful alerting capabilities.

Oracle Management Pack for Oracle Data Integrator is a new Management Pack, licensed separately. From an installation perspective it is packaged in the Fusion Middleware plug-ins for Enterprise Manager (version 12.1.0.4 and higher) and comes with the base Enterprise Manager 12c download. There is no separate installation file to download.

Try it out! Getting Started with Oracle Fusion Middleware Management Plug-in

Thursday Nov 07, 2013

Time to Get Started with Oracle Data Integrator 12c!

It is time to get started with Oracle Data Integrator 12c!

We would like to highlight for you a great place to begin your journey with ODI.  Here you will find the Getting Started section for ODI, which provides a few options.



Step 1 –

Start by downloading the Getting Started (PDF). The document provides general background information and detailed examples to help you learn how to use Oracle Data Integrator. This document guides you through a first project with Oracle Data Integrator 12c, and the instructions in this document are required for using the Getting Started Demonstration Environment.

If you would like to run the Getting Started Demonstration Environment on your system go to Step 2 A.

If you would like to run the Getting Started Demonstration Environment installed and pre-configured in a Virtual Image, go to Step 2 B.

Step 2 -

A-   A -  If you would like to run the Getting Started Demo Environment on your system, you will want to then download the Demo Environment for Getting Started (ZIP) archive containing ODI metadata, configuration scripts and sample data.


B-   B -  Alternatively, a pre-configured virtual machine is available with the Getting Started installation and configuration. The Virtual Machine platform uses Oracle Virtual Box technology, and use of this configuration would necessitate the download/installation of Virtual Box and the Getting Started virtual machine. If you prefer this route and would rather run the Getting Started Demonstration Environment installed and preconfigured in a Virtual Machine, please download the ODI 12c Getting Started Virtual Machine.