Thursday May 16, 2013

Sabre Holdings Case Study Webcast Recap

Last week at Oracle we had a very important event. In addition to the visit by the roaming Gnome, who really enjoyed posing for pictures on our campus, I had the priviledge to host a webcast with guest speaker Amjad Saeed from Sabre Holdings. We focused on Sabre's data integration solution  leveraging Oracle GoldenGate and Oracle Data Integrator for their enterprise travel data warehouse (ETDW).

Amjad, who leads the development effort for Sabre's enterprise data warehouse, presented us how they approached various data integration challenges, such as growing number of sources and data volumes, and what results they were able to achieve. He shared with us how using Oracle's data integration products in heterogeneous environments enabled right-time market insights, reduced complexity, and decreased time to market by 40%. Sabre was also able to standardize development for its global DW development team, achieve real-time view in the execution of the integration processes, and the ability to manage the data warehouse & BI performance on demand. I would like to thank Amjad again very much for taking the time to share their data integration best practices with us on this webcast.

In this webcast my colleague Sandrine Riley and I provided an overview of Oracle Data Integration products' differentiators. We explained architectural strengths that deliver a complete and integrated platform that offers high performance, fast time-to-value, and low cost of ownership.  If you have not had a chance to attend the live event we have the webcast now available on demand via this link for you to watch at your convenience:

Webcast Replay: Sabre Holdings Case Study: Accelerating Innovation using Oracle Data Integration

There were many great questions from our audience. Unfortunately we did not have enough time to respond to all of them. While we are individually following up with the attendees, I also want to post the questions and answer for some of the commonly asked questions here.

    Question: How do I learn Oracle Data Integrator or GoldenGate? Is training the only option ?

    Answer: We highly recommend training through Oracle University. The courses will cover all the foundational components needed to get up and running with ODI or GoldenGate. Oracle University offers instructor-led and online trainings. You can go to to get a complete listing of the courses available.Additionally – but not in replacement of training – you can get started with a guided ‘getting started’ type tutorial which you can find on our OTN page:  in the ‘Getting Started’ section of the page.Also, there are some helpful ‘Oracle by Example’ exercises/videos which you can find on the same page.

    For Oracle GoldenGate, we recommend watching instructional videos on its Youtube channel: Youtube/oraclegoldengate. A good example is here

    Last but not least, at Oracle OpenWorld there are opportunities to learn in depth by attending our hands-on-labs, even though it does not compare to/replace taking training.

    Question: Compare and contrast Oracle Data Integrator to the Oracle Warehouse Builder ETL process. Is ODI repository-driven and based on creation of "maps" when   creating ETL modules?

    Answer: ODI has been built from the ground up to be heterogeneous – so it will excel on both Oracle and non-Oracle platforms.  OWB has been a more Oracle centric product.  ODI mappings are developed with a declarative design based approach, and processes are executed via an agent – who is orchestrating and delegating the workload in a set-based manner to the database.  OWB deploys packages of code on the database and produces more procedural code.   For more details – please read our ODI architecture white paper

    Question:  Is the metadata navigator for ODI flexible and comprehensive so that it could be used as a documentation tool for the ETL?

    Answer: Oracle Data Integrator's metadata navigator has been renamed – now called ODI Console.  The ODI console is a web-based interface to view in a more picture based manner what is inside of the ODI repository.  It could be used as documentation.  Beyond the ODI console, ODI provides full documentation from the ODI Studio.  For any given process, project, etc.  you are able to right click – and there is an option that says ‘print’ – and this will provide you with a PDF document including the details down to the transformations.  These documents may be a more appropriate method of documentation.  Also – please check out a whitepaper on Managing Metadata with ODI.

If you would like to learn more about Oracle Data Integration products please check out our free resources here. I also would like to remind you to follow us on social media if you do not already. You can find us on Facebook, Twitter, YouTube, and Linkedin.

Wednesday May 01, 2013

When The Roaming Gnome Conquers Data Integration

It is always fascinating to see how our customers turn Oracle Data Integration products into a major force for their critical initiatives. I particularly like the success stories that tie back to the products or services that I use in my personal life. A little gnome that travels around the world is my new hero when it comes to seeing Oracle Data Integration in action, in day-to-day life.  Well, of course, I am referring to the Travelocity Gnome that we are familiar with from TV ads. And we know that behind this little gnome, is a great innovative IT team serving Sabre Holdings, which owns Travelocity. They deserve the praise for supporting business innovation with cutting edge data warehousing/BI solutions.  

the Traveling Gnome in the office (Photo thanks to Flickr user Ian Kershaw, available under by-nc-sa v2.0)

the Traveling Gnome in the office (Photo thanks to Flickr user Ian Kershaw, available under by-nc-sa v2.0)

Sabre Holdings demonstrated its ability to excel in implementing data integration solutions by winning Oracle Excellence Awards for Fusion Middleware Innovation in Data Integration category in 2011. We now have the privilege to hear directly from Sabre how they used Oracle Data Integrator and Oracle GoldenGate for their critical enterprise data warehouse that drives all kinds of innovative products and services for Sabre employees, partners, and customers. Sabre partnered with Oracle to achieve major improvements from reducing complexity,  better handling growing data volumes and decreasing time to market by 40%.

Next week on May 8th we will host a free webcast where you can hear Sabre's Amjad Saeed, who manages development for their enterprise data warehouse, present how they leveraged advanced data integration approaches in achieving their data warehouse solution goals.

Sabre Holdings Case Study: Accelerating Innovation with Oracle Data Integration

May 8th 1pm ET/ 10am PT

If you have not seen Oracle Data Integration in action, this is a must-see event to attend. I also would like to remind you that this year's  Oracle Excellence Awards for Oracle Fusion Middleware Innovation is open for submissions. You can submit your nomination by June 18th here.

Thursday Apr 11, 2013

Why Real Time?

Continuing on the five key data integration requirements topic, this time we focus on real-time data for decision making. 

[Read More]

Tuesday Apr 02, 2013

Starwood Hotels Presents in Customer Success Forum on April 11th

Register before April 5th for the online Customer Success Forum with Starwood Hotels and Resorts (April 11th 11am ET), and ask questions about how they used Oracle Exadata, Oracle GoldenGate, and Oracle Data Integrator for data warehousing and operational reporting.[Read More]

Thursday Mar 28, 2013

More Complete, Open, Real Time.

If you have been following our blogs you know about the strength of our data integration products, and our passion about helping our customers achieve their information management goals. Our customer testimonials in all kinds of solutions have proven our leadership in both data integration and data availability spaces. We decided to put our long story in a short video and incorporate examples of impressive benefits our customers achieve every day.

Here is the short video we came up with about Oracle Data Integration product family. Please take a look and let me know what you think.

To learn more about Oracle Data Integration products please download our free resources.

Wednesday Mar 27, 2013

On Premise and Cloud

Follow-on blog about five key data integration requirements topic, which focuses on data integration for cloud architectures. 

[Read More]

Friday Mar 15, 2013

Pervasive Access to Any Data

In my previous blog, I shared with you the five key data integration requirements, which can be summarized as: integrating any data from any source, stored on premise or in the cloud, with maximum performance and availability, to achieve 24/7 access to timely and trusted information. Today, I want to focus on the requirement for integrating “any data”.

We all feel the impact of huge growth in the amount of raw data collected on a daily basis. And big data is a popular topic of information technology these days. Highly complex, large volumes of data bring opportunities and challenges to IT and business audiences. The opportunities, as discussed in McKinsey’s report, are vast, and companies are ready to tap into big data to differentiate and innovate in today’s competitive world.

One of the key challenges of big data is managing the unstructured data, which is estimated to be %80 of enterprise data. Structured and unstructured data must coexist and be used in conjunction with each other in order to gain maximum insight. This means, organizations must collect, organize, and analyze data from sensors, conversations, e-commerce websites, social networks, and many other sources.

Big data also changes the perspective into information management. It changes the question from “How do you look at your data?” to “How do you look at the data that is relevant to you?” This shift in perspective has huge implications in terms of information-management best practices and technologies applied. Data integration technologies now need to support unstructured and semi-structured data, in addition to structured transactional data, to be able to support a complete picture of the enterprise that will drive higher efficiencies, productivity and innovation.

Oracle addresses big data requirements with a complete solution.

In addition to Oracle Big Data Appliance for acquiring and organizing big data, Oracle offers Oracle Big Data Connectors that enable an integrated data set for analysis. Big Data Connectors is a software suite that integrates big data across the enterprise. Oracle Data Integrator offers an Application Adapter for Hadoop, which is part of the Big Data Connectors, and allows organizations to build Hadoop metadata within Oracle Data Integrator, load data into Hadoop, transform data within Hadoop, and load data directly into Oracle Database using Oracle Loader for Hadoop. Oracle Data Integrator has the necessary capabilities for integrating structured, semi-structured, and unstructured data to support organizations with transforming any type of data into real value.

If you would like to learn more about how to use Oracle’s data integration offering for your big data initiatives take a look at our resources on Bridging the Big Data Divide with Oracle Data Integration.

Thursday Mar 07, 2013

5 New Data Integration Requirements for Today’s Data-Driven Organizations

How are the latest technology trends affecting data integration requirements? Read about the 5 key data integration requirements and Oracle's Data Integration product strategy in meeting these requirements.[Read More]

Monday Feb 25, 2013

Connecting Velocity to Value: Introducing Oracle Fast Data

To understand fast data, one must first look at one of the most compelling new the breakthroughs in data management: big data. Big data solutions address the challenge today’s businesses are facing when it comes to managing the increasing volume, velocity, variety of all data - not just data within as well as about the organization. Much of the buzz and to-do around big data has been around Hadoop, NoSQL technologies, but little has been talked about velocity. Velocity is about the speed this data is generating. In many cases the economic value of this data diminishes fast as well. As a result, companies need to process large volumes of data in real-time and make decisions in a more rapid fashion to create value from highly-perishable, high-volumes of data in business operations.

This is where fast data comes in. Fast data solutions help manage the velocity (and scale) of any type of data and any type of event to enable precise action for real-time results.

Fast data solutions come from multiple technologies, and some of the concepts, such as complex event processing and business activity monitoring, have been in use in areas such as the financial services industry for years. But often, the pieces were used in isolation—a complex event process engine as a standalone application to apply predefined business rules to filter data, for example. But when these concepts are tied to analytics, capabilities expand to allow improved real-time insights. By tying together these strands, companies can filter/correlate, move/transform, analyze, and finally act on information from big data sources quickly and efficiently, enabling both real-time analysis and further business intelligence work once the information is stored.

Oracle’s Fast Data solutions offer multiple technologies that work hand-in-hand to create value out of high-velocity, high-volume data. They are designed to optimize the efficiency, scale for processing high volume events and transactions.

[Read More]

Friday Jan 11, 2013

Customer Spotlight:Raymond James

Back in October we announced that Raymond James is a winner of 2012 Fusion Middleware Innovation Awards in the Data Integration category. As I mentioned on this site before, Raymond James used Oracle Data Integrator and Oracle GoldenGate in an integrated solution for its operational data store that supports the application services across the enterprise. By using Oracle's real-time data integration solution from heterogeneous systems into a centralized ODS, the team achieved higher application performance, reduced complexity and cost, and up to 50 percent faster time to market. 

I had a pleasure meeting the team, Ryan Fornett, Tim Garrod, and Michael Wells, from Raymond James at Oracle OpenWorld 2012. In addition to hearing from them during their presentations, we had a chance to record interviews with Tim and Ryan while at OpenWorld. You can find the links to their videos below:

 Yesterday, with their permission we posted a detailed blog on the FMW blog site about their implementation:

Data Integration Customer Spotlight: Raymond James Financial

I highly recommend reading it to see Oracle Data Integration products in action.

Many thanks to Ryan, Tim, and Mike for sharing the details of their innovative solution with us.

Friday Dec 28, 2012

ODI - Reverse Engineering Hive Tables

ODI can reverse engineer Hive tables via the standard reverse engineer and also an RKM to reverse engineer tables defined in Hive, this makes it very easy to capture table designs in ODI from Hive for integrating. To illustrate I will use the movie lens data set which is a common data set used in Hadoop training.

I have defined 2 tables in Hive for movies and their ratings as below, one file has fields delimited with '|' the other is tab delimited. 

  1. create table movies (movie_id int, movie_name string, release_date string, vid_release_date string,imdb_url string) row format delimited fields terminated by '|';
  2. create table movie_ratings (user_id string, movie_id string, rating float, tmstmp string) row format delimited fields terminated by '\t';

For this example I have loaded the Hive tables manually from my local filesystem (into Hive/HDFS) using the following LOAD DATA Hive commands and the movie lens data set mentioned earlier; 

  1. load data local inpath '/home/oracle/data/u.item' OVERWRITE INTO TABLE movies;
  2. load data local inpath '/home/oracle/data/' OVERWRITE INTO TABLE movie_ratings;

The data set in the file u.item data file looks like the following with '|' delimiter;

  • 1|Toy Story (1995)|01-Jan-1995|||0|0|0|1|1|1|0|0|0|0|0|0|0|0|0|0|0|0|0
  • 2|GoldenEye (1995)|01-Jan-1995|||0|1|1|0|0|0|0|0|0|0|0|0|0|0|0|0|1|0|0
  • 3|Four Rooms (1995)|01-Jan-1995|||0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|1|0|0

In ODI I can define my Hive data server and logical schema, here is the JDBC connection for my Hive database (I just used the default);

I can then define my model and perform a selective reverse using standard ODI functionality, below I am reversing just the movies table and the movie ratings table;


After the reverse is complete, the tables will appear in the model in the tree, the data can be inspected just like regular datastores;

From here we see the data in the regular data view;

The ODI RKM for Hive performs logging that is useful in debugging if you hit issues with the reverse engineer. This is a very basic example of how some of the capabilities hang together, ODI can also be used to design the load of the file into Hive, transformations within it and subsequent loads using Oracle Loader for Hadoop into Oracle and on and on.

Monday Dec 10, 2012

Maximum Availability with Oracle GoldenGate

Oracle Database offers a variety of built-in and optional products for maximum availability, and it is well known for its robust high availability and disaster recovery solutions. With its heterogeneous, real-time, transactional data movement capabilities, Oracle GoldenGate is a key part of the Maximum Availability Architecture for Oracle Database.

This week on Thursday Dec. 13th we will be presenting in a live webcast how Oracle GoldenGate fits into Oracle Database Maximum Availability Architecture (MAA). Joe Meeks from the Oracle Database High Availability team will discuss how Oracle GoldenGate complements other key products within MAA such as Active Data Guard. Nick Wagner from GoldenGate PM team will present how to upgrade to latest Oracle Database release without any downtime. Nick will also cover 2 new features of  Oracle GoldenGate 11gR2:  Integrated Capture for Oracle Database and Automated Conflict Detection and Resolution. Nick will provide in depth review of these new features with examples.

Oracle GoldenGate also offers maximum availability for non-Oracle databases, such as HP NonStop, SQL Server, DB2 (LUW, iSeries, or zSeries) and more. The same robust, reliable real-time, bidirectional data movement capabilities apply to all supported databases.

 I'd like to invite you to join us on Thursday Dec. 13th 10am PT/1pm ET to hear from the product experts on how to use GoldenGate for maximizing database availability and to ask your questions. You can find the registration link below.

Webcast: Maximum Availability with Oracle GoldenGate

Thursday Dec. 13th 10am PT/1pm ET

Look forward to another great webcast with lots of interaction with the audience.

ODI 11g - Dynamic and Flexible Code Generation

ODI supports conditional branching at execution time in its code generation framework. This is a little used, little known, but very powerful capability - this let's one piece of template code behave dynamically based on a runtime variable's value for example. Generally knowledge module's are free of any variable dependency. Using variable's within a knowledge module for this kind of dynamic capability is a valid use case - definitely in the highly specialized area.

The example I will illustrate is much simpler - how to define a filter (based on mapping here) that may or may not be included depending on whether at runtime a certain value is defined for a variable. I define a variable V_COND, if I set this variable's value to 1, then I will include the filter condition 'EMP.SAL > 1' otherwise I will just use '1=1' as the filter condition. I use ODIs substitution tags using a special tag '<$' which is processed just prior to execution in the runtime code - so this code is included in the ODI scenario code and it is processed after variables are substituted (unlike the '<?' tag).

 So the lines below are not equal ...

  • <$ if ( "#V_COND".equals("1")  ) { $> EMP.SAL > 1 <$ } else { $> 1 = 1 <$ } $>
  • <? if ( "#V_COND".equals("1")  ) { ?> EMP.SAL > 1 <? } else { ?> 1 = 1 <? } ?>

When the <? code is evaluated the code is executed without variable substitution - so we do not get the desired semantics, must use the <$ code. You can see the jython (java) code in red is the conditional if statement that drives whether the 'EMP.SAL > 1' or '1=1' is included in the generated code. For this illustration you need at least the ODI release - with the vanilla release it didn't work for me (may be patches?). As I mentioned, normally KMs don't have dependencies on variables - since any users must then have these variables defined etc. but it does afford a lot of runtime flexibility if such capabilities are required - something to keep in mind, definitely.

Thursday Dec 06, 2012

ODI 11g - Cleaning control characters and User Functions

In ODI user functions have a poor name really, they should be user expressions - a way of wrapping common expressions that you may wish to reuse many times - across many different technologies is an added bonus. To illustrate look at the problem of how to remove control characters from text. Users ask these types of questions over all technologies - Microsoft SQL Server, Oracle, DB2 and for many years - how do I clean a string, how do I tokenize a string and so on. After some searching around you will find a few ways of doing this, in Oracle there is a convenient way of using the TRANSLATE and REPLACE functions. So you can convert some text using the following SQL;

  • replace( translate('This is my string'||chr(9)||' which has a control character', chr(3)||chr(4)||chr(5)||chr(9), chr(3) ), chr(3), '' )

If you had many columns to perform this kind of transformation on, in the Oracle database the natural solution you'd go to would be to code this as a PLSQL function since you don't want the code splattered everywhere. Someone tells you that there is another control character that needs added equals a maintenance headache. Coding it as a PLSQL function will incur a context switch between SQL and PLSQL which could prove costly.

In ODI user functions let you capture this expression text and reference it many times across your mappings. This will protect the expression from being copy-pasted by developers and make maintenance much simpler - change the expression definition in one place.

Firstly define a name and a syntax for the user function, I am calling it UF_STRIP_BAD_CHARACTERS and it has one parameter an input string; 

We then can define an implementation for each technology we will use it, I will define Oracle's using the inputString parameter and the TRANSLATE and REPLACE functions with whatever control characters I want to replace;

I can then use this inside mapping expressions in ODI, below I am cleaning the ENAME column - a fabricated example but you get the gist.

 Note when I use the user function the function name remains in the text of the mapping, the actual expression is not substituted until I generate the scenario. If you generate the scenario and export the scenario you can have a peak at the code that is processed in the runtime - below you can see a snippet of my export scenario;

 That's all for now, hopefully a useful snippet of info.

Monday Dec 03, 2012

ODI 11g - Scripting a Reverse Engineer

A common question is related to how to script the reverse engineer using the ODI SDK. This follows on from some of my posts on scripting in general and accelerated model and topology setup.

Check out this viewlet here to see how to define a reverse engineering process using ODI's package.

Using the ODI SDK, you can script this up using the OdiPackage and StepOdiCommand classes as follows;

  1.  OdiPackage pkg = new OdiPackage(folder, "Pkg_Rev"+modName);
  2.   StepOdiCommand step1 = new StepOdiCommand(pkg,"step1_cmd_reset");
  3.   step1.setCommandExpression(new Expression("OdiReverseResetTable \"-MODEL="+mod.getModelId()+"\"",null, Expression.SqlGroupType.NONE));
  4.   StepOdiCommand step2 = new StepOdiCommand(pkg,"step2_cmd_reset");
  5.   step2.setCommandExpression(new Expression("OdiReverseGetMetaData \"-MODEL="+mod.getModelId()+"\"",null, Expression.SqlGroupType.NONE));
  6.   StepOdiCommand step3 = new StepOdiCommand(pkg,"step3_cmd_reset");
  7.   step3.setCommandExpression(new Expression("OdiReverseSetMetaData \"-MODEL="+mod.getModelId()+"\"",null, Expression.SqlGroupType.NONE));
  8.   pkg.setFirstStep(step1);
  9.   step1.setNextStepAfterSuccess(step2);
  10.   step2.setNextStepAfterSuccess(step3);

The biggest leap of faith for users is getting to know which SDK classes have to be used to build the objects in the design, using StepOdiCommand isn't necessarily obvious, once you see it in action though it is very simple to use. The above snippet uses an OdiModel variable named mod, its a snippet I added to the accelerated model creation script in the post linked above.


Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality


« April 2014