Wednesday Aug 01, 2012

ODI 11g - Hadoop integration self study

There is a self study available at the link below which is a great introduction to the Hadoop related integration available in ODI 11.1.1.6 (see earlier blog here). Thanks to the curriculum development group for creating this material. You can see from the study how ODI was extended to support integration in and out of the Hadoop ecosystem.

https://apex.oracle.com/pls/apex/f?p=44785:24:0::NO:24:P24_CONTENT_ID,P24_PREV_PAGE:6130,29

The paper here titled 'High Performance Connectors for Load and Access of Data from Hadoop to Oracle  Database' which describes the raw capabilities in the Oracle Loader for Hadoop and Oracle Direct Connector for HDFS is encapsulated in the HDFS File/Hive to Oracle KM, so the different options for loading described in the paper are modeled as capabilities of the Knowledge Module. Another great illustration of the capabilities of KMs.

Much more to come in this space... 

Monday Jul 30, 2012

Four Ways to Wrestle a Big Data Elephant

He’s large. He’s fast. He’s strong. And very very hungry! Meet the big data elephant. Perhaps you have seen him stalking the corners of your data warehouse looking for some untapped data to devour? Or some unstructured weblogs to weigh in on. To wrestle the elephant to work for you rather than against you, we need data integration. But not just any kind, we need newer styles of data integration that are poised for these evolving types of data management challenges. I've put together four key requirements below with some pointers to industry experts in each category. Hopefully this is useful. And, good luck with that 8 and ¼ tons of data!

Four Ways to Wrestle a Big Data Elephant

  • Leverage existing tools and skill-sets
  • Quality first
  • Remember real-time
  • Integrate the platform

Leverage existing tools and skill-sets

While Hadoop technologies are cool to say, and can seem to add an impressive ‘buzz’ to your LinkedIn/Twitter profiles, a word of caution that not every big data technology may actually be necessary. The trend now is that tools are becoming ‘integrated’ in such a way that designing ETL and developing mapReduce can be implemented in a single design environment. Data Integration tools are evolving to support new forms of connectivity to source in NoSQL, HDFS. This is as opposed to keeping these two worlds separate. Something that I referred to recently in my blog on Bridging two Worlds: Big Data and Enterprise Data.

The advantages of a single solution allow you to address not only the complexities of mapping, accessing, and loading big data but also correlating your enterprise data – and this correlation may require integrating across mixed application environments. The correlation is key to taking full advantage of big data and requires a single unified tool that can straddle both environments.

Quality First

Secondly, big data sources consist of many different types and in many different forms. How can anyone be sure of the quality of that data? And yes, data stewardship best practices still do apply. In the big data scenario, data quality is important because of the multitude of data sources Multiple data sources make it difficult to trust the underlying data. Being able to quickly and easily identify and resolve any data discrepancies, missing values, etc in an automated fashion is beneficial to the applications and systems that use this information.

Remember real-time

I covered this very subject in last week’s blog on Is Big Data Just Super Sexy Batch. No it’s not. But at the same time, it would be an overstatement to say that big data addresses all of our real-time needs. [The cheetah still runs faster than the elephant… although I still wouldn’t want to try to outrun an elephant!]. Tools such as Oracle GoldenGate and techniques in real-time replication, change data capture don’t simply disappear with big data. In fact, the opposite will happen. They become even more crucial as our implementations cross over between unstructured and structured worlds where both performance, low-latency become increasingly paramount as volumes rise and velocity requirements

Integrate the platform

Taking all the miscellaneous technologies around big data – which are new to many organizations - and making them each work with one another is challenging. Making them work together in a production-grade environment is even more daunting. Integrated systems can help an organization radically simplify their big data architectures by integrating the necessary hardware and software components to provide fast and cost-efficient access, and mapping, to NoSQL and HDFS.

Combined hardware and software systems can be optimized for redundancy with mirrored disks, optimized for high availability with hot-swappable power, and optimized for scale by adding new racks with more memory and processing power. Take it one step further and you can use these same systems to build out more elastic capacity to meet the flexibility requirements big data demands.

To learn more about Oracle Data Integration products, see our website or to follow more conversations like this one join me on twitter @dainsworld.

Sunday Jul 22, 2012

Is Big Data just Super Sexy Batch?

One of the key expectations we have for big data and our information architecture is to yield faster, better and more insightful analytics. That appeal of processing so much information quickly is why the Hadoop technologies may have originally been invented. But is it nothing more than a super sexy batch? Yes – on sexy. But there’s definitely an important real-time element involved. Read the rest of the article to see more on our take on the intersection of batch, real-time, big data, and business analytics. [Read More]

Monday Apr 02, 2012

Why Oracle Data Integrator for Big Data?

Big Data is everywhere these days - but what exactly is it? It’s data that comes from a multitude of sources – not only structured data, but unstructured data as well.  The sheer volume of data is mindboggling – here are a few examples of big data: climate information collected from sensors, social media information, digital pictures, log files, online video files, medical records or online transaction records.  These are just a few examples of what constitutes big data.   Embedded in big data is tremendous value and being able to manipulate, load, transform and analyze big data is key to enhancing productivity and competitiveness. 

The value of big data lies in its propensity for greater in-depth analysis and data segmentation -- in turn giving companies detailed information on product performance, customer preferences and inventory.  Furthermore, by being able to store and create more data in digital form, “big data can unlock significant value by making information transparent and usable at much higher frequency." (McKinsey Global Institute, May 2011)

Oracle's flagship product for bulk data movement and transformation, Oracle Data Integrator, is a critical component of Oracle’s Big Data strategy. ODI provides automation, bulk loading, and validation and transformation capabilities for Big Data while minimizing the complexities of using Hadoop.  Specifically, the advantages of ODI in a Big Data scenario are due to pre-built Knowledge Modules that drive processing in Hadoop. This leverages the graphical UI to load and unload data from Hadoop, perform data validations and create mapping expressions for transformations.  The Knowledge Modules provide a key jump-start and eliminate a significant amount of Hadoop development. 

Using Oracle Data Integrator together with Oracle Big Data Connectors, you can simplify the complexities of mapping, accessing, and loading big data (via NoSQL or HDFS) but also correlating your enterprise data – this correlation may require integrating across heterogeneous and standards-based environments, connecting to Oracle Exadata, or sourcing via a big data platform such as Oracle Big Data Appliance.

To learn more about Oracle Data Integration and Big Data, download our resource kit to see the latest in whitepapers, webinars, downloads, and more… or go to our website on www.oracle.com/bigdata

Wednesday Mar 28, 2012

New Feature in ODI 11.1.1.6: ODI for Big Data

By Ananth Tirupattur

Starting with Oracle Data Integrator 11.1.1.6.0, ODI is offering a solution to process Big Data. This post provides an overview of this feature.

Before getting into the details of ODI for Big Data and with all the buzz around Big Data, I will provide a brief introduction to Big Data and Oracle Solution for Big Data.

[Read More]

Thursday Sep 29, 2011

Get hands-on with Oracle Data Integrator!

If you are visiting Oracle OpenWorld next week, you will have ample opportunity to see the latest Oracle Data Integrator live in action and work with it yourself. We will have the following hands-on labs in the Marriott Marquis - Salon 1/2:

  • Introduction to Oracle Data Integrator (Monday 2-3pm)
    See how easy it is to load and transform heterogeneous data from files, XML, and multiple databases.
  • Oracle Data Integrator and Oracle GoldenGate (Monday 3:30-4:30pm)
    Combine the best of both worlds to load real-time data into your Data Warehouse!
  • Data Integration with Oracle Business Intelligence (Thursday 10:30-11:30am)
    Do you know where your data is from? See the complete ODI data lineage from within OBIEE.

These hands-on labs fill up quickly ahead of time, so register yourself now to get a spot!

We also demo these use cases and more at our Demogrounds pods 253, 254, and 255 (highest 8-bit number, easy to remember...) at Moscone South, right side. At our pods you can see:

  • Loading a data warehouse with Oracle Data Integrator
  • Realtime data warehousing with Oracle Data Integrator and Oracle GoldenGate
  • Integrating Oracle SOA Suite with Oracle Data Integrator
  • Data Lineage from Oracle Data Integrator to Oracle BI
  • Loading Big Data into Exadata
  • Data Masking during Loads
  • Data Federation using Oracle Data Service Integrator
About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
2
3
5
6
7
8
9
10
12
13
14
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today