Tuesday Apr 15, 2014

Fusion Application Bulk Import Process

Written by Ayush Ganeriwal, Oracle Data Integrator Product Management

In the previous blog post we looked at the Fusion Applications end-to-end bulk data integration use cases. Now let’s take a closer look at the Bulk Import process that transforms and moves data from Interface tables to internal tables. For this use case ODI is bundled along with Fusion Application and get configured transparently by the Fusion Application provisioning process. The entire process is automated and controlled through the Fusion Application User Interface. It also seeds the ODI repository with the Fusion Application specific Models, Interfaces, Procedures and Packages which are then dynamically modified through ODI SDK for any Fusion Application customizations.


Fusion Application Bulk Import Process

The above diagram shows the Bulk import process in Fusion Application where ODI is used for data transformation. Here the Interface tables are the source tables which were populated by other processes before the kicking off the Bulk Import process. The Fusion Application internal tables are the target for these integrations where the data needs to be loaded. These internal tables are directly used for Fusion Application functionalities therefore a number of data validations are applied to load only the good quality data into the internal tables. The data validation errors are monitored and corrected through Fusion Application User Interface. The metadata of Fusion Application tables is not fixed and gets modified as the Application is customized for customer’s requirement. Any change in such source or target tables would require corresponding adjustments in ODI artifacts too and is taken care of by the AppComposer which uses ODI SDK to make such changes in ODI artifacts. If auditing is enabled then any change in the internal table data or the changes in ODI artifacts are recorded in centralized auditing table.

Packaged ODI Artifacts

There are a large number of ODI models, interfaces and packages seeded in the default ODI repository used for Bulk Import. These ODI artifacts are built based upon the base metadata of Fusion Application schema.

Extensibility

As part of the customization, Fusion Application entities are added or modified as per the customer’s requirement. Such customizations result in changes in the underlying Fusion Application’s internal tables and interface tables, and require the ODI artifacts to be updated accordingly. The Fusion Application development team as built the extensibility framework to update ODI artifacts dynamically along with any change in Fusion Application schema. It leverages the ODI-SDK for performing any changes in the ODI repository. The dynamic generation of ODI artifacts is automatically kicked off as part of Patching and Upgrades process. Fusion Application AppComposer User Interface also supports explicitly triggering this process so that administrators can regenerate ODI artifacts whenever they make any customizations.

Validation Error Reporting

The validation errors are populated in intermediate tables and are exposed through BI Publisher so that admin users can correct and recycle these error records.

Auditing

The Fusion Application auditing framework keeps track of the changes performed by each of the users and at what time. There are two levels of auditing captured in Fusion Application audit table for Bulk Import use case. First, metadata changes in ODI artifacts through ODI SDK during customizations. Second, the transactional data changes in the Fusion Application table data as part of ODI interfaces execution. For these purposes the ODI team has exposed some substitution APIs that are used by Fusion Application development team to customize ODI KMs to perform such auditing during the actual data movement.

Provisioning and Upgrade

The provisioning process takes care of install and configuring ODI for the Fusion Application instance.

It takes care of automatically creating ODI repository schemas, configuring topology, setting up ODI agents, setup configurations for ODI –ESS bridge, seeding packaged ODI artifacts, apply modifications to seeded artifacts and create internal users in IDM for external authentication. There is a separate process to apply patches or upgrade the environment to the newer release. Such patching or upgrade processes not only take care of importing newer ODI artifacts but also kick off a CRM extensibility process that modifies ODI artifacts as per the Fusion Application customizations.

External Authentication

There is a dedicated IDM configured with each Fusion Application instance and all Fusion Application components are expected to have their users authenticated through this centralized IDM. For Bulk Import use case ODI is configured with external authentication and there are internal users created in IDM that are used for communication with ODI agent and kicking off ODI jobs.

Enterprise Scheduler Service (ESS) - ODI Bridge

The ODI scenarios are kicked off through ODI-ESS bridges. It is a separate library build for ODI-ESS integration and gets deployed along with Enterprise Scheduler Service (ESS) in Fusion Application environment. It supports both synchronous and asynchronous modes of invocation for ODI jobs. In the asynchronous mode the session status is updated to callbacks to the ESS services. There is a topology editor provided to manage the ESS callback service connectivity exclusively for Fusion Application use cases.

Note: Use of ESS-ODI Bridge is restricted to Fusion Application use case only at the moment.

High Availability

The ODI agent is deployed on Weblogic cluster in the Fusion Application environment to take advantage of ODI high availability capabilities. By default there is only one managed server in the Weblogic cluster created for ODI but as the load increases more managed servers can be added to the cluster to distribute execution of ODI sessions among ODI agent instances in the cluster.

Stay tuned for the last post on this topic coming soon.  This was part two in a series of three posts.  The initial post can be found here.

Friday Apr 04, 2014

Turning Big Data into Real-Time Action for a Greater Customer Experience

The power shifted to us, consumers. The digital revolution allows us to access broader set of services, and communicate without boundaries. Today we demand more and better choices in a competitive market, putting pressures on businesses to catch up with our expectations.

By offering differentiated and improved experience to their customers organizations see that they can drive revenue growth via higher loyalty, and improved brand perception.  Because technology is a key enabler for delivering superb and consistent customer experience across all touchpoints, in recent years customer experience solutions have become a top priority for CIOs. Thanks to the availability of big data analytics, organizations can now analyze a broader variety of data, rather than a few basic data points, and gain deeper insight into their customers and operations. In turn, this deeper insight helps align their business to provide a seamless customer experience.

In our digital, fact-paced world we produce large volumes of data with unprecedented velocity. This data contains perishable value that requires fast capture, analysis, and action to be able to influence the operations or the interaction with the customer. Otherwise the insight or action may become irrelevant, which decreases the value for the customer and the organization significantly.  To extract the maximum value from highly dynamic and perishable data, you need to process much faster and take timely action. This is the main premise behind Oracle's Fast Data solutions, which we have discussed in previous blogs and webcasts.

Real-time data integration and analytics play a crucial role in our new world of big and fast data. Organizations that look into leveraging  big data to create greater customer experience, need to evaluate the analytical foundation behind their customer-facing systems and resulting interactions, and determine whether they can improve how and when they collect, analyze, and act on their ever-growing data assets.

In our next webcast my colleague Pete Schutt in the Oracle Business Analytics team and I will discuss how organizations can create value for their customers using real-time customer analytics, and how to leverage big data to build a solid business analytics foundation using the latest features of Oracle Data Integration and Oracle Business Analytics. We will provide multiple customer examples for different solution architectures.

Join us on April 15th 10am PT/ 1pm ET by registering via the link below.

Turning Big Data into Real-Time Action for a Greater Customer Experience

Tuesday, April 15th 10am PT/ 1pm ET

Until we meet at this webcast, please review my related article on this topic published on DBTA earlier this year:

Tuesday Apr 01, 2014

Looking for Cutting-Edge Data Integration: 2014 Excellence Awards

It is nomination time!!!

This year's Oracle Fusion Middleware Excellence Awards will honor customers and partners who are creatively using various products across Oracle Fusion Middleware. Think you have something unique and innovative with one or a few of our Oracle Data Integration products?

We would love to hear from you! Please submit today.

The deadline for the nomination is June 20, 2014.

What you win:

  • An Oracle Fusion Middleware Innovation trophy
  • One free pass to Oracle OpenWorld 2014
  • Priority consideration for placement in Profit magazine, Oracle Magazine, or other Oracle publications & press release
  • Oracle Fusion Middleware Innovation logo for inclusion on your own Website and/or press release

Let us reminisce a little…

For details on the 2013 Data Integration Winners:

Royal Bank of Scotland’s Market and International Banking and The Yalumba Wine Company, check out this blog post: 2013 Oracle Excellence Awards for Fusion Middleware Innovation… and the Winners for Data Integration are…

and for details on the 2012 Data Integration Winners:

Raymond James and Morrisons, check out this blog post: And the Winners of Fusion Middleware Innovation Awards in Data Integration are… 

Now to view the 2013 Winners (for all categories).

We hope to honor you!

Here's what you need to do: 

Click here to submit your nomination today.  And just a reminder: the deadline to submit a nomination is 5pm Pacific Time on June 20, 2014.

Thursday Mar 27, 2014

Interested in presenting and sharing your insights around Data Integration at OpenWorld?

You have been successful in making your organization run smoother. Faster. More cost-effectively. You have come up with the perfect solution to increase your staff retention, speed up your lead to sales pipeline, or minimize your supply management costs. We want to hear your story. Submit your proposal today and share your success at OpenWorld 2014.

Send us a proposal covering your Data Integration success. If it is selected, you will share your idea, experiences, and stories with Oracle customers, developers, and partners from around the world. You will also get a complimentary full pass to the conference.

Learn more about submitting your proposal.

Conference attendees want to hear it straight from you.

Don't wait—proposals must be submitted by April 15!

Tuesday Feb 18, 2014

Recap of Oracle GoldenGate 12c Webcast with Q&A

Simply amazing! That’s how I would summarize last week’s webcast for Oracle GoldenGate 12c.  It was a very interactive event with hundreds of live attendees and hundreds of great questions. In the presentation part my colleagues, Doug Reid and Joe deBuzna, went over the new features of Oracle GoldenGate 12c. They explained Oracle GoldenGate 12c key new features including:

  • Integrated Delivery for Oracle Database,
  • Coordinated Delivery for non-Oracle databases,
  • Support for Oracle Database 12c multitenant architecture,
  • Enhanced high availability via integration with Oracle Data Guard Fast-Start Failover,
  • Expanded heterogeneity, i.e. support for new databases and operating systems,
  • Improved security,
  • Low-downtime database migration solutions for Oracle E-Business Suite,
  • Integration with Oracle Coherence.

We also had a nice long and live Q&A section. In the previous Oracle GoldenGate webcasts, we could not respond to all audience questions in a 10-15 minute timeframe at the end of the presentation. This time we kept the presentation part short and left more than 30 minutes for Q&A. To our surprise, we could not answer even half of the questions we received. 

If you missed this great webcast discussing the new features of Oracle GoldenGate 12c,  and more than 30 minutes of Q&A with GoldenGate Product Management, you can still watch it on demand via the link below.

On Demand Webcast: Introducing Oracle GoldenGate 12c: Extreme Performance Simplified

On this blog post I would like to provide brief answers from our PM team  for some of the questions that we were not able to answer during the live webcast.

1) Does Oracle GoldenGate replicate DDL statements or DML for Oracle Database?

    Oracle GoldenGate replicates DML and DDL operations for Oracle Database and Teradata.

2) Where do we get more info on how to setup integration with Data Guard Fast-Start Failover (FSFO)?

     Please see the following blog posts or documents on My Oracle Support:

Best Practice - Oracle GoldenGate and Oracle Data Guard - Switchover/Fail-over Operations for GoldenGate    [My Oracle Support Article ID   1322547.1] 

Best Practice - Oracle GoldenGate 11gr2 integrated extract and Oracle Data Guard - Switchover/Fail-over Operations  [My Oracle Support Article ID 1436913.1] 

3) Does GoldenGate support SQL Server 2012 extraction? In the past only apply was supported.

Yes, starting with the new 12c release GoldenGate captures from SQL Server 2012 in addition to delivery capabilities.

4) Which RDBMS does GoldenGate 12c support?

GoldenGate supports all major RDBMS. For a full list of supported platforms please see Oracle GoldenGate certification matrix.

5) Could you provide some more details please on Integrated Delivery for dynamic parallel threads at Target side?

Please check out our white papers on Oracle GoldenGate 12c resource kit for more details on the new features, and how Oracle GoldenGate 12c works with Oracle Database. 

6) What is the best way to sync partial data (based on some selection criterion) from a table between databases?

 Please refer to the article: How To Resync A Single Table With Minimum Impact To Other Tables' Replication? [Article ID 966211.1]

7) How can GoldenGate be better than database trigger to push data into custom tables?

Triggers can cause high CPU overhead, in some cases almost double compared to reading from redo or transaction logs. In addition, they are intrusive to the application and cause management overhead as application changes. Oracle GoldenGate's log-based change data capture is not only low-impact in terms of CPU utilization, but also non-intrusive to the application with low maintenance requirements.

8) Are there any customers in the manufacturing industry using GoldenGate and for which application?

We have many references in manufacturing. In fact, SolarWorld USA was our guest speaker in the executive video webcast last November. You can watch the interview here. RIM Blackberry uses Oracle GoldenGate for multi-master replication between its global manufacturing systems. Here is another manufacturing customer story from AkzoNobel.

9) Does GoldenGate 12c support compressed objects for replication? Also does it supports BLOB/CLOB columns?

Yes, GoldenGate 12c and GoldenGate 11gR2 both support compressed objects. GoldenGate has been supporting BLOB/CLOB columns since version 10.

10) Is Oracle Database 11.2.0.4 mandatory to use GoldenGate 12c Integrated Delivery? Not earlier versions?

Yes. To use GoldenGate 12c’s Integrated Delivery, for the target environment Oracle Database 11.2.04 and above is required .

11) We have Oracle Streams implementation for more than 5 years. We would like to migrate to GoldenGate, however older version of GoldenGate were not supporting filtering individual transactions. Is it supported in GoldenGate 12c?

      Yes, it is supported in GoldenGate 12c.


In future blog posts I will continue to provide answers for common questions we received in the webcast. In the meanwhile I highly recommend watching the Introducing Oracle GoldenGate 12c: Extreme Performance Simplified webcast on demand.

Wednesday Jan 22, 2014

Deep Dive Into Oracle Data Integrator Changed Data Capture Leveraging Oracle GoldenGate

Check out this blog post below from Christophe – first the details related to Oracle Data Integrator’s (ODI) Journalizing Knowledge Modules (JKMs) as well as a deeper dive into the particulars around the seamless out-of-the-box integration between Oracle Data Integrator (ODI) and Oracle GoldenGate.

http://www.ateam-oracle.com/understanding-the-odi-jkms-and-how-they-work-with-oracle-goldengate/

Happy reading!

Thursday Dec 05, 2013

Are You Ready for the Future?

Have you wondered where you and your organization will be in the future as the information technology advances in an unprecedented way? Have you wondered whether your company, or team, is equipped with the right tools and strategy to be ahead of competition, as everything around us changes? If yes, you are not alone. If not, you should. 

When it comes to leveraging data and creating opportunities for innovation and growth, data integration is a critical technology that sits in the center of many key initiatives: big data analytics, cloud integration, modernization to name a few. Many industry leaders that excel and continue to innovate are the ones that keep an eye on how they collect and distribute their data.They know the value of turning data into an asset, rather than a cost driver.

With the new release of Oracle Data Integrator 12c and Oracle GoldenGate 12c, Oracle established itself as the leader in the data integration and replication space.  And how Oracle differentiates its offering is now highlighted in an interactive e-book: The Path to the Future.

In this e-book we told Oracle Data Integration product family's story through the words of our customers; including Starwood Hotels and Resorts, BT, Herbalife, SK Telecom, MediSwitch, Aria Systems, MegaFon and others.  The e-book reveals how Oracle Data Integration products help you stay ahead of the curve by integrating any data, anywhere, at the speed of thought, with maximum performance and availability,  and full trust in its high quality.  You will also find out how the latest 12c release helps you keep pace with the latest trends while improving developer productivity and performance. 

Take a look at our new interactive e-book and let us know what you think!


Thursday Nov 21, 2013

Recap of Oracle GoldenGate 12c and Oracle Data Integrator 12c Launch Webcast

Last week we had a great video webcast for Oracle GoldenGate 12c and Oracle Data Integrator 12c. Our development executives, Brad Adelberg and Tim Hall,  talked about the new features and how the new release helps with delivering future-ready data integration solutions with extreme performance and high IT productivity. 

If you missed the webcast you can watch it on demand via the following page:
On-Demand Video Webcast: Introducing 12c for Oracle Data Integration

In previous blogs we have talked about the new features of Oracle GoldenGate 12c and Oracle Data Integrator 12c, so I am not going to repeat them here. But worth repeating are what our customers and partners say about the new 12c release.

SolarWorld’s Senior Database Administrator, Russ Toyama was in the studio with our executives for the launch webcast, and Russ discussed their GoldenGate implementation for SolarWorld's manufacturing process. SolarWorld is the largest U.S. solar panel manufacturer for more than 35 years and needed operational intelligence to continuously improve the quality of its products, while ensuring the systems operate with high performance and stability.

SolarWorld uses GoldenGate to move data from multiple manufacturing databases into a single decision support system database in real time, freeing up the OLTP systems for transaction processing, which improves their performance and stability. The DSS database is very flexible in meeting reporting needs, and provides a comprehensive view of multiple manufacturing processes. This provides the traditional roles of reporting and engineering analysis to continuously improve product quality, yield and efficiency, and enables real time monitoring of the production manufacturing process. Using this real-time monitoring capability SolarWorld is able to detect the deviations from the norm right away, and take action to remedy or understand the situation. This not only improves production quality, but also improves cost management. The manufacturing process is a series of steps building upon the previous, so if there is an issue, it needs to be corrected as soon as possible to prevent waste and reduce manufacturing costs.

For GoldenGate 12c we heard from Surren Partabh, CTO for Technology Services for BT. Surren explained the role of Oracle GoldenGate for their private cloud initiative and how they have improved customer experience and availability, while managing costs as well.  Surren highlighted that Oracle's data integration product family is one of the cornerstones for BT"s cloud migration project and enable to migrate to cloud simply, and in an agile manner. BT used GoldenGate to build a replication hub to help with the migration from legacy systems to the cloud with a reliable fallback strategy. Surren also commented on the tight integration between GoldenGate 12c and Oracle Data Integrator 12c,  saying that it is a "step in the right direction" as it simplifies the actual installation, configuration, management, and monitoring of solutions. You can watch the interview with BT's Surren Partabh here.

In the launch webcast we also heard from Mark Rittman, CTO of Rittman Mead Consulting. Mark talked about the Oracle Data Integrator 12c new features in great depth, given that he was closely involved with the Beta testing program. Mark shared his opinion that the new flow-based design interface is the most critical feature of this 12c release and will bring major productivity gains for developers. He added that interoperability with Oracle Warehouse Builder for easier migration and tighter integration with Oracle GoldenGate are very valuable for customers. Mark also discussed the new release of Oracle BI Applications and how its use of Oracle Data Integrator for data movement and transformations simplifies life for Oracle BI Apps customers.  The complete interview with Mark Rittman is available for you as well.

If you missed the launch webcast last week, I hope you take the time to watch it on demand and discover how Oracle has changed the data integration and replication technology space with Oracle Data Integrator 12c and Oracle GoldenGate 12c. For more information including white papers and podcasts you can also download free resources.

Friday Nov 15, 2013

ODI 12c - OWB Migration, Integration Plus Components

Today is another big day for ODI and OWB! Get your hands on the much anticipated functionality covering everything from new components added into the ODI 12c mapping designer, to the ETL design migration utility (announced here) to move designs from OWB 11.2.0.4 into ODI 12c which all complement the integration and auditing of OWB jobs from ODI which was included in 12c.

The patches for ODI and OWB are great news on many fronts, not only in being a major point in the roadmap. As well as the OWB migration this is great for the ODI the product, the new components provide a lot of great capabilities (and will help with a lot of those 'how do I do this' questions).

Included in the ODI patch you'll find a component bundle including transformation components from pivot/unpivot through subquery filtering on to table functions and the like. All great examples of the product's extensibility - a fusion of declarative design and technology specific knowledge modules in harmony. Lots of good stuff that you will hear much more of.

There are multiple patches - if you are only interested in ODI features and enhancements then all you need is the ODI patch below;

  • ODI 12.1.2.0.0 installed (ODI 12.1.2.0.0 plus patch number 17053768)

If you want the OWB migration utility then you will need the following patch to OWB as well as the ODI patch above;

  • OWB 11.2.0.4 installed (OWB 11.2.0.4 plus patch number 17547241)

If you install the ODI patch you will find some new components in the palette, the components let you express many more transformations - these were transformations which could be designed in OWB, now in ODI as well as the components we have knowledge modules to support their code generation for heterogenous environments.

The ODI 12c release included a new ODI Tool and support for monitoring OWB jobs - jobs can be initiated using the tool OdiStartOwbJob tool which is available for use from within packages, procedures, KMs or command line. The jobs can be monitored from ODI, this includes OWB mappings and process flows. The patch provides the migration utility plus components.

In subsequent postings we will deep dive into these areas. 

Thursday Nov 07, 2013

Time to Get Started with Oracle Data Integrator 12c!

It is time to get started with Oracle Data Integrator 12c!

We would like to highlight for you a great place to begin your journey with ODI.  Here you will find the Getting Started section for ODI, which provides a few options.



Step 1 –

Start by downloading the Getting Started (PDF). The document provides general background information and detailed examples to help you learn how to use Oracle Data Integrator. This document guides you through a first project with Oracle Data Integrator 12c, and the instructions in this document are required for using the Getting Started Demonstration Environment.

If you would like to run the Getting Started Demonstration Environment on your system go to Step 2 A.

If you would like to run the Getting Started Demonstration Environment installed and pre-configured in a Virtual Image, go to Step 2 B.

Step 2 -

A-   A -  If you would like to run the Getting Started Demo Environment on your system, you will want to then download the Demo Environment for Getting Started (ZIP) archive containing ODI metadata, configuration scripts and sample data.


B-   B -  Alternatively, a pre-configured virtual machine is available with the Getting Started installation and configuration. The Virtual Machine platform uses Oracle Virtual Box technology, and use of this configuration would necessitate the download/installation of Virtual Box and the Getting Started virtual machine. If you prefer this route and would rather run the Getting Started Demonstration Environment installed and preconfigured in a Virtual Machine, please download the ODI 12c Getting Started Virtual Machine.