Tuesday Apr 15, 2014

Fusion Application Bulk Import Process

Written by Ayush Ganeriwal, Oracle Data Integrator Product Management

In the previous blog post we looked at the Fusion Applications end-to-end bulk data integration use cases. Now let’s take a closer look at the Bulk Import process that transforms and moves data from Interface tables to internal tables. For this use case ODI is bundled along with Fusion Application and get configured transparently by the Fusion Application provisioning process. The entire process is automated and controlled through the Fusion Application User Interface. It also seeds the ODI repository with the Fusion Application specific Models, Interfaces, Procedures and Packages which are then dynamically modified through ODI SDK for any Fusion Application customizations.


Fusion Application Bulk Import Process

The above diagram shows the Bulk import process in Fusion Application where ODI is used for data transformation. Here the Interface tables are the source tables which were populated by other processes before the kicking off the Bulk Import process. The Fusion Application internal tables are the target for these integrations where the data needs to be loaded. These internal tables are directly used for Fusion Application functionalities therefore a number of data validations are applied to load only the good quality data into the internal tables. The data validation errors are monitored and corrected through Fusion Application User Interface. The metadata of Fusion Application tables is not fixed and gets modified as the Application is customized for customer’s requirement. Any change in such source or target tables would require corresponding adjustments in ODI artifacts too and is taken care of by the AppComposer which uses ODI SDK to make such changes in ODI artifacts. If auditing is enabled then any change in the internal table data or the changes in ODI artifacts are recorded in centralized auditing table.

Packaged ODI Artifacts

There are a large number of ODI models, interfaces and packages seeded in the default ODI repository used for Bulk Import. These ODI artifacts are built based upon the base metadata of Fusion Application schema.

Extensibility

As part of the customization, Fusion Application entities are added or modified as per the customer’s requirement. Such customizations result in changes in the underlying Fusion Application’s internal tables and interface tables, and require the ODI artifacts to be updated accordingly. The Fusion Application development team as built the extensibility framework to update ODI artifacts dynamically along with any change in Fusion Application schema. It leverages the ODI-SDK for performing any changes in the ODI repository. The dynamic generation of ODI artifacts is automatically kicked off as part of Patching and Upgrades process. Fusion Application AppComposer User Interface also supports explicitly triggering this process so that administrators can regenerate ODI artifacts whenever they make any customizations.

Validation Error Reporting

The validation errors are populated in intermediate tables and are exposed through BI Publisher so that admin users can correct and recycle these error records.

Auditing

The Fusion Application auditing framework keeps track of the changes performed by each of the users and at what time. There are two levels of auditing captured in Fusion Application audit table for Bulk Import use case. First, metadata changes in ODI artifacts through ODI SDK during customizations. Second, the transactional data changes in the Fusion Application table data as part of ODI interfaces execution. For these purposes the ODI team has exposed some substitution APIs that are used by Fusion Application development team to customize ODI KMs to perform such auditing during the actual data movement.

Provisioning and Upgrade

The provisioning process takes care of install and configuring ODI for the Fusion Application instance.

It takes care of automatically creating ODI repository schemas, configuring topology, setting up ODI agents, setup configurations for ODI –ESS bridge, seeding packaged ODI artifacts, apply modifications to seeded artifacts and create internal users in IDM for external authentication. There is a separate process to apply patches or upgrade the environment to the newer release. Such patching or upgrade processes not only take care of importing newer ODI artifacts but also kick off a CRM extensibility process that modifies ODI artifacts as per the Fusion Application customizations.

External Authentication

There is a dedicated IDM configured with each Fusion Application instance and all Fusion Application components are expected to have their users authenticated through this centralized IDM. For Bulk Import use case ODI is configured with external authentication and there are internal users created in IDM that are used for communication with ODI agent and kicking off ODI jobs.

Enterprise Scheduler Service (ESS) - ODI Bridge

The ODI scenarios are kicked off through ODI-ESS bridges. It is a separate library build for ODI-ESS integration and gets deployed along with Enterprise Scheduler Service (ESS) in Fusion Application environment. It supports both synchronous and asynchronous modes of invocation for ODI jobs. In the asynchronous mode the session status is updated to callbacks to the ESS services. There is a topology editor provided to manage the ESS callback service connectivity exclusively for Fusion Application use cases.

Note: Use of ESS-ODI Bridge is restricted to Fusion Application use case only at the moment.

High Availability

The ODI agent is deployed on Weblogic cluster in the Fusion Application environment to take advantage of ODI high availability capabilities. By default there is only one managed server in the Weblogic cluster created for ODI but as the load increases more managed servers can be added to the cluster to distribute execution of ODI sessions among ODI agent instances in the cluster.

Stay tuned for the last post on this topic coming soon.  This was part two in a series of three posts.  The initial post can be found here.

Friday Apr 11, 2014

Oracle Data Integrator (ODI) Usage in Fusion Applications (FA)

Written by Ayush Ganeriwal, Oracle Data Integrator Product Management

Oracle Data Integrator (ODI) is the bulk data transformation platform for Fusion Applications (FA). ODI is used by Fusion Customer Relationship Management (CRM), Fusion Human Capital Management (HCM), Fusion Supply Chain Management (SCM), Fusion Incentive Compensation (IC) and Fusion Financials family products and many other Fusion Application teams are following suit. Among all these product families CRM is the biggest consumer of ODI leveraging a breadth of ODI features and functionality, out of which some features were developed specifically for Fusion Applications use. Some ODI features they utilize include: ODI SDK, high availability, external authentication, various out of the box and customized Knowledge Modules, ODI-ESS bridge, callbacks to ESS EJBs, auditing, open tools, etc. In this post we will first talk about the different Fusion Application use cases at higher level and then take a closer look at different integration points.

Figure 1 shows data integration need of a typical on-premise Fusion Applications deployment.

  1. Bulk Import: Fusion Applications exposes a set of interface tables as the entry point for data load from any outside source. The bulk import process validates this data and loads it in the internal table which can then be used by the fusion application.
  2. Data Migration: Extracting data from external applications, legacy applications or any other data source and loading it into Fusion Application’s interface table. ODI can be used for such data load.
  3. Preparing Data Files: Converting data into Comma Separated Values (CSV) files that can be imported through Fusion Application’s files import wizard. ODI can be used to extract data into such CSV file.

Figure 1: Data Integration Needs in On-Premise Fusion Application

Figure 2 shows the on-demand or cloud environment requirements, which are slightly different as there is no direct connectivity available to the interface tables.

  1. Bulk Import: Fusion Application exposes a set of interface tables as the entry point for any data load from any outside source. The bulk import process validates this data and then loads it in the internal table which can then be used by the application.
  2. Preparing Data Files: Converting data into CSV files that can be imported through Fusion Application’s files import wizard. ODI can be used for creation on such CSV files.
  3. Uploading data files: The data files are uploaded to the Tenant File repository through either Fusion Application’s File import page or Oracle WebCenter Content Document Transfer Utility. The WebCenter Utility is built using ODI open tool framework allowing orchestrating entire process through ODI package.
  4. Loading Interface Table: Data files to be loaded in the interface tables so that it can be consumed by the Bulk Import process. ODI is used for loading these interface tables.

Figure 2: Data Integration Needs for On-Demand Fusion Application

Stay tuned for more blog posts on this topic coming next week. This was part one in a series of three posts.


Wednesday Sep 04, 2013

Extending the Value of Oracle Applications with Oracle Data Integration

Can you imagine a business operating without any application software? Not anymore. Transactional applications are in the center of business operations in today's digital world. With a wide range of offering, Oracle Applications bring customers the latest technology and make IT a strategic differentiator for their business. These applications’ data is also a major asset to the organization and critical to the health of the business. There are several ways Oracle's data integration solutions help Oracle Applications customers drive more value out of their application investments. 

A key method to maximize return on business applications is to connect their data to gain a unified view. A complete and up-to-date view enables businesses to make better business decisions and meet customers’ needs more efficiently. This is where Oracle Data Integration comes in. By connecting applications at the data-layer, at any latency, Oracle’s data integration products enable Oracle Applications users to find answers that takes their business to the next level. Oracle Data Integrator (ODI) , the flagship product for bulk data movement and transformation, has a long list of Knowledge Modules for Oracle Applications to leverage best practices in loading, transforming, or integrating data for that specific application. Oracle Data Integrator is also embedded in the Oracle BI Applications 11.1.1.7.1 release. With its tight integration with Oracle GoldenGate, ODI enables Oracle BI Applications users to leverage GoldenGate’s real-time data integration capabilities for performing analysis with up-to-date data.

In addition to removing data silos, IT organizations need to meet users’ expectations for high performance and continuous operations. With a single real-time data replication software platform, GoldenGate addresses both of these requirements. Oracle E-Business Suite, Oracle PeopleSoft Applications, JD Edwards, Siebel CRM, ATG Web Commerce customers can use GoldenGate to offload queries from production systems to reduce overhead and increase performance. In addition, GoldenGate is certified for enabling application upgrade for Siebel CRM, JD Edwards, and Communications Billing and Revenue Management without interrupting business operations. With bidirectional and heterogeneous real-time data replication GoldenGate also eliminates planned downtime for database, operating system, and hardware migration projects.

You can find out how Oracle Data Integration can help your Oracle Application deployment by visiting our new resource center: Powering Oracle Applications with Next-Generation Data Integration.

I also would like to invite you to learn more about how Oracle Fusion Middleware offers incremental value to Oracle Applications customers, which we call Oracle AppAdvantage, with innovative, best-of-breed solutions that simplify IT, provide a competitive edge, and enable innovation. You can read about Oracle AppAdvantage Story to discover how Oracle AppAdvantage helps align IT and business initiatives.

Friday Feb 10, 2012

Eliminating Batch Windows Using Real-Time Data Integration

When we invest in technology solutions we expect improvement in productivity, agility, performance and more. We don’t want to be limited by the technology we select. While data warehouses are designed to give us the freedom to access complete and reliable information, the underlying data integration architecture and the type of solution used can lead to significant constraints on how we manage our critical production systems.

With data warehousing solutions, one of the most common constraints is the time window available for batch extract processing on the source systems. The resource intensive extract process typically has to be done in off-business hours and restricts access to critical source systems.

A low-impact, real-time data integration solution can liberate your systems from batch windows. When the extract component uses a non-intrusive method, such as reading database transaction logs to capture only the changed data, it does not burden source systems. Hence, data extract can happen at any time of the day, and throughout the day, while all users are online.

SunGard is a great example for achieving a major transformation in data warehousing solutions by using log-based real-time data integration. The company removed batch processing timeline related constraint by using Oracle GoldenGate to capture all of the intraday changes that take place on the selected tables. As a result they reduced the nightly extract process from the Oracle E-Business Suite application by 9 hours. In addition, Oracle GoldenGate enables to feed EBusiness Suite data to the Oracle BI Application for Finance throughout the day. Hence, end users have access to up-to-date information on the Oracle Business Intelligence dashboards as changes occur.

To read more about how real-time data integration can free your critical systems from batch window constraints and how SunGard leveraged Oracle GoldenGate in their data warehousing implementation, I invite you to check out the SunGard case study and the article “Freedom from Batch Windows Using Real-Time Data Integration we wrote for TDWI's What Works in Emerging Technologies publication. As always, you can find more resources on Oracle GoldenGate on our recently redesigned website.

Wednesday Jul 27, 2011

Maximize the Return on your Oracle Applications with Oracle Data Integration

Application data is the lifeline of any enterprise.  Organizations rely heavily on their applications whether supply chain, HR, HCM, Payroll or any other kind to run their businesses.  Being able to integrate the data from one application into another is a powerful thing.  Siloed applications are useful, but for that particular silo.  Here's where data integration can help alleviate the situation where application data is disparate and separated.

Oracle Data Integration consists of Oracle Data Integrator and Oracle GoldenGate.  ODI's bulk data movement and data transformation features provide the 'heavy lifting' needed to make application data integration a success.  GoldenGate provides zero downtime upgrades as well as the capacity to offload reporting to a different database, that could be of lesser cost, size, etc --thereby making application data integration cost effective as well. Join us on July 28th at 10 AM PT/1PM ET to hear exactly how Oracle Data Integration can help increase the return on your Oracle Applications with data integration technology.  Register today.

About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
2
3
5
6
7
8
9
10
12
13
14
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today