What's New in ODI 11g? - Part 1: Architecture
By FX on Aug 29, 2010
Oracle Data Integrator 11gR1 includes a large number of features and enhancements that make ODI even better than 10g. In this blog series I will explain the major directions taken by the product and give a quick overview of these features and enhancements.
For a detailed list of features, the following documentation link can be used as a reference.
There are major areas of changes in this release:
- New Architecture for Enterprise-Scale Deployment
- New Design-Time Experience
- New Run-Time Experience
- Core Enhancements to E-LT and Declarative Design
Enterprise Scale Architecture
The new architecture of ODI targets Enterprise-Scale deployments. Enterprise-scale means very high standards in terms of uptime availability, security, and horizontal scalability with load balancing.
Data Scalability at run-time was already ensured by the ODI E-LT architecture. In short, ODI E-LT scales linearly with your source/target data storage systems since it uses those CPUs for the data transformation processing. Yet, scaling the agents horizontally with load balancing was something not fully bullet-proof (some ODI 10g built-in load balancing was available, but without full fall-back capabilities).
High-Availability could be handled in 10g by mixing load balancing and by starting the agent Java processes as windows services.
Security was handled internally, and with some LDAP integration, but it did not encompass all ways that enterprises wanted to integrate with LDAP authentication procedures, including centralized security systems and single sign-on (SSO).
Java EE vs. Standalone Agents
In 11g, the agents can optionally be installed as applications in a Oracle WebLogic Server, and automatically benefit from the clustering, load-balancing, datasources and connection pooling features available with the application server. They are by default scalable and highly available.
Wait ! Did I say that the agent was deployed as a WLS application? Yes, but I said "optionally". The agent still (and will continue to) exists as a standalone process that can be deployed on top of a Java Machine and run on any system that has Java implementation.
Why? In some cases, the agent needs to be located close to the data, or on systems where WLS is not a viable option. In such situation, the architecture (and the performance) rely on this low footprint piece of code call the Agent.
Now, when speaking about ODI agents, we have to bear in mind that there are Standalone Agents (like in 10g) and Java EE Agents (in WLS). Well, they are (almost) the same piece of code. The main difference is where/how you install them (in WLS or on top of a JVM) and the features that you benefit from this installation.
- The Standalone agent is easier to deploy anywhere, but does not have clustering or connection pooling. It can rely on Oracle Process Manager and Notification Server (OPMN) for being protected and monitored like a service and it can use the built-in ODI Load Balancing feature.
- The Java EE agent is slightly more complex to set up (you need to install WLS first, set up a domain, and so forth), but gives you access to a different world in terms of enterprise-scale deployment: clustering, load-balancing, centralized monitoring and so forth.
Note that the choice between the two type of agents is really a user choice, and it is easy to mix both these type of agents seamlessly into an ODI architecture. (yes, you read that correctly, the standalone and JEE agents can be used together in the same network topology!)
Other Java EE Components
The other components that require to be deployed in an application server have been also modified.
ODI Console and EM Plugin
ODI has a great history with web interfaces, in which we can find successively Repository Explorer, Metadata Navigator and Lightweight Designer.
All these interfaces are now consolidated into a single ADF-Faces application called ODI Console. This console provides features for setting up and monitoring an ODI environment, and browse the design and run-time metadata. Figure 2: ODI Console, the ADF-Faces thin client for browsing ODI metadata.
Monitoring an enterprise-scale ODI infrastructure can be undertaken from the Oracle Fusion Middleware Control Console (a.k.a. Enterprise Manager - EM). ODI introduces a plugin that can be deployed in EM to monitor ODI resources (repositories, agents, ODI Console instances) and see their status, activities and notification. I will provide more details in the Part 3 of this series.
Web Services to start and monitor sessions are now merged in the (Java EE and Standalone) agent. Web Service requests can directly be sent to the agent, and do not require the installation of a separate service. A dedicated service (Public Web Service) exists for web service calls that do not involve an agent (e.g. listing contexts).
The Data Services feature remains the same, but no longer require to be deployed in an Axis web service stack. A Jax-WS Stack is sufficient.
Deployment & Configuration
When you start working with Java EE applications and application servers, the deployment and configuration effort is sometimes tremendous and requires a different set of skills. To help any user to set up an enterprise scale deployment (or to deploy any of the Java EE components), ODI provides a set of cool features, including:
- Pre-packaged WLS Templates for Java EE components, including configuration and datasources ready for deployment
- Java EE Agent Template Generation, to create templates for agents based on ODI topology information.
- Remote WLS Datasource Creation from the ODI IDE.
Your Own ODI Applications
Last component of the architecture: your own ODI applications !
ODI 11g provides a Java API to manipulate both the design-time and run-time artifacts of the product. This API allows you for example to create or modify interfaces programmaticaly, create your topology, perform import or export operations, launch or monitor sessions. This API can be used in any Java SE and Java EE applications, or in the context of Java-based scripting languages like Groovy or Jython.
Hey ! what about Security?
Good catch ! Enterprise-Scale deployments mandates support for enterprise security systems. ODI 11g introduces support for:
- External Password Storage, to have source/target data servers (and contexts) passwords stored in an enterprise credential store.
- External Authentication, to have user/password information stored in an enterprise identity store (e.g.: LDAP, Oracle Directory, Active Directory), and ODI authenticating against this store.
These two features let you optionally store critical information in dedicated storages and not within the ODI repository. The ODI Console may also use Oracle's single-sign on systems with ODI.
To learn more about the new ODI architecture, please review the following documents:
- The What's New in Oracle Data Integrator? Chapter in the ODI Developer's Guide.
- The Installation Guide for Oracle Data Integrator explain the installation process for Java EE and Standalone components.
- High Availability for Oracle Data Integrator chapter in the Fusion Middleware High Availability Guide.
The next part of this series for focus on the design-time experience and on the ODI Studio component.