Wednesday Jan 21, 2009

Using Cisco ACE Series Hardware Load-Balancers with EBS 12

Editor's Note:  This certification is an unusual departure from our typical approach to load-balancer support for the E-Business Suite.  In general, we do not perform explicit certifications of specific load-balancers with the E-Business Suite.  It's not possible for us to keep up with the sheer number of different vendors and models available.

Our standard support policy is that standards-compliant load-balancers are assumed to be compatible with the E-Business Suite.  Load-balancers do not require formal testing by our Applications Technology Integration group to be supported for the E-Business Suite.  Our load-balancing-related recommendations are designed to be compatible with all generic load-balancers:

When time and resources permit, we occasionally partner with networking vendors to test unusual or notable new hardware with the E-Business Suite.  Networking vendors are welcome to drop me an email if you'd like to discuss joint projects.

As a rule, we do not recommend a specific vendor's products over any other vendors.  This article should not be considered an endorsement for this particular vendor or networking configuration.  As with all hardware,  customers should perform independent benchmark tests to verify a vendor's claims of improved performance or manageability in E-Business Suite environments.

Oracle and Cisco Systems jointly certified the Cisco ACE Series Application Control Engine with Oracle E-Business Suite Release 12.  Cisco ACE Series Application Control Engine is a new offering from Cisco Systems that has the potential to improve high availability, performance, security, optimized application deployment and lower the cost of ownership for Oracle E-Business Suite R12.  The new ACE platform targets three main aspects, namely manageability, security and performance.  The figure below shows the Virtualization and Role-based Admin features of the ACE module.

Architecture diagram showing Cisco ACE load balancer Virtual Device and RBA

[Read More]

Tuesday Nov 18, 2008

Advanced Deployment Architectures for Oracle E-Business Suite (OpenWorld 2008 Recap)

I'm (still) highlighting OpenWorld 2008 presentations that cover some of the most popular E-Business Suite technology stack topics. A catalog of all of the Applications Technology track sessions with links to the presentations is available here:

E-Business Suite sysadmins know that there are a lot of different ways to deploy their system in production.  You can split EBS services across multiple application tier and database tier server nodes, you can scale up with load-balancers and Real Application Clusters, you can integrate your E-Business Suite instance with optional external services like Oracle Single Sign-On and the Oracle SOA Suite, and much, much more.

Sample physical architecture diagram showing E-Business Suite integrated with Single Sign-On and Oracle Internet Directory with firewalls separating DMZs

The number of architectural options can be pretty bewildering, and it can be difficult to get a high-level overview of the relative benefits of each option.  We have lots of detailed documentation and introductory blog articles on, say, implementing RAC, but it's very difficult to get a sense of whether you can combine a reverse proxy in front of a load-balanced cluster with a RAC-enabled database tier (this is feasible, by the way).

[Read More]

Wednesday Aug 06, 2008

In-Depth: Using Third-Party Identity Managers with E-Business Suite Release 12

This article is an updated R12 version of an earlier one written for Oracle E-Business Suite Release 11i.

Like most of our customers, you probably already have a corporate identity management system in place. And, you've probably not been enjoying the experience of redundantly administering the same user in your corporate identity management system as well as the E-Business Suite. If this describes your environment, this in-depth article about integrating Oracle E-Business Suite Release 12, Oracle Single Sign-On and Oracle Internet Directory with third-party identity management systems will show you a better way of managing your EBS users.

[Read More]

Saturday Mar 29, 2008

Two Essential Tools for Diagnosing E-Business Suite Network Issues

Network problems seem to be on the rise again, either due to the increase in service demands, integration of different technologies such as Voice Over IP (VoIP), and the inevitable increases in uptake of technologies at new locations. Not only do network administrators have to concern themselves with various WAN optimization techniques, but they also have to deal with ad-hoc problems as they occur. As we all know too well, intermittent problems are the most difficult and resource intensive to address.

Network Diagnostic Tools for Oracle E-Business Suite

There are two diagnostic tools available in the Oracle E-Business Suite:
  • The Network test Form - available in all versions
  • The Client Analyzer - introduced in 11.5.10
Commonly, many people have found it difficult to understand the output of these tools and how they can be used to troubleshoot network problems. The Oracle E-Business Suite Network Utilities white paper explains:
  • How these two utilities work - they actually test a "form round trip"
  • How to understand the figures and what they tell you 
  • How end-users can help narrow down the source of problems and even identify which part of the system is not performing
Getting Into the Details

This last point needs a little more explanation, as this previously underused form now becomes an additional tool in a holistic approach to system performance troubleshooting. The techniques in the paper show the following:
  • How end users can gather diagnostic information saving the time and expense of deploying specialized personnel
  • How to compare the E-Business Suite measurements with standard network utilities such as ping
  • How to interpret the information to isolate a problem to the network, middle tier, or other system component.
The techniques do not require specialized skills and therefore much progress can be made towards identifying the root cause of the problem by junior system administration staff and suitably experienced end-users. Understanding the purpose of the tools and how to interpret their output enables end-users to collect the requisite diagnostic information and perform rudimentary diagnosis when performance problems occur. Inevitably, this means that this reduces the load on the support infrastructure as well as freeing up having to wait for hours for a problem to occur.

In addition to showing example usage of the tools, the paper presents sample output for users to compare with their own measurements, and thereby draw conclusions more quickly and effectively.

The chart below shows an example of how the Network Test form measurement maps to the network ping time.

Form Measured Latency Chart: Chart showing simulated Forms Measured Latency vs Actual Network Latency

By comparing the Network Test form measurement with the network ping time, you can identify:
  • If there is a problem in the network
  • If there is a problem in the middle tier
  • If there is a problem in the network and middle tier
  • Or Whether there is a problem in another system component
A couple of important points ......
  • Understand the limitations of these tools
  • Know which figures to use and how much they can be expected to fluctuate in a given scenario
  • If ping (ICMP) traffic is blocked for security reasons, the paper describes two Linux utilities that could be used to measure the network ping time
Related Articles


Tuesday Feb 19, 2008

Optimizing R12 Performance via OC4J Load-Balancing

Oracle Application Server provides features that allow customers to load balance their middle tier deployments.  OC4J Clustering in OracleAS 10g is one such deployment that is widely used in load balanced configurations.

Starting with Oracle E-Business Suite Release 12.0.2 (Release Update Pack 2), we support OC4J Clustering as part of AutoConfig's load balancing deployment options. This configuration option is supported for the OC4J instance running out of R12's 10.1.3 ORACLE_HOME.

Clustering Model

Oracle Application Server 10g 10.1.3 supports various models for OC4J clustering:
  • Dynamic node discovery
  • Static discovery servers
  • Cross topology gateways
  • Manual node configuration.
E-Business Suite Release 12 uses the manual node configuration model, which is also referred to as static node-to-node communication. Node and port information is manually specified in this mode. Configuration details are managed by Autoconfig, which handles all necessary settings in the R12 configuration files.

Figure-1 static node-to-node communication model


Deployment Options

Release 12 supports the following deployment options with OC4J clustering:
  1. Single Web Entry Point
  2. Multiple Web Entry Points


  1. Oracle Applications Technology Stack Patch Set 2 5917601 or higher.
    • This Patch Set is included in the R12.ATG_PF.A.DELTA.2 5917344 and the Release 12.0.2 (Release Update Pack 2)
  2. A correctly-configured hardware load balancer

Configuring OC4J Load Balancing

Configuring both deployment options requires:
  1. Changing the E-Business Suite Context file using the Context File Editor
  2. Running AutoConfig
  3. Restarting the application tier server processes.
Single Web Entry Point

In this deployment option, there is single web entry point for the OC4J applications - oacore, forms and oafm services are configured to run on all or some of the application tier nodes.

Figure-2: Deployment model with Single Web Entry point and OC4J application running on all the nodes

SingleWebEntry: Architecture diagram showing an E-Business Suite environment with two application tier servers and a single webentry point

Multiple Web Entry Points

In this deployment option, there are multiple web entry points with OC4J applications running on selected application tier nodes.

Figure-3: Deployment model with Multiple Web Entry points and OC4J applications running on selected nodes

multipleweb: Architecture diagram showing a multinode Apps environment

Related Articles

Wednesday Sep 05, 2007

Troubleshooting DMZ Setups for Apps

It's possible to expose selected Oracle E-Business Suite applications such as iStore or iRecruitment to users outside of your corporate intranet.  As part of our security best practices recommendations, we recommend the use of reverse proxies in demilitarized zones (DMZ) for these types of deployments.

DMZ Reverse Proxy:

While simple in concept, the actual execution is sometimes a little trickier.  These projects are often complicated by the separation between different groups that manage network operations, enterprise security, and the E-Business Suite environments themselves.  Coordinating all three organizational groups can be a project in itself.  Even small missteps can result in some of the following issues:
  • Misconfigured firewalls and other networking components
  • Incorrectly configured reverse proxies
  • Incomplete or incorrect E-Business Suite setups
  • Inconsistencies between testbeds and production setups
One Step at a Time

Debugging environments with lots of complex moving parts can be frustrating.  The best strategy is to take a systematic approach and test the critical components in sequence.  To help you with that, our hardworking Oracle Support team has assembled some of the best tips for debugging these types of configurations here:
They've also published a companion document with a crisp walkthrough:
These documents are written specifically with Release 11i in mind but the principles and techniques apply equally to Release 12, too.  Great stuff and highly recommended if you're working on implementing a DMZ in your Apps environment.


Sunday Aug 12, 2007

WebCache 10g ( Certified with Apps 11i and 12

Oracle Web Cache 10g ( is now certified with E-Business Suite Release 11i and 12.  E-Business Suite administrators may use the latest version of Oracle Web Cache to provide caching, reverse-proxy, and failover and surge protection for their environments.

E-Business Suite WebCache: Diagram showing Oracle Web Cache in front of an E-Business Suite environment

Supported Configurations

Release 11i
  • E-Business Suite 11.5.9 or higher
  • Web Cache 10g
Release 12
  • E-Business Suite 12.0.0 or higher
  • Web Cache 10g

Upgrading to Web Cache

E-Business Suite users can upgrade to the latest version by installing Oracle Application Server 10g, and then applying the Oracle Application Server 10g Release 2 Patch Set 2 (

For More Information

Tuesday Jun 05, 2007

Comparing Bandwidth Requirements between Release 11i and 12

[June 8, 2007 Update:  Clarified background on the second set of benchmarks]

A few questions have been raised in our new Release 12 Upgrade Forum about the differences in network bandwidth requirements between Release 11i and 12.

The usual disclaimers apply:  benchmark tests are conducted by the Applications Performance Group on reference E-Business Suite environments.  These environments and the selected transactions will not necessarily match up with your own transaction mix, so your mileage will vary.  If you're in need of precise benchmarks, it's always best if you perform your own tests with your own mix of transactional data.

Differences Between JInitiator and the native Sun JRE Plug-in

Release 11i currently requires Jinitiator to display Oracle Forms-based content.  Release 12 requires the native Sun Java Runtime Engine (JRE) to display Forms-based content. 

Our Applications Performance Group has published a full whitepaper comparing bandwidth requirements between these two configurations:
Differences in Page Sizes

As for our web-based applications, many of the products changed their pages and flows in Release 12.  There are actually two comparison points: 
  1. 11.5.10.CU2 vs. 11.5.10.CU2 with on the Release 12 ATG Rollup
  2. 11.5.10.CU2 vs. Release 12
Here's the first comparison of page loads:

Comparison of page load sizes - Part 1: Comparison of page load sizes between Release 11i 11.5.10.CU2 vs. 11.5.10.CU2 with the Release 12 ATG Rollup applied

[June 8, 2007 Update:  The section above is generating a lot of similar questions, perhaps understandably.  Here's the scoop:  the comparison above shows the additional overhead associated with the new R12 technology stack running the same Applications code.  This
would be the equivalent of running the 11i versions of these screens in Release 12.  Our Applications Performance team is very meticulous about comparing equivalent things.  No "apples to oranges" comparisons for them.

They deemed it important to distinguish between added network bandwidth overhead in Release 12 due to new code at the Apps layer vs. new R12 technology stack requirements.

So, their baseline for network bandwidth was the Release 11i.5.10.CU2 environment.  They then ran benchmark tests with the same Applications code and the new Release 12 ATG components.  Important note:  this configuration isn't documented, supported, or recommended -- it was strictly a Development-only configuration used for benchmarking purposes.  You cannot apply the Release 12 techstack to an E-Business Suite Release 11i environment, and we don't provide any patches to do so.]

Here's a thumbnail of the second comparison (click on it to see the full-size version, which is admittedly low resolution):

Thumbnail - Full page load comparison: Thumbnail of table comparing Release 11i and 12 page load sizes

The table above compares a Release 11i environment with a full E-Business Suite Release 12 environment.  The Release 12 environment includes not only the ATG changes but also all of the Apps code changes.  You'll see that the page sizes did increase, but these increases were mainly due to Apps code changes and user interface changes.  The new OA Framework (Swan) changes accounted for less than 4%.

New Functionality Trumps All

As Apps technologists, it's tempting for us to get caught up in these types of discussions.  "Look -- the login page is X bytes larger!" 

It's always worth remembering that your end-users don't really care much about such things, however.  They're more likely to remark on the fact that the login page now has two new capabilities:  reminding them of their password, and reminding them of their userid, too. 

It's been my experience that new functionality trumps all, at least in the eyes of your users.  My recommendation:  go ahead and consider these benchmarks as part of your Release 12 evaluations, but more meaningful comparisons will come from spending some quality time with your end-users in assessing and prioritizing the new functional benefits in R12.


Friday May 18, 2007

Debugging General Performance Issues with Oracle Apps

Identifying performance bottlenecks can sometimes be a black art with any distributed computing system.  It is sometimes difficult to know where to start.  This article gives some high-level guidance on the sort of information you may need to gather in order for Oracle Support to assist you with this task for the E-Business Suite.

Performance issues can potentially occur across any or several different areas of the Technology Stack, or may be restricted within Functional Code. For example (but not restricted to)

  • Architecture issue (e.g. high latency WAN, firewall)
  • Operating System problem or resource constraint
  • SQL or general RDBMS configuration issue
  • Database Deadlock
  • Apache Listener


Types of Performance Issues

"Simpler" Problems

These issues will hopefully be relatively straightforward to define and investigate.  For example: -
Single report consistently slow, SQL trace discovers one or more SQL statement(s) taking the majority of the elapsed time

More complex issues

These can take more work to be able to confidently define the real issue and often involve complex investigations involving different parties and much data gathering and analysis

For example :-
Any intermittent issue, System wide issues, Issues where SQL trace time represents only a small proportion of the elapsed time

Gathering Data for Performance Issues

It's helpful to follow a systematic process for investigating performance issues.  First steps include identify the nature, scope and extent of the problem. 

1. Identify the Extent of the Problem

Determine whether the extent of the problem is:

  • System Wide
  • Confined to one Technology Are. For example does it only happen in one of the Self Service, Forms or Reports areas
  • For a specific Product Area(s) For example, were it to only occur in HR, GL or iStore
  • Single Report/Form/Page
  • Which Instance or instances is the problem observed. Can it be reproduced in UAT/TEST instances

2. Narrow the Scope Further

Determine whether the problem occurs for:

  • All users
  • Only one geographic location
  • Only for users with certain Browser/type of PC
  • Only during peak periods

3. Qualify the Nature of the Problem

  • Is the problem intermittent or reproducible?
  • Is the problem due to slow performance or a process hanging or spinning?
  • When was the last time the process was completed without experiencing poor performance?  Document any changes since then.
  • Is there a workaround available?   For example, restarting Browser or restarting Apache.
  • What is the frequency of occurrence?

4.  Capture Additional Clues

  • Document and differentiate factual information from user perceptions. Example - users may say "it's always slow" but reality may be that it takes 10 seconds most of the time, but 40 seconds between 10am and 11am (peak load).
  • How long does it take for the process to complete when running slowly?  How long did it previously take (before having the performance issue)?  What is the expected performance for this process?
  • What is load on server(s) when issue occurs (CPU/Memory/Network)?  Is there any unusual activity when problems occur?  For example, memory used suddenly growing or CPU at 100%.
  • Is the problem reproducible with other browsers, such as Firefox, as well as with Internet Explorer?

5.  Identify likely causes and eliminate other possible causes

  • Are there any customizations?  If so, can they be eliminated for testing purposes?
  • If the problem only occured since patches have been applied or any configuration changes have been made, can these be reverted?
  • Do you have any Resource Limits enabled on the database that may be effecting the Applications users runtime or Concurrent Manager?
  • Does the number or frequency of certain Concurrent Requests correlate to performance issues occurring?
  • Have the database parameters been configured as per the current recommendations in Database Initialization Parameters and Configuration for Oracle Applications 11i (Metalink Note 216205.1)?
  • Does AWR or Statspack show anything unusual for Top waits, Top buffer gets, etc?
  • Have you searched Metalink generally, but specifically reviewed Recommended Performance Patches for Oracle E-Business Suite (Metalink Note 244040.1) for any known performance issues, tuning guidelines and/or patches?

Getting Help from Oracle Support

Once you understand the issue and reach the point where you need Oracle Support involved, it may be useful for you to review the "Performance Tuning" section of:

The first key decision you need to make is selecting a product code for your Service Request (SR): 

  1. If the issue is with an individual Form/Report/Page or only in one product area then log the SR for that particular product support team
  2. If issue seems to be with the Technology Area or System wide then log the SR with the AOL team (Oracle Application Object Library)

It is very important to provide a good problem definition (nature, scope, extent) so be as verbose as needed to give a good description of the issue.

It is also very important to reproduce the issue on a non-Production instance. If you are able to reproduce the problem outside of your Production instance, then any recomended patches or changes can be quickly assessed for their impact and any detailed debug or tracing required to identify the issue can be easily implemented.

A Quick Aside:  One issue per Service Request

You may need several issues investigated simultaneously and may be reluctant to raise different SRs or have different people investigating. Unfortunately, even similar-looking issues can have different root causes, which means they will need to go to different support teams or have different SR statuses. This is why it is important to ensure each issue is raised as a separate SR.

Performance Issues with Specific Forms, Reports, or Pages

For these "simpler" types of issues, we can generally track down the issue with the following information:

  • Form, Report or Page name, version and navigation path
  • Full versions of Forms, Reports or Framework as well as the iAS and Database versions, including rollup or interop patches applied
  • Relevant Family Packs or Maintenance Packs applied
  • Description of symptoms, what have you tried, your investigations, conclusions and/or thoughts/ideas
  • Does it reproduce constantly, in other environments as well?
  • When did you last run "Gather Schema Statistics" and at what percentage?
  • When did you last run any relevant purging processes?
  • SQL trace with binds and waits (raw file and TKPROF output)
  • Wall clock time elapsed from user perspective
  • What are the target and acceptable times for this process?

Performance Issues for Other Areas

These are the more complex issues and will normally require additional information (and patience) to resolve.  We will need the same information as listed above, but also:

  • Description of your System Architecture and network diagram
  • Technology Stack configuration files (Apache, Forms, Reports and Database, as relevant to the problem)
  • List of any Metalink notes or product documentation you have reviewed already, and what steps you implemented from this documentation
  • List of any patches applied specifically to try to address this issue
  • Details of profile options or configuration settings you have tried changing, explaining why you tried these settings and what effect they had (if any)
  • AWR or Statspack output for "good" and "bad" performing time periods
  • Details about your pinning, purging and gather schema statistics strategy
  • Relevant Log files
  • Debug and trace files

Good starting points for collating all this information are:

Additional Tips for Troubleshooting for Apache / JVM problems

Additional Tips for OA Framework-related issues

  • It is often useful to enable "STATEMENT" level logging (for ONE user only!) using the FND:Debug% profile options if you have a reproducible test case


Performance issues can sometimes be tricky to isolate, particularly those having more than one root cause.  This article has presented some ideas about the approach to take, in addition to the sort of information Oracle Support would likely be asking for if you need to log a Service Request.

If there is sufficient demand, I can write further articles in future, expanding on the topics introduced here.


Monday Apr 30, 2007

Lease Management Modules Supported for DMZs

[Editor Apr 30 Update:  Overloaded acronyms... >sigh<   Corrected entry from Oracle Learning Management to Oracle Lease Management.  Support is only for Release 11i presently.]

Two new Oracle Lease Management (OKL) modules in the Financials product family are now certified for external use in demilitarized zone (DMZ) configurations:

  • Customer Self-Service
  • Vendor Self-Service
These certifications apply to E-Business Suite Release 11i and are available for immediate deployment for production Apps environments.  For more details, see the DMZ documentation below.


What Does "DMZ Certification" Mean?

Depending on whom you ask, the E-Business Suite has somewhere around 200 functional applications products, clustered into larger product families such as Oracle Financials.  A subset of those products are specifically certified for deployment in an externally-facing configuration via demilitarized zones (DMZ).  For example, products certified for these types of "external" deployments include iRecruitment, iStore, and iSupplier Portal.

DMZ Reverse Proxy:

The diagram above shows a common DMZ configuration for the E-Business Suite Release 11i.  All of the points I'll make in this article apply equally to Release 11i and 12.

Loopbacks are Incompatible with DMZs

Some E-Business Suite products use loopbacks, which I've discussed in a previous article.  Apps products certified for external use in demilitarized zone configurations are tested to ensure that they don't use loopbacks. 

In fact, we turn off loopback support completely as part of the DMZ certification process for externally-facing products.  If a particular product breaks during testing in these environments, this means that their code must be upgraded to eliminate the use of loopbacks.

Which Products are Certified for DMZs?

Products certified for external deployment are listed in:
Not all Apps products are appropriate for use in demilitarized zones, so product testing in these configurations isn't comprehensive across all product families.  For example, regardless of security measures, no sane Apps architect would consider allowing their Chart of Accounts to be modified via the Internet.  So, there's no point in certifying that particular product with in a DMZ configuration.

If a product isn't in listed in the appendices of the Notes listed above, it could mean one of two things:
  1. It uses loopbacks and is not certified for external use in a DMZ configuration
  2. It hasn't been tested in a DMZ configuration, and may or may not use loopbacks
What If a Product Isn't Certified?

Here's a hypothetical situation:

You'd like to deploy a particular application externally in a DMZ configuration. It's not listed in either of the referenced Metalink Notes.  What do you do?

The answer:  log a Service Request against the specific application via Metalink stating your requirement.  It always helps to include a network diagram of your proposed topology, by the way.  If all goes as planned, the Development team for the product will be notified of your requirement and will respond with an update on their plans for that certification.


Wednesday Mar 14, 2007

Loopbacks, Virtual IPs and the E-Business Suite

[Editor Mar 27 , 2007Update:  Updated with more information about loopback requirements for internal and external applications.]

[Editor Mar 16 , 2007 Update:  Added loopback diagram and updated test section with additional comments.

An area that seems to be perennially troublesome for E-Business Suite architects and sysadmins is that of multiple domain names, virtual IPs and loopback issues.  Here's a quick primer on some of the key concepts that you should be familiar with.

What's a Virtual IP?

Let's say that you'd like to set up an E-Business Suite environment with two different domain names, following Metalink Note 287176.1 (for Release 11i) or Note 380489.1 (for Release 12):
  • for external users
  • for internal users
If you've got a generous networking budget, one approach would be to set up a physical architecture shown below.  This architecture uses two different physical load-balancers, one dedicated for external users (LBR1), and another dedicated for internal users (LBR2):

Physically Separate Load-Balancers:

Note that in the architecture above, the reverse proxy server acts as the "web-entry point" -- that is, the primary point from which all end-user traffic gets dispatched -- for external users.  That's the gold-plated approach. 

However, many customers have big, sophisticated load-balancers that can handle the combined traffic for both domains.  In these situations, the single physical device can be assigned two virtual domain names, each with their own virtual IP address, like this:

Virtual IPs for a Single Load-balancer:

In this architecture, the load-balancer has two virtual domain names, each of which serves as the "web-entry point" for the respective domains:
  • Traffic from User 1 for goes to the external pool of application servers, either Node 1 or 2
  • Traffic from User 4 for goes to the internal pool of application servers, either Node 3 or 4
Technical Requirements for E-Business Suite Environments

Virtual IPs and multiple domain names are supported for E-Business Suite, for both Release 11i and Release 12.  A number of basic requirements need to be met:
  1. Each virtual domain name must have its own virtual IP address, and your end-users should be able to access those IPs in your DNS. 
  2. E-Business Suite technology stack components need to be able to access the load-balancer's virtual IPs, too, due to loopback requirements (see below).
If your load-balancer is capable of it, you should enable additional features such as resource monitoring, fail-over, and immediate returns of failed traffic.  For more details about our recommendations for those features, see the Metalink Notes below.

What are Loopbacks?

For reasons too arcane to delve into here, some components in the E-Business Suite technology stack sometimes need to call other techstack components (or themselves) at various times.  They do so by sending those calls to the web-entry point -- the load-balancer or reverse proxy for their domain -- which forwards those calls to the requested technology stack component, as shown in the following diagram:

Revised Loopback Example:

Loopbacks in Internal and External Applications

Some E-Business Suite modules can be deployed for external use.  These include iSupplier Portal, Oracle Sourcing, iRecruitment, iStore, iSupport, and others listed in Appendix A of Metalink Note 287176.1.  These applications do not require loopbacks.

Applications that aren't on this list are intended for internal deployments.  These applications may require loopbacks.  In some internal architectures, the internal web-entry point is separated from the actual application tier server node by a firewall.  In the diagram above, E-Business Suite techstack components in the application server pool used by internal users will send their loopback requests to the HTTP LBR2 device.  It's not shown in the diagram, but one can envision an architecture where a firewall exists between HTTP LBR2 and Web Nodes 3 and 4.  If this firewall blocks outbound loopback traffic from the Web Nodes to the load-balancer, then the techstack services for internal applications that depend on those loopback connections will start to fail... sometimes in puzzling ways.

Firewall Rules & Production Rollouts

This is a major source of problems that I see reported when moving from testbeds to production rollouts.  Testbeds usually include only one or two machines and they're never separated by firewalls.  However, production systems are usually spread across multiple physical servers, each separated by firewalls. 

Making the situation even more entertaining, networking, security, and E-Business Suite administrators are often in different groups.  Invariably, one team forgets to fill in the other team on their networking requirements.  The E-Business Suite (which worked fine in the testbed environment) seems cranky and unstable in pre-production.  The production rollout gets hung up until the problem is diagnosed and appropriate firewall rules are tweaked.

Don't let that happen to you.  Identifying a loopback problem is really simple:  on each of your application servers in each of your domains, use either ping or telnet to hit each domain's web-entry point.  Likewise, from both an internal and external end-user desktop, ping the respective internal or external E-Business Suite domain name.  If you get a response in all cases, then your firewall rules are
configured correctly.  If not, then give your network and security
teams a call.

Update:  It's been pointed out that the ping test alone is a necessary but not sufficient test, since it doesn't prove whether a given port is accessible to the calling client or application tier service.  Additional tests such as using wget or telnet (the latter documented in the latest updates to Metalink Note 217368.1) may also required to demonstrate that a particular port is accessible through one or more firewalls.


[Unrelated postscript:  Possibly due to sunspots, the lunar cycle, or global warming, any emails you sent me on March 11 or March 12, 2007 may be delayed, possibly indefinitely.  If you sent me anything in that time, I'd recommend resending it.]

Thursday Aug 24, 2006

DMZs, SSL and RAC for OracleAS 10g + Release 11i

I know that many of you have been waiting for this announcement for a long time, so it's a real pleasure (and relief) to be able to tell you that Build 4.0 is finally here.

A new version of the OracleAS 10g integration with the E-Business Suite has been released for use with ATG Family Pack H Rollup 4.  This long-awaited integration patch, also known as Build 4.0, includes full support for three additional configurations:  DMZs, RAC, and SSL.

Demilitarized Zones and OracleAS 10g Integrations

In prior releases, there were a number of challenges to integrating an OracleAS 10g instance with an E-Business Suite environment deployed in a demilitarized zone (DMZ) configuration with multiple web entry points.  Some awkward workarounds existed, but they were incomplete, technically clumsy, and didn't work consistently in all circumstances.

With this latest release, full support for OracleAS 10g + E-Business Suite + DMZ configurations is now available.  This release allows you to register multiple E-Business Suite application servers (e.g. internal and external Oracle9i Application Server instances) with an OracleAS 10g and Single Sign-On instance, supporting the proper redirection of traffic to the appropriate server after authentication. 

This means that architectures like this are now fully supported:

DMZ + OracleAS 10g + E-Business Suite Architecture:

Registration with SSL-Enabled Oracle Internet Directory Hosts

In prior releases, it wasn't possible to register your E-Business Suite environment with an Oracle Internet Directory host deployed in a Secure Sockets Layer (SSL) configuration.

In this latest release, if your Oracle Internet Directory host is configured for SSL-enabled LDAP operations, you can use wallets in Oracle Wallet Manager to secure all LDAP operations.

Oracle Internet Directory Integration with RAC-Enabled Release 11i Databases

In prior releases, if your E-Business Suite database was configured to use Real Application Clusters (RAC), the synchronisation of user information between Oracle Internet Directory and FND_USER was handled by a specific database server in your RAC cluster.

If that database node failed, the synchronisation of user attributes between Oracle Internet Directory and the E-Business Suite wouldn't failover to other database server nodes.  Updates of user information in either direction would be suspended until the designated RAC database node came back online.

In this latest release, the E-Business Suite RAC service name is used when registering the Release 11i instance with Oracle Internet Directory.  All user synchronisation events are handled by the E-Business Suite RAC cluster now, so if a given RAC node fails, synchronisation of user information will continue as long as other RAC nodes are still running.


Tuesday Jul 25, 2006

IPv6 and the E-Business Suite

[May 27, 2008 Update:  The E-Business Suite is now certified to be IPv6-compatible; see this announcement for details.]

[May 9, 2007 Update:  As of today, this article still represents our latest status on IPv6 certification for the E-Business Suite, for both Release 11i and 12.  We have been briefed on the US Federal requirements for the 2008 changeover.  Aside from those US governmental organizations, if you haven't already contacted us about your IPv6 requirements, please drop me a line.]

A very small number of E-Business Suite customers have expressed interest in Internet Protocol Version 6, otherwise known as IPv6. From the the IPv6 Information Page:

IETF Logo:

IPv6 is the "next generation" protocol designed by the IETF to replace the current version Internet Protocol, IP Version 4 ("IPv4").

Most of today's internet uses IPv4, which is now nearly twenty years old. IPv4 has been remarkably resilient in spite of its age, but it is beginning to have problems. Most importantly, there is a growing shortage of IPv4 addresses, which are needed by all new machines added to the Internet.

IPv6 fixes a number of problems in IPv4, such as the limited number of available IPv4 addresses. It also adds many improvements to IPv4 in areas such as routing and network autoconfiguration. IPv6 is expected to gradually replace IPv4, with the two coexisting for a number of years during a transition period.

Certification Plans for the E-Business Suite

Certification of the E-Business Suite with IPv6 is in the queue for evaluation and feasibility analysis, but we don't have any commitments or timelines that we can share at this point.

Help Influence Our Priorities

If your organization is committed to migrating IPv6, please add a comment to this article or drop me an email with the details, including timelines and how you expect this to affect your E-Business Suite deployments.  Your feedback helps us prioritize this certification for future releases.

Friday Jun 16, 2006

In-Depth: Load-Balancing E-Business Suite Environments

As I watched blood drain from my arm, my thoughts turned to load-balancing and system redundancy.  My phlebotomist had just cheerfully informed me the lab's primary testing PC had failed that morning, so there would be an unusual delay in getting blood test results until a replacement arrived.

Increasing Fault Tolerance at Lower Cost

You can use load-balancing routers (LBRs) to protect your E-Business Suite from similar types of system failures.  Load-balancers increase your environment's fault-tolerance and scalability by distributing load across a pool of application servers like this:

Generic Apps Load-balancing:

Besides fault-tolerance and scalability, another appealing benefit is that you can use load-balancing to substitute expensive SMP boxes with clusters of inexpensive Linux-based commodity servers. 

Linux Load-Balancing on Oracle's Global Single Instance

In fact, this is what we've done at Oracle:  our own E-Business Suite environment is a Global Single Instance running on about 58 Linux-based application servers.  This has reportedly saved us millions in operating costs. 

If you're joining us at this year's OpenWorld conference, make sure you attend Bret Fuller's ever-popular session on how we run the E-Business Suite internally; some of the statistics he shows on our transactional volumes are mind-blowing.

Supported Load-Balancing Methods

The E-Business Suite supports the following types of load-balancing:
I'll cover only the first two methods in this article.

HTTP Layer Load-Balancing

HTTP Layer load-balancing is the most common method used in E-Business Suite environments. 

HTTP Layer Load-Balancing:

In this configuration, end-users navigate to a specific Web Entry Point that represents your E-Business Suite's domain name.  An HTTP Layer load-balancer routes all subsequent traffic for a specific user to a specific Web Node.

HTTP Layer load-balancers may use heartbeat checks for node death detection and restart, and sophisticated algorithms for load-balancing.

DNS-Based Load-Balancing

When an end-user's browser attempts to access your E-Business Suite environment, your local Domain Name Server (DNS) can direct that user to a specific application server in a pool based on available capacity:

DNS-Based Load-balancing:

Traffic for that user's session will be handled by the application server, while other users' traffic may be directed to other application servers in the pool.  Like HTTP layer load-balancers, many DNS-based load-balancers use heartbeat checks against nodes and sophisticated algorithms for load-balancing.

Business Continuity ("Disaster Recovery")

Our larger enterprise-class customers combine DNS-based and HTTP layer load-balancers to support their business continuity plans.  In the event of a disaster, end-users are directed via a DNS-based load-balancer from the primary E-Business Suite environment to an offsite standby site.

Disaster Recovery using DNS + HTTP LBRs:

Minimum Requirement:  Session Persistence

Remember that although Oracle doesn't certify specific load-balancers or networking hardware with the E-Business Suite, we do support their use generically.  In other words, we've designed the E-Business Suite to be able to use load-balancers in general.

The minimum requirement is that a load-balancer support session persistence, where a client's initial HTTP connection is directed to a particular application server, then subsequent HTTP requests from that client are directed to the same server.  As long as a load-balancer is able to handle session persistence (also referred to as "stickiness"), it's likely to work with the E-Business Suite.


Wednesday Jun 14, 2006

Using Third-Party Networking Hardware with Oracle Application Server 10g

If you recall, the E-Business Suite isn't explicitly certified with third-party networking components but is expected to work with them. 

Load-balancer with OracleAS 10g Identity Management:

If you're integrating the E-Business Suite with Oracle Application Server 10g, there's another piece that you can add to the puzzle.

Oracle Application Server 10g isn't explicitly certified with third-party networking components, either, but some testing has been performed with specific vendor products.

These tests cover load-balancers, firewalls, and SSL accelerators.  Some of the load-balancer vendors and products include F5's BIG-IP, Foundry, Citrix's NetScaler, Nortel, and Radware.  The firewall and SSL vendors include Check Point, Cisco, Sonic Wall, and Ingrian.

Remember that Oracle testing doesn't equate to certification.  It's the responsibility of the third-party vendor to certify their hardware with Oracle Application Server 10g.  Regardless of that, you might find it reassuring to know that Oracle's tried some of these combinations in the Oracle Application Server 10g labs. 

Even if your networking vendor hasn't certified their hardware explicitly Oracle Application Server (or even the E-Business Suite), it's generally expected that their products will work if they're standards-compliant.


Thursday May 18, 2006

Certification and Support for Third-Party Products

One of the most frequently asked questions I answer daily is this:  "Is my third-party product __________ certified with the E-Business Suite?"

The short answer is, "No, it's not."  But wait...don't leave just yet!  That doesn't mean that you can't use these products with the E-Business Suite, it just means that we haven't certified them ourselves.

To understand how this affects you, we need to make a distinction between certification and support for third-party products in E-Business Suite environments.

How Are Technology Stack Components Certified?

From an E-Business Suite standpoint, our certification process looks like this:

Phase 1:  Applications Technology Group Testing
  • Creation of formal installation documentation and patches
  • Installation into one or more Linux-based technology stack testbed environments, testing upgrade paths from previous older configurations, and compatibility testing with other architectural deployment options such as load-balancers and firewalls
  • Manual and automated regression testing of core technology stack functions and a subset of key Apps business flows
Phase 2:  E-Business Suite Division-Wide Testing
  • Installation into one or more Linux-based testbed environments shared by all Applications products
  • Testing by one or more Applications product teams, including manual and automated regression tests of key product functionality
Phase 3:  Platform-Specific Testing
  • Installation into testbed environments running on other platforms, such as HP-UX, Sun Solaris, IBM AIX, and Microsoft Windows
  • Manual and automated regression tests of key product functionality
Phase 4:  Early Adopter Program Testing

For highly-complex or potentially disruptive new technologies, we sometimes include this final phase:
Once all of the phases are passed successfully, we release the associated documentation and patches and the configuration is considered Generally Available and certified.

We Support More Than We Certify

The number of possible configurations and third-party components far exceeds the number of things that we certify through the process above. 

Our general policy is that our generic documentation should apply to all third-party components, even if we haven't certified those products specifically. 

For example:

We support the use of demilitarized zones and firewalls with the E-Business Suite. 

It isn't possible for us to put every single commercial firewall through the certification process above, so we've developed generic patches and documentation for this configuration.  These generic patches and documentation should apply to all firewall vendors' products.  Regardless of what firewall you choose, the E-Business Suite should work the same way. 

So, all firewalls are supported with the E-Business Suite, provided that they meet the minimum functional requirements listed in our DMZ documentation.

What Does "Support" Really Mean?

If you encounter a problem with a third-party component, the odds are that we won't have a setup that's identical to yours.  Here's what you can expect when logging a Service Request for a third-party component in your E-Business Suite environment:
  1. Oracle Support will check that you've followed the standard configuration documentation in setting up your environment.

  2. Oracle Support may attempt to replicate your issue in a generic environment that most-closely resembles your environment.

  3. Oracle Support may work with you to see if the problem goes away when the third-party component is temporarily removed or deactivated.

  4. If the problem exists in a generic environment or when the third-party component is temporarily deactivated, then Oracle Support will attempt to identify a patch or workaround for the issue.  A bug may be logged to request a new patch.

  5. If the problem disappears when the third-party product is removed, then it suggests that the third-party product is the source of the issue.  If so, Oracle Support may suggest that you contact the third-party vendor for their help in debugging their product.
Third-Party Hardware and Software

The processes and policies that I've described above apply to nearly all third-party products in E-Business Suite environments.   This includes software (like Citrix) and hardware (like F5 or Cisco load-balancers) and combinations such as third-party cloud-based hosting services (e.g. Amazon Web Services EC2).

Escalate As Needed

A major part of my job is working directly with customers, and I know that you can sometimes find these integration issues frustrating to work through. 

Remember:  we won't turn your Support Requests away -- we'll do our best to reproduce and isolate the issue in a generic environment. 

If you get stuck or feel that your Service Request is going around in circles, don't hesitate to contact an Oracle Support Duty Manager and ask for your Service Request to be escalated.  Sometimes, trickier architecture issues may require direct help from us in the Applications Technology Group; Support will log a bug in those cases to get us engaged.


Wednesday May 17, 2006

In-Depth: Demilitarized Zones and the E-Business Suite

If you've been wondering how to support end-users who'd like connect to your E-Business Suite environment from outside of your corporate firewall, a combination of a demilitarized zone and a reverse proxy might be an alternative to traditional VPN-based solutions. 

The E-Business Suite is Not A War Zone

The term demilitarized zone (DMZ) is said to have been coined following the Korean War armistice.  After the cessation of overt military engagements, a 4 km buffer zone was established between North and South Korea, each side of the border bristling with armaments and troops watching the other warily.

Although major upgrades can sometimes feel like a battle, Apps sysadmins thankfully don't have to worry about armed attacks.  Military security concepts are still useful, though:  Oracle's Chief Security Officer, Mary Ann Davidson, is a former military officer and frequently draws lessons from military history for IT security.

DMZs For Civilians

In the IT industry, a demilitarized zone is a single or multi-segment perimeter network that demarks the portion of the corporate network that lies between the intranet and outside networks.  Corporate DMZ borders are enforced by firewalls and other dedicated networking devices. 

Generic DMZ Concept:

DMZs for the E-Business Suite

AutoConfig supports the use of DMZs with the E-Business Suite Release 11i, and an increasing number of our customers have either already implemented them or are planning to do so.  This is a common configuration:

DMZ Internal External Servers:

In the configuration above, there are two different E-Business Suite application servers, each with its own unique domain name and setup.  External users access the E-Business Suite via the external "" address, and internal users access it via the "" address.

Different Responsibilities for Internal and External Servers

It's possible (and recommended) to restrict the general set of Applications Responsibilities based on the application server that you're using. 

For example, there should be no reason to allow external users to modify your company's Chart of Accounts, so that responsibility can't be used if the end-user is logging in from outside the corporate intranet.

Possible Weak Points

There are two possible weaknesses with the first configuration shown above:
  1. If your external firewall is compromised, your external application server is also compromised, exposing an attack on your E-Business Suite database.
  2. There's nothing to prevent your internal users from attacking your internal application server, also exposing an attack on your E-Business Suite database.
Reverse Proxies and DMZs

If you're concerned about your external firewall being hacked, one possible countermeasure is to use layered DMZs and put a reverse proxy in the first DMZ. 

DMZ Reverse Proxy:

The reverse proxy has restricted capabilities and and the authority only to speak with the external application server.  It's possible to use the following as reverse proxies with the E-Business Suite:
  • Oracle Web Cache
  • Oracle HTTP Server
  • Other third-party reverse proxy servers, including Apache and Microsoft Proxy Server
An Inside Job

I'm a big fan of heist and con artist movies.  According to Hollywood, you can't pull off a big job without someone on the inside. 

It seems a lot of IT security analysts are fans, too, since they regularly publish surveys that suggest that the majority of security breaches are the result of employees with their hand in the till.  If we're to learn anything from movies, it's this:  trust nobody, not even your internal end-users.

That's why the second configuration above shows the E-Business Suite database server protected by its own firewall.  Even if your internal application server is compromised by an industrious but disgruntled fellow employee, your database is still protected.

Scratching the Surface

There are a number of other interesting DMZ-related architectural options for the E-Business Suite.  If you'd like to get more details, the following document is recommended reading:

Friday Apr 21, 2006

Managing E-Business Suite Configurations with AutoConfig

Work is its own reward.  You can expect great rewards this year.

~ Paraphrased from Dilbert, Scott Adams

Historically, one of the biggest challenges facing any E-Business Suite system administrator was managing the countless configuration files for technology stack components. 

In the past, different E-Business Suite products would each have their own technology stack configuration recommendations.  For example, iProcurement might recommend that a certain parameter be set to a given value in httpd.conf.  Naturally, it's inevitable that a different product family would come along and recommend a completely contradictory setting for the same parameter. 

Further complicating things:  if the hapless system administrator chose to follow either recommendation, there was no guarantee that the new setting wouldn't break a third unrelated product.  It was enough to make grown sysadmins weep.

Enter stage left, AutoConfig

AutoConfig is a tool that automates the management of all configuration files for all E-Business Suite Release 11i technology stack components.  The Applications Technology Group now centrally controls all parameter settings for all configuration files for the E-Business Suite. 

You might reasonably have expected from the start, but there are over 200 products within the E-Business Suite, and gaining agreement from all development groups to centralize this kind of control within the Applications Technology Group was about as simple a political process as nominating a presidential candidate.  This took years.

That's all behind us now, and today, individual product families are no longer permitted to make recommendations for technology stack configurations of any kind, and any changes to known-good parameter settings are now centrally tested to ensure that they work with all 200 or so E-Business Suite products.  It's about time, and you're the main beneficiary.

Beneath the Hood:  AutoConfig

All of the information required for configuring an Applications instance is collected into two local XML repositories called the Applications Context and the Database Context.  This information describes your instance name, location of servers, and so on.  With the latest Rapid Installs, the information you originally provided at install time is the basis for the Applications and Database Context files.

When AutoConfig runs on the Application tier, it merges information from the Applications Context with presupplied configuration file templates to generate all application-tier configuration files and update database profiles.

When AutoConfig runs on the Database tier, it uses information from the Database Context file to generate all configuration files used on the Database tier.

If you're updating the configuration for an existing instance with AutoConfig, it will take a snapshot of your current configuration before installing the new configuration files.  You can roll back your configuration to any snapshot taken at any given date.  This allows you to experiment safely with different configuration options.  Didn't like the effect of the last change?  Just roll back to the previous AutoConfig snapshot.

But wait... there's more.  AutoConfig can also start and stop all technology stack components that it manages, and there are additional options for pregenerating test configuration files and examining the differences with your existing configuration.

AutoConfig Now Preserves Customizations

With all of these great features, as well as AutoConfig's devotion to motherhood and apple pie, the reluctance of sysadmins to use AutoConfig has been a source of some... ahh... perplexity within our team. 

After all, who wouldn't want to use a tool that guarantees a known-good configuration that works for all E-Business Suite products?  Who could possibly want the burden of managing configuration files themselves, with this as an alternative?

Well, a lot of you, as it turns out.  More than we expected, in fact. 

With some digging, we learned that you've invested in building your own custom configurations and don't want us overwriting your hard-earned changes.   Your customizations might address the need to:
  • Start additional services or processes when you start Oracle Applications services
  • Define and add zones to your JServ configuration
  • Extend Forms to integrate with a third party Java version
  • Develop customer applications that are maintained by AutoConfig
That's understandable, so we enhanced AutoConfig.  Your customizations are now preserved after running AutoConfig, and persist even after new AutoConfig templates are installed. 

So, What Did We Miss?

Despite all this, we still have the nagging impression that the majority of E-Business Suite system administrators still don't use AutoConfig.  We really don't know why.

Assuming that you're all rational, there must be good reasons.  Clearly, you have requirements that AutoConfig doesn't meet yet.

If you haven't switched over to AutoConfig yet, I would appreciate your posting a comment about new features that would encourage you to make the switch.




« July 2016