Wednesday Jun 17, 2009

Which is Better: Forms Servlet or Socket Mode?

Many products within the Oracle E-Business Suite have screens that are built with Oracle Forms.  Oracle Forms can be run in either servlet mode or socket mode.  Apps 11i is based on Forms 6i and is configured to run in socket mode by default.  Apps 12 is based on Forms 10g and is configured to run in servlet mode by default. What are these modes, and which is better?

What is Forms Servlet Mode?

The Forms Listener Servlet is a Java servlet that delivers the ability to run Oracle Forms applications over HTTP and HTTPS connections. It manages the creation of a Forms Server Runtime process for each client, as well as network communications between the client and its associated Forms Server Runtime process.

The desktop client sends HTTP requests and receives HTTP responses from the web server. The HTTP Listener on the web server acts as the network endpoint for the client, keeping other servers and ports from being exposed at the firewall.

Forms listener servlet diagram showing firewalls desktop client and oc4j container on application tier

What is Forms Socket Mode?

Initial releases of the Oracle Forms Server product used a simple method for connecting the client to the server. The connection from the desktop client to the Forms Listener process was accomplished using a direct socket connection.  The direct socket connection mode was suitable for companies providing thin client access to Forms applications within their corporate local area networks. For the direct socket connection mode, the client had to be able to see the server and had to have permission to establish a direct network connection.

Although the direct socket connection mode is perfectly suited for deployments within a company’s internal network, it's not the best choice for application deployment via unsecured network paths via the Internet. A company connected to the Internet typically employs a strict policy defining the types of network connections that can be made by Internet clients to secure corporate networks. Permitting a direct socket connection from an external client exposes the company to potential risk because the true identity of the client can be hard to determine.

Servlet Mode Advantages

  1. HTTP and HTTPS traffic is easily recognizable by routers, while socket mode communications is generally considered suspect and treated on an exception basis. 
  2. Existing networking hardware can be used to support basic functions such as load-balancing and packet encryption for network transit.
  3. More resilient to network and firewall reconfigurations.
  4. More robust: servlet connections can be reestablished if network connections drop unexpectedly for Forms, Framework, and JSP-based pages.
  5. Is the only supported method for generic Oracle Forms customers, and therefore is more thoroughly tested by the Forms and E-Business Suite product groups.
  6. Performance traffic can be monitored via tools like Oracle Real User Experience Insight (RUEI).
  7. Socket mode is not supported on Windows-based server platforms.

Socket Mode Advantages

  1. Uses up to 40% less bandwidth than Forms servlet mode.  This may be perceived by Wide Area Network (WAN) users as causing slower responsiveness, depending upon network latency.
  2. Uses fewer application-tier JVM resources than servlet mode, due to fewer TCP turns and lack of overhead associated with HTTP POST handling.

Switching Apps Deployments Between Modes

Due to its numerous advantages, Forms servlet mode is the preferred and recommended deployment model for Forms on the web. 

There may be circumstances where you need to switch between the default Forms modes.  You might wish to switch your Oracle E-Business Suite Release 12 environment to socket mode to improve performance or reduce network load.  You might wish to switch your Apps 11i environment to servlet mode as part of your rollout to external web-based end-users outside of your organization.

If you're running Apps 11i and would like to switch to servlet mode, see:

If you're running Apps 12 and would like to switch to socket mode, see:

Related Articles

Tuesday May 26, 2009

15 New Technology Stack Enhancements in EBS 12.1.1

Now that our latest Applications Release 12.1.1 is available, here's a list of new technology stack configuration features you might be interested in learning about.  While you're reviewing new R12.1.1 content, please do not miss our newly revamped and user-friendly AutoConfig guide:

This updated AutoConfig guide has been restructured to present you more in-depth and practical information to get you get started quickly with AutoConfig and all its related utilities. Please check it out and let us know what you think about it!

Diagram showing the process by which the AutoConfig Build Context Utility consolidates data from the EBS database environment variables and context template to generate a context file

What's New in 12.1.1's Techstack Utilities?

Here's what's new in our Apps 12.1.1 technology stack tools and utilities:

Enhanced Support for Sharing Application Tier File System

  • Enables the sharing of the application tier file system amongst two or more Oracle E-Business Suite instances.
  • More information in My Oracle Support Knowledge Document 384248.1

Enhanced Support for Application Tier Load Balancing

  • Provides configuration support for major load balancing categories: DNS, OC4J Native, HTTP layer (hardware/software).
  • More information in My Oracle Support Knowledge Document 380489.1

Enhanced Support for DMZ Deployments

  • New demilitarized zone (DMZ) deployment options added, including support for forward proxies, reverse proxies with no external web tiers, and the use of hardware load-balancers without an external web tier.
  • More information in My Oracle Support Knowledge Document 380490.1

Network Traffic Encryption Capability

  • Provides AutoConfig support for securing the major communication routes with SSL.
  • More information in My Oracle Support Knowledge Document 376700.1

Advanced Configuration Wizards

  • Examples of such advanced configurations include DNS load balancing, HTTP load balancing, SSL setup on web server, SSL Accelerator setup, and others.

Oracle Connection Manager Technology Integration

  • Reverse proxy support for the database using Oracle Connection Manager
  • Oracle Connection Manager (CMAN) is a lightweight, highly-scalable program that works as a proxy server, forwarding connection requests to database servers or to the next proxy server. Oracle Connection Manager listener receives connection requests, evaluates them against a set of rules, and determines whether to deny or allow access.
  • More information in My Oracle Support Knowledge Document 558959.1

Support for Integrated SOA Gateway

  • More improvements and automation around integrating with Service Oriented Architectures (SOA) and web services.
  • More information in My Oracle Support Knowledge Document 556540.1

Technology Stack Inventory Validation Report

  • Allows administrators to review the versions of various installed technology components, patchsets and interim ("one-off") patches.

AutoConfig Profiler

AutoConfig Parallelization

  • AutoConfig can be run in parallel on different nodes of an Oracle E-Business Suite system, reducing the overall time needed to run AutoConfig.

AutoConfig Service Control Dependency Management :

  • Now it is possible to enable and disable specific OC4J instances on the Application Tier Servers.

AutoConfig Check Config Utility

AutoConfig Support for Oracle Database 11g

  • Support to run on and configure an Oracle 11g database using AutoConfig.

Build Context XML Utility

AutoConfig Search Utility

  • A new search utility that can be used to obtain detailed information about context variables and the templates that use them.
Note: The enhanced functionality above is not available as standalone downloads or for previous Apps releases.  They can only be obtained via Oracle E-Business Suite Release 12.1.1.

Other References

Related Articles

Monday Apr 20, 2009

New Whitepaper: Manually Cloning Apps 11i Databases Running 10g or 11g RAC

Our Applications Technology Group recently published a matrix showing certified cloning scenarios for E-Business Suite databases running in Real Application Cluster (RAC) configurations.  Since then, some concerns have been raised about the lack of Rapid Clone support for EBS environments running on the 10g and 11g databases.

In response to those concerns, our Applications Technology Group has published a new whitepaper:

This new whitepaper describes a certified and supported method for manually cloning Apps 11i environments running 10g or 11g RAC.  The Note assumes a high degree of familiarity with Oracle Applications AD Utilities, RapidClone, AutoConfig, Recovery Manager, and SQL*Plus.

Related Articles

Friday Apr 17, 2009

Products and Families and Versions - Oh, My!

[Oct 1, 2010 Update: Tweaked the Apps Unlimited section and updated EBS release versions]

[Apr 15, 2009 Update: Added EBS version names and numbers, database version names and numbers, latest version numbers of other components, and new definitions for Applications Unlimited, Consolidated Updates, Critical Patch Collections, Release Update Packs, and the ever-confusing "RUP".  Added link to one-page summary of EBS Certifications]

I spend a depressing amount of time explaining the relationships between Oracle marketing brands, products, product families, versions, and patchsets to customers as well as internal Oracle staff.  You're confused too?  Don't worry, you're not alone.  Here's a cheatsheet for the things I spend the most time explaining:

Oracle E-Business Suite Release 11i or 12

An integrated suite of over 200 enterprise resource planning applications, including modules for Procurement, Accounts Payables, Accounts Receivables, Order Management, Payroll, Supply Chain Planning, Customer Call Centers, and many, many others.

Also Known As:  Oracle Applications, Oracle Apps, EBS, eBS, E-Biz, R11, 11i, R12, 12i (incorrect name!)

Oracle Application Server 10g

An integrated suite of development, runtime, and systems management tools, including Forms, JDeveloper, Oracle Application Server Containers for J2EE (OC4J), Single Sign-On, Oracle Internet Directory, Portal, Discoverer, Web Cache, Integration, Oracle BPEL Process Manager, Business Activity Monitoring, Enterprise Manager, and others.

Also Known As:  OracleAS 10g, Application Server 10g, App Server 10g, AS10g, 10gAS, AS10gR1, AS10gR2, AS10gR3, 10gR2, 10gR3

Oracle Database

Well, it's Oracle's flagship product, so if I need to describe it, we're in real trouble.  This includes the Real Application Clusters (RAC) feature.  Everyone frequently confuses the Database with the Application Server products.  If someone says, "10gR2," the chances are pretty good that they're talking about the database, but it's always safe to verify that.

Also Known As:  8i, 9i, 10gR1, 10gR2, 11gR1, 11gR2

Fusion Middleware

A family of middleware products including Oracle Application Server as well as Grid, Business Intelligence, Business Process Management, Collaboration, Content Management, Data Integration, Developer Tools, Event Driven Architecture, Service-Oriented Architecture, SOA Governance, Transaction Processing, Identity Management, and other middleware tools.

Also Known As:  FMW, OFM

Fusion Applications

The next-generation of our integrated enterprise resource planning suite, representing the convergence of Oracle E-Business Suite, PeopleSoft, JD Edwards, and perhaps even more to come. 

Also Known As:  Project Fusion, Fusion Apps

Applications Unlimited

This term officially refers to the lifetime support program for Oracle's applications product lines, including E-Business Suite, PeopleSoft Enterprise, Siebel, JD Edwards EnterpriseOne, JD Edwards World, Hyperion Performance Management, Primavera Enterprise Project Portfolio Management, Agile Product Lifecycle Management, AutoVue Enterprise Visualization, and Oracle Fusion Applications.  This term is sometimes incorrectly used to refer to all of these products as a group -- it's not a "group," it's a formal program with support and release implications).

Also Known As:  Apps Unlimited, AU

E-Business Suite Release and Patch Naming Conventions

"Product Families"   Groups of applications modules that are functionally related.  For example, Accounts Payable, Accounts Receivable, and Chart of Accounts are members of the Financials product family

"Emergency Patch" (a.k.a. interim patches)   A patch containing a fix for a specific bug for a specific product.  For example, Order Management might release patch 3968068 to fix a very tightly-defined bug.  Some emergency patches are released to fix a cluster of interrelated bugs.

"Product Mini-Pack"  A collection of bug fixes for a specific applications module.  For example, fixes for XML Publisher would be released in an XML Publisher mini-pack called 11i.XDO.H.

"Product Family Patchset"   A collection of product mini-packs for a specific, individual product family.  For example, fixes for Payroll, Benefits, and Training Administration would be released together in a Human Resources Suite Product Family Patchset called 11i.HR_PF.K.  New features are not supposed to be included in product family patchsets, but it happens. This is sometimes also called a Rollup Patchset, or RUP (see below).

"Recommended Patch List"  A list of individual patches for a specific product family that you should have applied.  These lists might include recommended emergency patches as well as product mini-packs.  If a product family recommends a patch via these lists, it's usually a very good idea to heed that.

"Critical Patch Collection"  This term was introduced for the EBS 12 codeline.  It includes the latest patches from the Recommended Patch List for a single EBS 12 product family.  If a product family (e.g. Financials) releases a new Critical Patch Collection, it's usually a very good idea to apply it at your earliest convenience.

"Maintenance Packs"   A comprehensive collection of all of the latest product family patchsets and new features.  For example, the Maintenance Pack includes product family patchsets for Financials, Procurement, HR, Supply Chain, and everything else in the E-Business Suite.

"Consolidated Updates"   A large collection of all of the latest EBS 11i recommended patch lists, including new features.  Released after a maintenance pack, such as the Consolidated Update for 11.5.10.  This term is used only for EBS 11i.  The EBS 12 equivalent term is "Release Update Pack."

"Release Update Packs"  A large collection of all of the latest EBS 12 recommended patch lists, including new features.  This term is used only for EBS 12.  The EBS 11i equivalent term is "Consolidated Update."

"Technology Stack Updates"  Any combination of patchsets or mini-packs that change the underlying services that product families depend upon.  For example,  the latest Applications Technology Family Pack is released in 11i.ATG_PF.H.

Generally, fixes to functional applications products like iReceivables don't require changes to the E-Business Suite technology stack, and vice versa.  There are exceptions to that, of course, but that's our general strategy.

"RUP"  This one's a bit confusing since this term is used and abused in varying ways even internally within the E-Business Suite division.  Bear with me on this:

Remember that in the E-Business Suite Release 11i timeframe, a given product team (e.g. the Applications Technology Group) would release a Product Family Patchset that includes all previously-released patches, emergency patches, and new features.  This was often called a Rollup Patchset, or RUP.

In the E-Business Suite Release 12 timeframe, the term "Release Update Pack" is being used in place of "Consolidated Updates".  R12 Release Update Packs combine patches created across several E-Business Suite product families. Given the way neologisms form, these became referred to in shortened form as "RUPs" too.

Still with me?  In EBS 11i, RUP referred to a single product family patchset, while in EBS 12, RUP generally refers to a consolidated update spanning multiple products.  Now brace yourself.  Here's the confusing part:

In EBS 12, we're also still releasing product family patchsets, and -- wait for it -- they're sometimes called rollup patchsets, too.  These R12 rollup patchset references will invariably be shortened to RUP, too.  So, there are two definitions of "RUP" even within the R12 codeline.

Tip:  If someone refers to a RUP in your presence, make sure that you clarify whether it's a "Release Update Pack" or a "Rollup Patchset."  The difference between the two is vast.

Oracle E-Business Suite Releases

Applications 10.7 Network Computing Architure (10.7 NCA)

Applications Release 11

  • 11.0.3

Oracle E-Business Suite Release 11i

  • 11.5.1 - 11.5.10
  • 11.5.10 Consolidated Update 1, 11.5.10.CU1 or
  • 11.5.10 Consolidated Update 2, 11.5.10.CU2 or

Oracle E-Business Suite Release 12

  • 12.0.0
  • 12.0.1
  • 12.0.2
  • 12.0.3
  • 12.0.4
  • 12.0.5 (HRMS and Financials only)
  • 12.0.6
  • 12.0.7 (HRMS only)
  • 12.0.8 (HRMS only)
  • 12.1.1
  • 12.1.2
  • 12.1.3

Oracle Application Server 10g Releases

OracleAS 10g Release 1 (10gR1)
  • Version
  • Version
  • Version
OracleAS 10g Release 2 (10gR2)
  • Version
  • Version
  • Version
OracleAS 10g Release 3 (10gR3)
  • Version 10.1.3

Oracle Database Server Releases

Oracle 8i

Oracle 9i Release 2 (9iR2)


Oracle 10g Release 1 (10gR1)


Oracle 10g Release 2 (10gR2)


Oracle 11g Release 1 (11gR1)

Patch Compatibility and Certification Matrix

Here's where I lose the three remaining readers of this post.

Only specific versions of Oracle products work together.  It's critical to check whether a specific patch works with your configuration.

It's a tricky system to use, but the final word on all supported configurations is captured in a massive database called Certify.  To access this system, log on to MetaLink and click the Certify tab in the upper-right corner.

It's easy to get lost in Certify, and it's sometimes very hard to get an answer to a simple question.  If you get stuck, the best route is to log a Service Request and let an Oracle Support Engineer wade through the Certify database for you.

A Simplified Version of Certify

Certify is the official repository for all Oracle certifications.  It's tricky to use, so you may find a simplified summary of all E-Business Suite technology stack certifications useful.  Hit the "Certifications" link in the menubar above, or just navigate to it directly here:

Whew.  Let's move on to more interesting topics; this one felt too much like real work.

Related Articles

Friday Feb 20, 2009

On Database Patching and Support: A Primer for E-Business Suite Users

The Oracle Server Technologies division has issued some important updates to their support policies in the following document:

These changes affect support policies for the database, Oracle Enterprise Manager, Fusion Middleware, and Collaboration Suite.  These changes are important enough to warrant an in-depth discussion about the implications of the database-related updates for E-Business Suite customers.  This article also discusses the E-Business Suite database certification process and the safety of applying interim patches to your Apps environments.  I'll cover the Apps-specific implications for the other technology products in a future article.

[Read More]

Thursday Dec 11, 2008

Oracle Configuration Manager Natively Supported With EBS 12

Oracle Configuration Manager (OCM) is a tool that automatically gathers configuration information from Oracle product installs and upload this information onto Oracle’s Support systems.  Customer support engineers can use this data to improve resolution times for your Service Requests.  Joshua Solomin, the OCM Product Manager, discusses the tool's benefits and the data that it collects in this article:

Screenshot of My Support page showing OCM inventory

Oracle Configuration Manager is now natively supported with Oracle E-Business Suite Release 12. Starting with OCM 10.2.7, OCM provides native support for shared ORACLE_HOMEs. This makes possible to configure the latest version of OCM with E-Business Suite instances.  EBS 12 customers can take advantage of the benefits provided by Oracle Configuration Manager without worrying about whether their Application Server and Database Oracle Homes are part of an E-Business Suite Instance. There are no EBS-specific configurations requirements with OCM 10.2.7 or higher.

[Read More]

Thursday Dec 04, 2008

New Whitepaper: Options for Reducing E-Business Suite Database Sizes

I have yet to encounter a database that ever got smaller.  Like waistlines and the US national debt, all databases seem destined to increase in size.  The E-Business Suite is no exception:  as we add more product capabilities and your business grows, so do your Apps databases.

Oracle-supplied solutions to managing Applications database size fall into two categories:  data growth control methods and data management methods.

Growth Control Methods

  • Archiving and purging
  • Database compression

Data Management Methods

A new Oracle whitepaper discussing these topics is now available:

Screenshot of Oracle Information Lifecycle Management Assistant used with E-Business Suite database tables

[Read More]

Tuesday Nov 18, 2008

Advanced Deployment Architectures for Oracle E-Business Suite (OpenWorld 2008 Recap)

I'm (still) highlighting OpenWorld 2008 presentations that cover some of the most popular E-Business Suite technology stack topics. A catalog of all of the Applications Technology track sessions with links to the presentations is available here:

E-Business Suite sysadmins know that there are a lot of different ways to deploy their system in production.  You can split EBS services across multiple application tier and database tier server nodes, you can scale up with load-balancers and Real Application Clusters, you can integrate your E-Business Suite instance with optional external services like Oracle Single Sign-On and the Oracle SOA Suite, and much, much more.

Sample physical architecture diagram showing E-Business Suite integrated with Single Sign-On and Oracle Internet Directory with firewalls separating DMZs

The number of architectural options can be pretty bewildering, and it can be difficult to get a high-level overview of the relative benefits of each option.  We have lots of detailed documentation and introductory blog articles on, say, implementing RAC, but it's very difficult to get a sense of whether you can combine a reverse proxy in front of a load-balanced cluster with a RAC-enabled database tier (this is feasible, by the way).

[Read More]

Wednesday Oct 29, 2008

EBS 12: Install + Cloning Techniques Deep Dive (OpenWorld 2008 Recap)

I'm highlighting OpenWorld 2008 presentations that cover some of the most popular E-Business Suite technology stack topics. A catalog of all of the Applications Technology track sessions with links to the presentations is available here:

If you're looking for a good overview of how Oracle E-Business Suite Release 12 environments can be cloned, you should check out the following presentation from Max Arderius, Development Manager in the Applications Technology Group, and Biju Mohan, Product Manager in the Applications Technology Group:

Graphic showing differences between Rapid Clone and automated cloning via Applications Management Pack AMP

[Read More]

Tuesday Oct 28, 2008

E-Business Suite Technology Essentials (OpenWorld 2008 Recap)

I'll be highlighting OpenWorld 2008 presentations that cover some of the most popular E-Business Suite technology stack topics.  A catalog of all of the Applications Technology track sessions with links to the presentations is available here:

The E-Business Suite technology stack can be a bit intimidating at first.  Lisa Parekh, Vice President of Applications Technology Integration, provides a crisp overview of the key technologies and concepts that every Apps manager or DBA should be familiar with in this presentation:


[Read More]

Monday Oct 13, 2008

Every Current E-Business Techstack Certification on One Page


I started this blog over two years ago as an experiment.  My goal: make it easier to find information about the E-Business Suite that was already published somewhere in the vast system that is Oracle Metalink.

Metalink has since received a dramatic facelift and has been renamed to My Oracle Support.  Good news, but an upgrade to one key part of that system is still pending: the Certify database.

The Certify database documents all possible combinations of all Oracle products with all possible operating systems. This is both the strength and Achilles heel of the Certify system.  It's comprehensive, but its sheer size can make it hard find a specific piece of information — even for us Oracle insiders.

It's time for another experiment.

[Read More]

Tuesday Oct 07, 2008

Three Options for Scaling Up E-Business Suite for Reporting

[Oct 10, 2008 Update: Added link to article comparing the use of Oracle Active Data Guard with Oracle Data Guard for EBS environments]

The run-up to our annual OpenWorld conference consists of frenzied activities to ensure that all of our planned certifications wrap up in time to be announced at the conference.  The follow-up from OpenWorld consists of handling questions, bug reports, and escalations from our sessions, panels, and private customer meetings.  Given that this is all on top of our regular day jobs, one day I'm going to print up some t-shirts that say, "I survived another Oracle OpenWorld."

So, back in the blogging saddle again. I'll address one of the architectural questions that seems to pop perennially:

How do I handle heavy reporting overhead without disrupting my E-Business Suite instance's transactional users?  Can I offload this to a separate reporting instance?

[Read More]

Wednesday Aug 27, 2008

Reducing Patching Downtimes with Staged Applications Systems


It's axiomatic that everyone wants to minimize maintenance downtimes for their E-Business Suite environments. This is particularly crucial for environments with users in multiple timezones. I've previously summarized some of the most-effective ways of reducing patching downtimes for E-Business Suite environments. As noted in that article, one of the best ways of minimizing your maintenance downtimes is to use a staged Applications system.

The staged approach allows you to perform as many changes as possible in an offline Apps environment, and defers taking down your production environment only for the final database patches tasks. Using this approach, you apply your new patches to an exact clone of your production E-Business Suite environment. This can be done while your production system is still running. The staged Applications environment is then used to run database updates and APPL_TOP changes into your production environment.

[Read More]

Wednesday Aug 06, 2008

In-Depth: Using Third-Party Identity Managers with E-Business Suite Release 12

This article is an updated R12 version of an earlier one written for Oracle E-Business Suite Release 11i.

Like most of our customers, you probably already have a corporate identity management system in place. And, you've probably not been enjoying the experience of redundantly administering the same user in your corporate identity management system as well as the E-Business Suite. If this describes your environment, this in-depth article about integrating Oracle E-Business Suite Release 12, Oracle Single Sign-On and Oracle Internet Directory with third-party identity management systems will show you a better way of managing your EBS users.

[Read More]

Tuesday Jul 15, 2008

Case Study: Oracle's Own Oracle E-Business Suite Release 12 Upgrade

[Oct. 27, 2008 Update:  The latest version of this popular presentation from OpenWorld 2008 is now available for download.  For links to the latest version, see this article.]

I've heard anecdotal reports suggesting that some customers hold off on upgrading to a given E-Business Suite release until we've done so ourselves here at Oracle. Oracle went live on R12.0.3 in January 2008, and a reader reminded me that I haven't highlighted this adequately. Here's a critical presentation from Eugene Weinstein and Sharon Leong at OAUG Collaborate 2008 (Denver) that I've been remiss in profiling:

Eugene and Sharon cover a lot of ground in this technically-oriented presentation, including:

  • A primer on the R12 file system
  • Supported upgrade paths from earlier Apps releases (11.5.x, 11.0)
  • A detailed step-by-step flowchart of the upgrade process
  • Applications DBA (AD) improvements relating to the upgrade process
  • Performance improvements relating to the upgrade
  • Best practices for:
  • Reducing downtime
  • Performing pre-upgrade, upgrade, and post-upgrade steps
[Read More]

Friday Jul 11, 2008

New Whitepaper: Mod_plsql and E-Business Suite 12

Mod_plsql is an Apache web server extension that can be used to develop web application pages using Server PL/SQL. Architecture diagram showing flow from client to mod_plsql Apache mod to Oracle database The Past is Prologue Unlike Oracle E-Business Suite Release 11i, Release 12 does not include mod_plsql as part of its technology stack. I've briefly discussed this architectural change in the following two articles: It should be stressed that Oracle is fully committed to supporting mod_plsql as part of Oracle Application Server and as part of the Oracle Database distribution into the indefinite future. The Oracle E-Business Suite is distinct from Oracle Application Server. Oracle E-Business Suite Development chooses to use specific Oracle Application Server components in the E-Business Suite technology stack. These decisions by E-Business Suite Development should not be interpreted to represent the release policies or plans for Oracle Application Server. Going Into More Detail Many of you have raised questions about why mod_plsql was removed from Release 12. Others have asked what to do about their mod_plsql-based Apps 11i customizations and extensions when upgrading to R12. George Buzsaki,our preeminent E-Business Suite architect, has put together an excellent new whitepaper that addresses these topics, and more: [Read More]

Thursday Jun 12, 2008

New Whitepaper: Best Practices for Adopting E-Business Suite Release 12 (First Edition)

A colleague has just pulled off an impressive feat that I wouldn't have attempted myself:  she's collected practical tips and advice on how to do Oracle E-Business Suite Release 12 implementations and upgrades.  She's consolidated input from Oracle's Support, Consulting, IT, and Development groups into a new whitepaper:

This whitepaper is mandatory reading if you're planning -- or in the middle of -- an Oracle E-Business Suite Release 12 deployment.  The whitepaper has a mix of concrete and strategic advice that covers topics such as:

[Read More]

Tuesday May 27, 2008

New Whitepaper: Database Partitioning for the E-Business Suite

Some readers complain that we don't have sufficient documentation to cover all possible scenarios and topics of interest.  This is a valid observation.  As your E-Business Suite deployments grow in complexity and scope, keeping ahead of your questions and new requirements is a constant challenge.

In my position as the editor of this blog, something I do only in my so-called free time, my situation is the odd inverse of yours, namely:  the rate at which we release new Metalink Notes far-outstrips my capacity to read and announce them to the world.  Here's a much-belated announcement about a database partitioning whitepaper produced by our Applications Performance Group.

What Is Database Partitioning?

Partitioning allows a single database table and its associated indexes to be broken into smaller components depending on the table and choice of index partitioning methods.  Several E-Business Suite modules take advantage of database partitioning right out of the box, and custom partitioning is also fully supported.  I've covered database partitioning concepts for Apps environments in more depth in this older article.

Database Partitioning Methods:

Best Practices for Partitioning Apps Databases

As I've noted before, we have a group in our Applications Development division that's dedicated to optimizing the E-Business Suite's performance.  As a member of this Applications Performance group, Mohsin Sameen has worked extensively with some of our enterprise-class customers -- including many of the largest companies in the world -- on fine-tuning the performance of of their high-volume Apps environments.

Mohsin has distilled these experiences into an extensive and in-depth paper on database partitioning:
Mohsin's excellent whitepaper covers topics such as:
  • Overview of database partitioning concepts
  • Table partitioning strategies involving range, list, hash, composite, and multi-column partitions
  • Index partitioning methods, including global and local partitioned indexes
  • Step-by-step decision framework for using partitions
  • Partition maintenance operations
  • Partitioning case study
The knee-jerk reaction answer to a performance problem is often to throw more hardware at it.  If you have a large E-Business Suite environment where the growth rate of your historical transactional data is starting to affect performance, I'd strongly recommend reading this whitepaper.  It's entirely possible that you could use it to squeeze some additional performance out of your existing environment without the added expense of new hardware.

Related Articles

Tuesday Apr 22, 2008

New Whitepaper: E-Business Suite Development Using OAF & ADF

Customers, partners and system integrators often develop extensions to Oracle E-Business Suite (EBS) applications. Such extensions have traditionally used the same technology stack (Forms or OA Framework) that the original EBS application was built with.  This ensured that the extensions are fully compatible with the rest of the installed EBS applications.

R12 OAF Techstack: Block diagram showing the OA Framework Technology Stack for E-Business Suite Release 12

With the emergence of the next generation Fusion Middleware technology stack, especially the Application Develoment Framework (ADF), an increasingly-common question is whether to use OAF or ADF to develop E-Business Suite Release 12 extensions.

Our E-Business Suite Applications Technology group has released a long-awaited whitepaper addressing this question. This whitepaper discusses the similarities, differences and overlaps between the OAF and ADF stacks. 

If you're considering these technology stacks for your Apps R12 extensions, I'd strongly recommend reviewing this whitepaper:
Related Articles

Tuesday Jul 24, 2007

10 Tips for Protecting your APPS Password

E-Business Suite security is a huge topic, as there are many different facets to consider. This article will consider a small but essential part of the security model: protecting your APPS user password.

The APPS user in the E-Business Suite is the master of its world. I have gathered together some thoughts on steps you can take and things to consider to help you protect your APPS password from being compromised:

  1. Stay current with our latest Security Best Practices

    Regularly review the latest version of Best Practices for Securing Oracle E-Business Suite (Note 189367.1). This note is regularly updated and will give security advice covering many different aspects of Applications 11i.    For Release 12, see Best Practices For Securing Oracle E-Business Suite Release 12  (Note 403537.1)

  2. Regularly change your APPS password

    This is an essential activity from a security perspective and needs to be part of your routine operating procedures. Same applies for other schema passwords and SYSADMIN user. As an aside, don't use predicable passwords, or have a system to create passwords, such as using "0ct0ber" for the password in October as this will make it easier to guess

  3. Always change passwords as part of a clone process from PROD

    It is recomended to change ALL schema passwords and ALL eBiz user passwords in a cloned instance.   You can use Removing Credentials from a Cloned EBS Production Database (Note 419475.1) to achieve this.    Similarly, you don't want to have any relation in the passwords used for PROD compared to any other instances. Data masking and obfuscation is a large topic in its own right, but is also something you may need to consider doing for the cloned instance to protect sensitive data generally. With Release 12, EM plugin provides some data scrambling facilities.

  4. Perform data masking on any files sent to outside parties from the PROD system

    When you need to send any log files or configuration files, ensure that you scan for any sensitive data before packing the files to be sent. In this article we are concerned about the APPS password, but this applies equally well for other data as well. For example, a crude mechanism would be to use "ed" or "sed" on all files to globally change any occurrences of the APPS password before creating a tar archive to email or upload. You may be uploading files to Oracle Support, or just emailing them within your Organization. Whenever the files are going to someone who cannot access them directly you should always check the files before sending.

  5. Create separate schemas with minimal access required for direct database access  

    If anyone requires direct access to the E-Business Suite database, ensure that you create a new unique schema with the specific permissions required for them to perform their job role. Except for a very few Apps DBAs, there should be no reason that anyone else needs the APPS user password. Sometimes pressures of work make it easier to just give someone APPS access, but this should be resisted and the time taken to provide only the minimum access absolutely required. Every person should also have their own unique login (but this is digressing into a separate area that I'll address in a later article).   When considering permissions to allocate, don't be tempted to give read only access to everything, as being able to read sensitive information may be just as damaging as being able to change it.

  6. Protect Apps 11i middle tier file system files

    These days, there is little need to give anyone UNIX-level access to the servers, but it is still important to ensure the "applmgr" operating system user password is well protected. Also consider whether any of your own startup scripts or monitoring scripts have the APPS password hard coded in them, and protect these scripts with chmod 700 permissions, or remove them if no longer needed

  7. Ensure no processes are running with APPS username/password in command line

    Generally the APPS password is not listed in "ps" output, but there may be some manual scripts or other processes intermittently running with the APPS password in clear text or trivially encoded. Ensure these scripts are changed to hide the APPS password. In addition, ensure operating system access is restricted to only those who really need it

  8. Protect OID access

    If you have integrated the E-Business Suite with Oracle Application Server 10g, Single Sign-On, and Oracle Internet Directory, then the Apps user password is stored in the OID database, as it is required for Provisioning to function. The OID administrator and anyone with ldapsearch rights in the Provisioning Profiles will be able to extract the APPS password from OID. This in turn implies the "AppsDN" OID password should be protected in the same way as the APPS password itself.    For assistance in security hardening OID, refer to the Oracle Internet Directory Administrator's Guide 10g ( - Part III Directory Security

  9. Encrypt SQLNET traffic from Middle Tier to RDBMS

    In a previous article, Steven highlighted that ANO is certified with the E-Business Suite. Use encryption to protect the APPS password from network sniffers tracing SQLNET connection packets and deciphering the APPS password on the wire.

  10. Allow only specific IP addresses to access RDBMS via SQLNET

    Slightly off topic, but if someone has acquired the APPS password they still have to be able to gain access to a tool that can use it. Restricting the IP addresses that can access your Apps database will help minimise this risk. If you are still using "fat" clients (Discoverer or ADI for example) then you will have to weigh up the risks against the administrative overhead. Oracle recommends upgrading to server-based equivalent tools or shared desktop technologies such as Citrix so desktop clients no longer need direct access.  This topic is discussed further in E-Business Suite Recommended Set Up for Client/Server Products (Note 277535.1)


Defence in depth is generally considered the best approach so hopefully these recommendations will give you some food for thought when you are reviewing how well your own system is protected.

Sound password policies are critical to enforce access policies and enforce individual accountability.  You need to jealously guard your passwords, particularly for the APPS user.


Thursday Jun 28, 2007

OA Framework or ADF?

[Editor Update 4/22/2008:  A new whitepaper from the Oracle Appliations Technology Group comparing OAF and ADF is now available.  See this article for more details.]

[July 9, 2007 - see "One more thing to consider", below - Sara]

A question I've been getting recently is "Should I use OA Framework (OAF) or the Oracle Application Development Framework (ADF) to develop extensions to E-Business Suite (EBS) products?  I know we are moving away from Oracle Forms, and I want to 'future-proof' my extensions as much as possible."

With all the buzz about Fusion (both the coming Fusion Applications and the current Fusion Middleware technology stack) and JDeveloper with ADF, it's hard not to get excited about using all the features of the latest versions of Fusion Middleware.  So would we recommend using OAF or ADF?

My general advice is to stick with OAF so long as you are working with the E-Business Suite, and wait until you move to the Fusion Applications before moving to ADF.  OAF is the development platform for the HTML EBS applications, so it makes sense to use the same development platform for custom extensions to the EBS applications.  OAF provides you with automatic, seamless integration with the EBS applications in terms of EBS menus, security, look and feel, flexfields, personalization, attachments, and so on.  These integrated EBS features aren't part of ADF.

Another part of the integrated OAF/EBS technology stack is Oracle Application Object Library (AOL), which provides many services to OAF, including security, messages, flexfields, and so on.  Developers using OAF also use the OA Extension, which is a JDeveloper add-in bundled with JDeveloper in specific EBS development environment patches.  OA Extension is an extension to JDeveloper 9.0.3 for EBS 11i, and to JDeveloper 10g (10.1.3) for EBS 12.  AOL and OA Extension are not parts of the ADF technology stack for the Fusion Applications.

A terminology note: "ADF UIX" is the same UIX that is used by OAF as the HTML UI display component.  Developers using OAF use ADF UIX indirectly by using OAF components.  ADF UIX is not part of the "ADF" Fusion Applications technology stack that I am discussing here. "ADF Faces" is the UI component of the ADF technology stack. 

Going It Alone

If you intend to build a completely standalone application that just happens to use the EBS tables, you can certainly use ADF, but you'd need to deal with things like security, look and feel, and so on yourself.  You also wouldn't have the personalization, flexfields, attachments, or other infrastructure that OAF provides.

Using ADF to build integrated extensions to the E-Biz Suite is an untested combination, so you won't find much experienced help available if you take that path.  Also, ADF isn't certified for use with E-Business Suite (EBS), and there are no plans for this certification.  ADF is a standalone development tool and is expected to be a key part of the Fusion Applications platform.  It's a very different technology stack from OAF, other than the business components.  Business Components for Java (BC4J) becomes ADF Business Components (ADF BC). 

Your Business Logic Moves Forward

Business logic that you create now with BC4J with the OAF technology stack will be transferable to the Fusion Applications technology stack later (possibly with minor changes).  If you follow our OAF coding standards, build your pages declaratively as much as possible, and avoid putting much code into controllers if you can, you'll have a much easier time moving to Fusion later.  All your business logic, validation, and so on should be in your BC4J objects.

Technology Directions

One further note: it's part of the general EBS Applications Technology and OAF direction (as part of Applications Unlimited), that we want to bring as much "Fusion technology" into the EBS applications as possible in our ongoing releases.  So over time, the technology stack gap will likely get smaller, and customers will get to use some of the newer technology within the E-Biz Suite until they are ready to move to the Fusion Applications. 

One example of this direction is that we now use JDeveloper 10g (10.1.3) for Release 12, instead of JDeveloper 9.0.3, which is used with Release 11.5.10.  So customers can now take advantage of new JDeveloper 10g features such as a much better Java coding/editing environment.  JDeveloper 10.1.3 can also dynamically recognize which files are in your path for a project.  One of my correspondents thinks that feature alone is a good reason to use 10.1.3!

Happy extending!

[July 9, 2007 - One more thing to consider: how big is the project?  If you are embarking on small extensions, a few pages here and there that you want to integrate into the existing applications, it's better to use the existing tool for that product line (EBS, Peoplesoft, etc.).  If you are building entirely new, monolithic applications (large applications with many pages),  where seamless integration with the current application pages isn't so important, you may want to use ADF (with ADF BC and ADF Faces).  Then you won't need to convert your custom applications when you move to the Fusion Applications later.]


Tuesday Jun 19, 2007

Top 7 Ways of Reducing Patching Downtimes for Apps

[June 20 Update:  Our blogging software temporarily lost its mind and somehow misnumbered the list at the heart of this article.  There are only seven tips here, not ten.]

I've previously discussed the Top 5 Myths About Patching Apps Environments.  One of the most commonly-cited reasons for not patching E-Business Suite environments is that it takes too much downtime.  If you're relatively new to Apps system administration, here's a primer on the key techniques compiled by our Applications Release Engineering group for reducing patching downtimes.  Although some of the linked documents are written specifically for Release 11i, these techniques may be used for both Apps 11i and 12.

  1. Use a staged applications system

    This major time-saver hinges on a key principle:  all of your applications filesystem patches are applied to a clone of your production Apps environment.  This can be done while your production system is still running.  Your production system is down only for the time needed to apply database patches.  For details, see:
  2. Use a shared application-tier file system

    If you have a pool of application-tier servers set up for load-balancing, make sure that all of the individual servers share a single application filesystem.  Patches applied to this central shared filesystem are instantly available to all application-tier servers.  I've previously given an overview of this technique in this article.

  3. Distribute worker processes across multiple servers

    When applying a patch that includes a large number of processes, you can reduce the downtime even further by distributing the worker processes across multiple servers on multiple nodes. Using the Distributed AD feature of AutoPatch and AD Controller, you can assign workers to run on the primary node and on other nodes that share the filesystem. See:

    Distributed AD (Metalink Note 236469.1)

  4. Merge multiple patches using AD Merge Patch

    Merging patches saves time because the AutoPatch overhead of starting a new session is eliminated for those patches that are consolidated.  Duplicate linking, generating or database actions are run once only.  If two patches update the same file, AD Merge Patch will save time by applying only the latest one.  Patches can -- and should -- be merged with their listed prerequisite patches. 

    For more details about this AD utility, see the Oracle Applications Maintenance Procedures guide for your Apps release.

  5. Run AD Patch in non-interactive mode

    Applying a set of patches using AD Patch in non-interactive mode eliminates the delay between successive tasks.

  6. Defer system-wide database tasks until the end

    Using adpatch options=nocompiledb,nomaintainmrc defers system-wide database tasks such as "Compile APPS schema" and "Maintain MRC" until after all patches have been applied.  As of AD.H, AutoPatch automatically compiles the APPS schema and maintains MRC when applying standard patches.
  7. Avoid resource-related bottlenecks

    Patching can grind to a halt if you bump into the ceiling on your system.  Before patching, make sure that you've enabled automatic tablespace management, and that you have sufficient hardware and free disk and temp space.
The cumulative downtime reductions of all of these techniques can be quite significant.  I've touched on some of the biggest timesavers, but this short article isn't comprehensive, by any means.  The linked Notes in this article and below discuss a number of other tips for shrinking your patching downtimes.  A small investment in learning these techniques can pay off in large reductions in patching times, and is well worth your time.


Thursday Jun 07, 2007

Top 5 Myths About Patching Apps Environments

When I was younger, I thought I could change this world.  Now I no longer think so but for emotional reasons I must keep on fighting a holding action.
~ Robert Anson Heinlein

A rational person might contend that actions follow attitudes which follow beliefs.  Bad news:  there's a substantial amount of psychological literature that suggests that this isn't how we really tick.

There's an equally large body of empirical data that suggest that beliefs are relatively intractable.  Once established, certain beliefs don't change, regardless of the best data, clear reasoning, and eloquence that can brought to bear on the subject.  Recent fMRI studies suggest that brain structures may develop in such a way that people with strongly-held beliefs are actually unable to process new information that contradicts those beliefs.

The broader implications of this are kind of depressing to contemplate.  But I digress.

For the narrow purposes of this discussion, I'm compelled to make an admittedly-quixotic attempt to persuade you that the benefits of keeping your E-Business Suite environment up-to-date far outweigh the costs.

"We Can't Upgrade Because..."
  1. It requires too much downtime
  2. Testing is too expensive for end-users
  3. It's too complicated
  4. We don't have enough staff
  5. It ain't broken; why fix it?
Myth #1:  It Requires Too Much Downtime

This seems rational on the surface.  After all, a downtime for patching seems worse than no downtimes.  

Remember that we issue patches for four major reasons: 
  1. To add new functionality
  2. To improve stability
  3. To improve performance
  4. To improve security
The corollary is that an unpatched system may have fewer or less-sophisticated features, and be slower, less stable, and less secure than a fully-patched one.  Questions to consider if you believe this myth:
  • How much downtime is currently caused by unplanned outages due to unpatched stability bugs?
  • How much downtime is needed to bounce your application servers due to unpatched JVM memory leaks?
  • How much downtime would be required if an attacker takes advantage of an unpatched security risk?
There are a number of key ways to reduce downtimes.  One of these is to use shared filesystems in environments with multiple application servers.  There are a number of other ways, too.  These are beyond the scope of this article, so I've covered the top seven way of reducing downtimes in this article.

Myth #2:  Testing is Too Expensive For End-Users

Many organizations recruit business users to participate in User Acceptance Tests for patches.  If you do this, it's true that this may reduce participants' productivity for the duration of your testing phase. 

If you don't keep your environment up-to-date, the key consideration are: 
  • How much productivity is lost for all end-users -- not just the testers -- due to unpatched performance, stability, or security bugs?
  • How much could productivity be improved by new features?

Myth #3:  It's Too Complicated

All DBAs know the drill:  before patching, print all patch READMEs and spread them in neat piles across a big surface.  Read them, highlight them, then reread them.  Download more patches.  Repeat. 

Every patch has some set of prerequisites.  These prerequisites lead to other prerequisites.  The key thing to remember is that it's always easier to patch an up-to-date system than something that's much older.  The bigger the gap between your current system and target patch level, the more complex the upgrade will be. 

The best way of reducing the complexity of upgrades is to keep your system up-to-date.

Myth #4:  We Don't Have Enough Staff

I need to be candid here:  there's a kernel of truth in this one.  This is a direct outcome of believing Myth #3.  If you haven't patched your E-Business Suite Release 11i environment since it was installed in 2001, then you may have a big and complex upgrade project on your hands.  This might take more staff than you have.

So, the best way of avoiding the truth of this one is to keep up-to-date with your patching.  If you're up-to-date, applying a given patch is less complex and requires fewer staff.

Myth #5:   It Ain't Broken; Why Fix It?

With apologies to English teachers everywhere, that old axiom has a seductive ring of truth to it.  After all, your system seems to be running just fine, thank you.  Why bother potentially destabilizing something that everyone's happy with?

Repeating the key points made in Myth #1:  new releases are issued to provide new functionality and improve stability, performance, and security.  If you're running an older release, it has -- by definition -- issues in these areas that you may not have noticed yet.

The Worst-Case Scenario

If I were an IT manager running an older E-Business Suite release, say 11.5.2, this would be the nightmare scenario that would keep me awake at night:

My business requirements changes due to an acquisition or a change in business practices.  These changes trigger a different load or usage profile on my E-Business Suite environment.  This triggers a Severity 1 outage.  My production system is down.

I call Oracle Support for help.  Good news: they can reproduce the problem on 11.5.10.CU2.  Bad news: due to technical dependencies (and supported by the fact that 11.5.2 is out of Premier Support status), they can only issue patches on top of 11.5.10.CU2.  There's no way of getting that patch backported to 11.5.2.

Now I'm faced with a huge upgrade from 11.5.2 to 11.5.10.CU2.  My production system is still down, and it will be down for the duration.  Users are burning me in effigy in the parking lot.  Business impact analysis, user acceptance testing, load-testing, fail-over testing -- all of those activities are jettisoned in my panicked attempts to get my environment running again.

It's Never Too Late

At this point, I suspect I've lost most of my readers.  By now, they've clicked on the latest YouTube video or something more cheerful. 

For the handful of my remaining readers:  if you're in a hole, it's never too late to stop digging.  Less metaphorically:  just because you're behind in patching doesn't mean that you should just give up entirely. 

We can help you put together a strategy for getting up-to-date.  Your first call should be to your Oracle account manager.  He or she can engage specialists in Sales Consulting or Consulting to help.  There are also excellent groups in our Advanced Customer Services organization (a.k.a. Field Support, On-Site Support, Premium Support) who offer a number of of packaged and customizable support offerings in the patching realm. Alternately, there are a number of excellent third-party consultants who can help you with this.  Or, if you're really intent on doing it yourself, there are forums on the Oracle Technology Network and elsewhere where you can brainstorm possible upgrade strategies.

Whatever path you choose:  start now.  It's never too late.


Thursday May 24, 2007

Virtualization and the E-Business Suite, Redux

Operating system vendors such as Hewlett Packard, IBM and Sun are increasingly promoting the use of their own virtualization solutions for reasons primarily related to the need for IT consolidation and better hardware utilization.

For instance, Solaris Containers is a feature of Solaris 10 that allows partitioning of an existing operating system into separate virtual hosts:

Sun Solaris Containers:

IBM promotes the use of its Dynamic Logical Partitioning (LPAR) technology that enables the virtualization of hardware resources which can be shared by multiple operating systems:

IBM LPAR overview:

HP's virtualization solution -- Virtual Server Environment -- includes various technologies such as nPartitions, vPars and Integrity VM that allow running multiple instances of HP-UX on the same server:

HP Virtual Server Environment:

Additionally other vendor-neutral solutions such as VMware, Microsoft Virtual Server, and Citrix allow different operating systems to run as guest operating systems on a single physical machine.  I've briefly discussed our support for these types of virtualization solutions in a previous article, with a special case for E-Business Suite client/server modules which make direct connections to the E-Business Suite database.

The use of operating system vendors' virtualization technologies to host E-Business Suite falls under the same 'not explicitly certified, but supported' category. These technologies are covered by Oracle's standard policy for third-party product support
  • Oracle will triage and attempt to diagnose issues reported for these configurations. 
  • Specific problems isolated to virtualization software that cannot be reproduced in standard Oracle environments -- i.e. environments without virtualization software -- may need to be referred to the third-party vendor for advanced debugging and resolution.
Use in Production Environments

If you plan to use virtualization software for your application and database servers in a production environment, the usual advice applies:  conduct thorough functional tests, perform peak load-testing, and have detailed fallback plans in case of issues with production environments.


Monday May 21, 2007

Preventing Apps 11i Performance Issues in Four Steps

In my previous article, I talked about what to look for once you have a performance issue with Apps 11i.  In this article, I'll discuss four maintenance activities that you can do proactively to reduce the chances of encountering certain types of performance issues:

  1. Check performance against a baseline
  2. Follow a regular purging schedule
  3. Gather schema statistics regularly
  4. Follow a systematic pinning strategy


1.  Check Performance Against a Baseline

  • Create a baseline so you can monitor performance in response to changes or over time. For example, you may create 6-10 repeatable short transactions of at least 10 seconds each, which represent common functions and/or areas of particular concern.
  • Always execute this baseline test from the same PC and the same location in order to get consistent results.
  • Rerun the test as part of your normal User Acceptance Testing for any system changes.
  • Review and update your performance baseline as part of any upgrade project.

2.  Follow a Regular Purging Schedule

You need to ensure that any data that needs to be purged is scheduled on a regular basis. The data that can (or should) be purged will vary between different products, so confirm the recommendations for the specific products you are using.  All customers will generally need to schedule purging activities for the FND and Workflow products.

Concurrent Jobs to Purge Data

Most customers will need to schedule these Concurrent Purge processes:

  • Purge Obsolete Workflow Runtime Data (FND)
  • Purge Debug Log and System Alerts (FND)
  • Purge Signon Audit data (FND)
  • Workflow Control Queue Cleanup (FND)
  • Delete Data from Temporary Tables (ICX)
Naturally, your business requirements may be unique; review our purging documentation for suitability and establish your own list of jobs to run.

Workflow Specific Purging Tips

Review the following documents for Workflow-specific tips on purging:
Also keep track of number of rows in underlying Workflow tables to ensure they are not continually increasing, to ensure the data really is being purged

3.  Gather Schema Statistics Regularly

In general, running this monthly to bi-weekly should be sufficient with 10%, unless there is any known data skew. As with any generic suggestion, this would need to be proven for suitability on your own environment. For example, it is more important to run this when your data distribution changes, rather than when the amount of data changes.

If your environment is a 24x7 system, you should pick "N" for "Invalidate Dependent Cursors" to prevent fragmentation of the shared pool

For more details about gathering statistics, see: 4. Follow a Systematic Pinning Strategy

Despite new 10g features making ORA-4031 errors a rare occurrence, it is still recommended to have a pinning strategy, even with Apps 11.5.10 running on 10gR2 databases.

General guidance

  • Monitor X$KSMLRU for candidates to pin (> 4100 bytes)
  • Do not pin more than 20% of the Shared Pool
  • Review your Pinning Strategy for changing business cycle, such as month end or overnight batch runs
  • No need to pin objects used only for batch jobs
  • Also have an "un-pinning" strategy
For more information about pinning in Apps environments, see:


Running regular maintenance tasks and ongoing monitoring are essential activities to ensure that your system is performing to the best of its ability. This article highlights a few of the areas that are sometimes overlooked in Apps DBA schedules.


Friday May 18, 2007

Debugging General Performance Issues with Oracle Apps

Identifying performance bottlenecks can sometimes be a black art with any distributed computing system.  It is sometimes difficult to know where to start.  This article gives some high-level guidance on the sort of information you may need to gather in order for Oracle Support to assist you with this task for the E-Business Suite.

Performance issues can potentially occur across any or several different areas of the Technology Stack, or may be restricted within Functional Code. For example (but not restricted to)

  • Architecture issue (e.g. high latency WAN, firewall)
  • Operating System problem or resource constraint
  • SQL or general RDBMS configuration issue
  • Database Deadlock
  • Apache Listener


Types of Performance Issues

"Simpler" Problems

These issues will hopefully be relatively straightforward to define and investigate.  For example: -
Single report consistently slow, SQL trace discovers one or more SQL statement(s) taking the majority of the elapsed time

More complex issues

These can take more work to be able to confidently define the real issue and often involve complex investigations involving different parties and much data gathering and analysis

For example :-
Any intermittent issue, System wide issues, Issues where SQL trace time represents only a small proportion of the elapsed time

Gathering Data for Performance Issues

It's helpful to follow a systematic process for investigating performance issues.  First steps include identify the nature, scope and extent of the problem. 

1. Identify the Extent of the Problem

Determine whether the extent of the problem is:

  • System Wide
  • Confined to one Technology Are. For example does it only happen in one of the Self Service, Forms or Reports areas
  • For a specific Product Area(s) For example, were it to only occur in HR, GL or iStore
  • Single Report/Form/Page
  • Which Instance or instances is the problem observed. Can it be reproduced in UAT/TEST instances

2. Narrow the Scope Further

Determine whether the problem occurs for:

  • All users
  • Only one geographic location
  • Only for users with certain Browser/type of PC
  • Only during peak periods

3. Qualify the Nature of the Problem

  • Is the problem intermittent or reproducible?
  • Is the problem due to slow performance or a process hanging or spinning?
  • When was the last time the process was completed without experiencing poor performance?  Document any changes since then.
  • Is there a workaround available?   For example, restarting Browser or restarting Apache.
  • What is the frequency of occurrence?

4.  Capture Additional Clues

  • Document and differentiate factual information from user perceptions. Example - users may say "it's always slow" but reality may be that it takes 10 seconds most of the time, but 40 seconds between 10am and 11am (peak load).
  • How long does it take for the process to complete when running slowly?  How long did it previously take (before having the performance issue)?  What is the expected performance for this process?
  • What is load on server(s) when issue occurs (CPU/Memory/Network)?  Is there any unusual activity when problems occur?  For example, memory used suddenly growing or CPU at 100%.
  • Is the problem reproducible with other browsers, such as Firefox, as well as with Internet Explorer?

5.  Identify likely causes and eliminate other possible causes

  • Are there any customizations?  If so, can they be eliminated for testing purposes?
  • If the problem only occured since patches have been applied or any configuration changes have been made, can these be reverted?
  • Do you have any Resource Limits enabled on the database that may be effecting the Applications users runtime or Concurrent Manager?
  • Does the number or frequency of certain Concurrent Requests correlate to performance issues occurring?
  • Have the database parameters been configured as per the current recommendations in Database Initialization Parameters and Configuration for Oracle Applications 11i (Metalink Note 216205.1)?
  • Does AWR or Statspack show anything unusual for Top waits, Top buffer gets, etc?
  • Have you searched Metalink generally, but specifically reviewed Recommended Performance Patches for Oracle E-Business Suite (Metalink Note 244040.1) for any known performance issues, tuning guidelines and/or patches?

Getting Help from Oracle Support

Once you understand the issue and reach the point where you need Oracle Support involved, it may be useful for you to review the "Performance Tuning" section of:

The first key decision you need to make is selecting a product code for your Service Request (SR): 

  1. If the issue is with an individual Form/Report/Page or only in one product area then log the SR for that particular product support team
  2. If issue seems to be with the Technology Area or System wide then log the SR with the AOL team (Oracle Application Object Library)

It is very important to provide a good problem definition (nature, scope, extent) so be as verbose as needed to give a good description of the issue.

It is also very important to reproduce the issue on a non-Production instance. If you are able to reproduce the problem outside of your Production instance, then any recomended patches or changes can be quickly assessed for their impact and any detailed debug or tracing required to identify the issue can be easily implemented.

A Quick Aside:  One issue per Service Request

You may need several issues investigated simultaneously and may be reluctant to raise different SRs or have different people investigating. Unfortunately, even similar-looking issues can have different root causes, which means they will need to go to different support teams or have different SR statuses. This is why it is important to ensure each issue is raised as a separate SR.

Performance Issues with Specific Forms, Reports, or Pages

For these "simpler" types of issues, we can generally track down the issue with the following information:

  • Form, Report or Page name, version and navigation path
  • Full versions of Forms, Reports or Framework as well as the iAS and Database versions, including rollup or interop patches applied
  • Relevant Family Packs or Maintenance Packs applied
  • Description of symptoms, what have you tried, your investigations, conclusions and/or thoughts/ideas
  • Does it reproduce constantly, in other environments as well?
  • When did you last run "Gather Schema Statistics" and at what percentage?
  • When did you last run any relevant purging processes?
  • SQL trace with binds and waits (raw file and TKPROF output)
  • Wall clock time elapsed from user perspective
  • What are the target and acceptable times for this process?

Performance Issues for Other Areas

These are the more complex issues and will normally require additional information (and patience) to resolve.  We will need the same information as listed above, but also:

  • Description of your System Architecture and network diagram
  • Technology Stack configuration files (Apache, Forms, Reports and Database, as relevant to the problem)
  • List of any Metalink notes or product documentation you have reviewed already, and what steps you implemented from this documentation
  • List of any patches applied specifically to try to address this issue
  • Details of profile options or configuration settings you have tried changing, explaining why you tried these settings and what effect they had (if any)
  • AWR or Statspack output for "good" and "bad" performing time periods
  • Details about your pinning, purging and gather schema statistics strategy
  • Relevant Log files
  • Debug and trace files

Good starting points for collating all this information are:

Additional Tips for Troubleshooting for Apache / JVM problems

Additional Tips for OA Framework-related issues

  • It is often useful to enable "STATEMENT" level logging (for ONE user only!) using the FND:Debug% profile options if you have a reproducible test case


Performance issues can sometimes be tricky to isolate, particularly those having more than one root cause.  This article has presented some ideas about the approach to take, in addition to the sort of information Oracle Support would likely be asking for if you need to log a Service Request.

If there is sufficient demand, I can write further articles in future, expanding on the topics introduced here.


Thursday May 17, 2007

Performance Tuning the Apps Database Layer

Performance tuning is an art that should be executed systematically.  I assume almost everyone has heard this from others talking about tuning, but it's often repeated because of its fundamental truth.

Whenever there's a response time issue for the E-Business Suite, it is initially treated as a performance problem.  I would go a step further and say that poor performance is not a problem by itself.  It is a result of a root cause which lies somewhere else.

Let's dig into each of the possible places for this root cause in the E-Business Suite's technology stack.  Oracle Applications has the following layers:

  • Operating system
  • Database
  • Techstack components (Concurrent tier, Forms tier and iAS techstack)
  • Application code (Forms, Jsp etc)

This article will touch on performance-related issues for the database layer. 

Possible Causes for Database Layer Performance Issues

Possible issues include:

  • Core optimizer issues
  • Known performance-related database bugs
  • Incorrect statistics to the cost-based optimizer (CBO), which is responsible for SQL optimization
  • Incorrect System Global Area or Program Global Area sizing

Other areas like locking and latching affect performance, too.  For the limited purposes of this article, we'll focus on these four setup-related areas.

Core Optimizer Issues

Most of the E-Business Suite database's recommended parameters are listed in:

Setting the values of memory-related parameters as is not a one-time job.  We recommend that you periodically review your Statspack and AWR reports to find if the your current settings meet your current load requirements.

For recommended database init.ora parameters, refer to Note 216205.1.  Pay attention to mandatory parameters.  Based on our benchmarks, we recommend that your Apps setup adhere to the init.ora parameters listed in the note.

Known Performance-Related Database Bugs

For issues with the Oracle optimizer and database, most of the known performance-related bugs and their recommended patches for E-Business Suite environments are listed in:

Incorrect Statistics to the Cost-Based Optimizer (CBO)

Now comes the most complex -- and paradoxically, the easiest to solve -- area:  ensuring that the cost-based optimizer works with accurate metadata.

Before going into the details, I'll remind you that the rule-based optimizer (RBO) is no longer supported.  Don't use it in your E-Business Suite environments.  If you have an optimizer-related issue that appears in the cost-based optimizer but not the rule-based optimizer, log a Service Request with Oracle Support.

Maximizing the Benefits of Cost-Based Optimization

The cost-based optimizer is not infallible, but it's a DBA's responsibility to ensure that correct metadata is available to the optimizer.  There are two things that system administrators should do:

    1. Set all of the optimizer-related parameters, following Note 216205.1
    2. Gather statistics for all database objects

Gathering Statistics for Database Objects

There are many different ways of gathering statistics:

  • Use the Analyze command
  • Use Dbms_stats
  • Use Fnd_stats

For Oracle E-Business Suite environments, we recommend using fnd_stats.

A Short Digression:  What is fnd_stats?

Fnd_stats is a wrapper around dbms_stats that suits most of the E-Business Suite's requirements.

We recommend using fnd_stats over dbms_stats for Apps environments because of the former's support for restarts.  Starting with the 10g version of the database, dbms_stats has also this feature.

If the Gather Schema Statistics concurrent program is used, fnd_stats does the bookkeeping for the run.  Should the run fail for any reason, the next run of the program starts from where the previous run was stopped. This saves lot of time.

We also recommend  fnd_stats because of its support for histograms.  Histograms are useful when:

  1. A table's column is used in an equality or equi-join predicate AND there are  skews in the column.
  2. A table's column is used in a range or like predicate AND there are  skews in the column.

Oracle Applications' data distribution is dependent on the functionality of the specific product modules.  Based on our benchmarks, a number of columns are useful for histograms.  These columns are listed in the FND_HISTOGRAM_COLS table.

When Gather Schema Statistics is executed, it reads FND_HISTOGRAM_COLS and builds the histograms.  

gather schema/table statistics:

Back to Gathering Database Statistics

When gathering the statistics for the entire applications database, we must use the Gather Schema Statistics concurrent program.  Only the Gather Schema Statistics and Gather Table Statistics should be used. Do not use the Gather Column Statistics program.

When Should You Gather Statistics?

There is again no hard and fast rule for the interval between gathering statistics.  A general rule-of-thumb is is to run the statistics collection after a 10% increase in the database size. 

Having said that, other factors may come into play.  From 11.5.10 onwards, fnd_stats was enhanced to gather only the statistics for those objects, which have undergone a predefined percentage of data increase, or for objects that have no statistics.  The latest versions of fnd_stats identifies STALE and EMPTY statistics and gather the statistics for those objects only.  This saves lot of time and allows sysadmins to set flexible statistics gathering intervals. 

gather auto:

Although there are a number of options that you can set, we recommend selecting  the GATHER AUTO option.

Incorrect System Global Area and Program Global Area Sizing

SGA_TARGET sets the maximum size of the System Global Area (SGA) for your E-Business Suite instance.  Parameters like db_cache_size and shared_pool_size affect the database buffer cache and shared pool for that instance.   PGA_AGGREGATE_TARGET determines the maximum value for the Program Global Area (PGA) for the instance.

Note 216205.1 lists the minimum values for these parameters. These values might need to be varied depending on the actual load on your environment.  In the same way.  If these are values are:

  • Set too high:   memory on the server is not utilised to the maximum
  • Set too low:  there will be performance issues

To find out whether these parameters are set properly on your system, you can run a Statspack or AWR report.

Interpreting Statspack or AWR Reports

Covering this area comprehensively is beyond the scope of this article, but here's some things to remember about interpreting your Statspack report:

Total Response time = Service time + Wait time

Service Time is generally the amount of "CPU used by this session".  From the 9.2 database version and upwards, this is reported in the Statspack.  You can use this to derive the Wait Time.

For example:

Using a Statspack report with a 9.2 or higher database, examine the figures reported for CPU time.  If it's 70% of the total time, then the wait time is 100-70=30%.

Total Response time = 70% CPU time + 30% wait time

Once we have identified the time taking component, look into the specific component.  For memory-related areas, look at the Advisory Statistics section.  This shows details about the sizing of SGA (cache and shared pool) and PGA (M-pass executes should be as small as possible for a optimal PGA). 

Based on this type of analysis, you can vary your SGA and PGA settings accordingly.

Getting Support for Performance Issues

As you can see, investigating performance issues can be tricky, especially if you find the topics briefly described here to be daunting.  Take heart:  you're not on your own.  We have teams that specialize in these areas, so if you run into any performance-related issues, log a Service Request via Metalink and we'll jump in to help.

In Summary

A quick recap of database setup-related issues and recommendations for tracking down performance-related issues:

  • Database Parameters:  Refer to Note 216205.1 for mandatory settings, and then check your Statspack/AWR reports periodically to see whether your memory-related parameters are set correctly.

  • Apply the Recommend Performance-Related Database Patches:    Refer to note 244040.1

  • Keep Your Statistics Fresh: Use the Gather Schema/Table Statistics concurrent program regularly.  The GATHER AUTO option can be of great help here.

It's possible to dive much deeper into database or instance tuning topics.  If you'd like me to go into more details, feel free to post a comment here.




« April 2014