Tuesday Apr 07, 2009

Power Tools: Configuring EBS R12 Database Tiers with AutoConfig

[Editor's Note: This is the fourth and last of a series of four articles on new AutoConfig features. These articles are written by members of our AutoConfig Development team. This is your opportunity to interact directly with that team with your feedback on this tool.]

The AutoConfig Build Context utility (adbldxml.pl) has traditionally been used by administrators to rebuild the context file in Oracle E-Business Suite 11i. In Apps 11i, the AutoConfig Build Context utility was introduced to allow customers to migrate to AutoConfig easily. Later on, due to numerous changes in the 11i technology stack, it became hard to keep the utility in sync for the Applications tier. So it was not recommended to customers for discretionary use, but instead only when absolutely needed like in case of cross-platform migrations and database upgrades.

Diagram showing the process by which the AutoConfig Build Context Utility consolidates data from the EBS database environment variables and context template to generate a context file

Revived for Apps 12

The AutoConfig Build Context utility has been revived and re-introduced in Oracle E-Business Suite R12 for the database tier. It is essential for enabling AutoConfig on the database tier of an R12 E-Business Suite instance in the following scenarios:

How Does it Work?

The AutoConfig Build Context utility reads information from EBS database and environment variables and builds the context file out of the database context template ($ORACLE_HOME/appsutil/template/adxdbctx.tmp). This is illustrated in the figure above.

Running the AutoConfig Build Context Utility

The utility can be run by issuing the following command on the database tier:

perl $ORACLE_HOME/appsutil/bin/adbldxml.pl

Note: In R12, this utility is supported only on the database tier.

Downloading the Latest AutoConfig Engine

Customers on Oracle E-Business Suite Release 12 can obtain this new feature by installing:


Related Articles

Power Tools: Previewing AutoConfig Changes on All Tiers

[Editor's Note:  This is the third of a series of four articles on new AutoConfig features.  These articles are written by members of our AutoConfig Development team.  This is your opportunity to interact directly with that team with your feedback on this tool.]

When we first launched AutoConfig for E-Business Suite environments, it quickly became clear that your confidence in the tool would depend upon your ability to review its actions before committing to its changes to your environments. The AutoConfig Check Config tool (adchkcfg) is used to identify the potential changes that would take effect on an E-Business Suite instance during the next AutoConfig run.

Until now, that tool has only been reporting expected changes to the file system and the database profile values. The adchkcfg tool has now been enhanced to report information about important non-profile database updates also. The enhanced report will help customers understand potential system configuration changes, thus minimizing custom configuration errors.

What Does the Check Config Report Show?

The Check Config tool generates a report in both HTML and text format. The text report for the database changes can be used for quick reference from the command line.

Here is a screenshot of the new Check Config HTML report (click to enlarge):

Thumbnail of Check Config HTML Report

For complete samples of Check Config reports, download this file:

Generating a Check Config Report

The Check Config tool can be run by executing the following command on both the Application tier and the Database tier:

  • On UNIX:

    sh adchkcfg.sh contextfile=<context_file>
  • On Windows:

    adchkcfg.cmd contextfile=<context_file>

Downloading the Latest AutoConfig Engine

Customers on Oracle E-Business Suite Release 12 can obtain this new feature by installing:

Customers on 11i can get this new AutoConfig feature by installing:

Your Thoughts?

We're still working on improving this tool.  For example, we're working on ways to make it easier to ignore false positives by flagging diffs which are neither real or interesting.  Please share with us your experience on using this feature. Are there other improvements to this report that you would find useful? 


Related Articles

Tuesday Mar 31, 2009

Power Tools: Running AutoConfig in Parallel in EBS 12

[Editor's Note:  This is the second of a series of four articles on new AutoConfig features.  These articles are written by members of our AutoConfig Development team.  This is your opportunity to interact directly with that team with your feedback on this tool.]

Our last article discussed ways of tuning your AutoConfig runs via profiling reports that identify bottlenecks during template instantiation.  This article discusses another method of speeding up your AutoConfig runs.  In an R12 E-Business Suite instance, AutoConfig can now be run simultaneously across multiple nodes. This new feature significantly lowers maintenance downtime for multi-node installations. One beta customer of this feature improved the time it takes them to run AutoConfig across their dozen mid tiers by 45%.

How Does AutoConfig's Parallel Mode Work?

Executing AutoConfig in 'parallel mode' engages a locking mechanism so that processes running on individual nodes are synchronized. This mechanism prevents any conflicting updates to the database or the file system. The following figure illustrates AutoConfig running in parallel across multiple nodes:

Diagram showing how AutoConfig runs in parallel on EBS 12

Executing AutoConfig in Parallel Mode

The following command can be used to run AutoConfig in 'parallel mode'

Application Tier

perl $AD_TOP/bin/adconfig.pl contextfile=<CtxFile> [product=<product_top>] –parallel

Database Tier

perl $ORACLE_HOME/appssutil/bin/adconfig.pl contextfile=<CtxFile> –parallel


<CtxFile> is the absolute path to the context file
<product_top> is the Product short name

Note that while running AutoConfig simultaneously on multiple nodes, it is very important to ensure that the '-parallel' option is specified while starting AutoConfig on each node to prevent unstable and/or inconsistent filesystem and database states.

Downloading the Latest AutoConfig Engine

Customers on Oracle E-Business Suite Release 12 can obtain this new feature by installing:

This feature is currently not available for Oracle E-Business Suite 11i.  We currently do not have precise plans for delivery of this feature in 11i.

Your Thoughts?

Please let us know about your experience on using this new feature. We'd be very interested in hearing about how long it takes to run AutoConfig on your multi-node installations with and without the 'parallel' option. The new AutoConfig Performance Profiler feature can be used to collect this information. Also let us know of any issues you encounter while using this feature.


Related Articles

Power Tools: Optimizing AutoConfig Performance Through Profiling

[Editor's Note:  This is the first of a series of four articles on new AutoConfig features.  These articles are written by members of our AutoConfig Development team.  This is your chance to get the inside track on these advanced features and provide your feedback directly to our developers.]

Ever wonder what's taking up the time during a given AutoConfig run in your E-Business Suite environment?  Want to optimize the performance of your techstack configuration customizations?  The AutoConfig Performance Profiler gathers data about an AutoConfig run and generates a consolidated AutoConfig profile report in HTML format. The report lists all product tops processed by AutoConfig along with the total instantiation and execution time of the templates within them. A beta customer of this feature helped us fix an indexing issue to allow AutoConfig to run in one third of the time.

The generated performance report allows you to drill down on each product top and view the following:

  • Source and target location of individual templates
  • Time consumed to instantiate or execute each template
  • Execution report for each template

Here's a screenshot of the first few lines of the report:

Mini screenshot of AutoConfig Performane Profiler report

A complete sample performance profiler report can be found here.

Identitying AutoConfig Performance Bottlenecks

This report is useful in analyzing the source of AutoConfig performance bottlenecks. It also helps administrators optimize template customizations (if any). For example, if you have performed customizations to the context variable values or to any of the product templates and you find that AutoConfig is taking more time after the customization, you can use this feature and generate the profiler report to see where exactly the delay is occurring. The profiler report allows you to determine which phase or product took more time to execute.

Then by further clicking on the link for that phase or product, you can see more details at the template level. Going through these details, you can determine which templates are taking an unreasonable amount of time for instantiation or execution. You can use this to verify and optimize your customizations to the templates.

Generating AutoConfig Performance Profiler Reports

To generate the AutoConfig Performance Profiler report, you can run AutoConfig in 'profile mode' by issuing the following command:

Application Tier
perl $AD_TOP/bin/adconfig.pl contextfile=<CtxFile> [product=<product_top>] –profile

Database Tier
perl $ORACLE_HOME/appsutil/bin/adconfig.pl contextfile=<CtxFile> –profile


<CtxFile> is the absolute path to the context file
<product_top> is the Product short name

Note that the -profile option can be used alongside other AutoConfig command line parameters.

Downloading the Latest AutoConfig Engine

Customers on Oracle E-Business Suite Release 12 can obtain this new feature by installing:

Customers on 11i can get this new AutoConfig feature by installing:

Your Thoughts?

We would appreciate if you could share with us your experience on using this new feature. Please post your comments here or email your profiling results to Ivo Dumovic at:


We're eager to hear about your thoughts about how we can improve this feature.


Related Articles

Monday Mar 02, 2009

Certified Oracle RAC Scenarios for Oracle E-Business Suite Cloning

[Editor's note: EBS cloning can be pretty involved. There are many different cloning scenarios, including cloning a RAC-based environment to an identical RAC environment, cloning from a RAC-based environment to a non-RAC environment, and adding or subtracting RAC nodes while cloning. Our Applications Technology Group has had to make some decisions about the relative priority of these scenarios. Filesystems like ASM, OMF, OCFS2, and NFS add further complexity to the certification matrix. If you would like to share your opinions about these decisions, post a comment below. Your feedback will be sent directly to our ATG Product Management team.]

Cloning an Oracle E-Business Suite system that uses an Oracle Real Application Clusters (Oracle RAC) enabled database involves numerous different technology components and steps, and would therefore be a complex, error-prone process if carried out manually. To speed up, simplify, and enhance the reliability of the process, Oracle offers the following tools for use in cloning RAC-enabled Oracle E-Business Suite systems:


[Read More]

Thursday Dec 04, 2008

New Whitepaper: Options for Reducing E-Business Suite Database Sizes

I have yet to encounter a database that ever got smaller.  Like waistlines and the US national debt, all databases seem destined to increase in size.  The E-Business Suite is no exception:  as we add more product capabilities and your business grows, so do your Apps databases.

Oracle-supplied solutions to managing Applications database size fall into two categories:  data growth control methods and data management methods.

Growth Control Methods

  • Archiving and purging
  • Database compression

Data Management Methods

A new Oracle whitepaper discussing these topics is now available:

Screenshot of Oracle Information Lifecycle Management Assistant used with E-Business Suite database tables

[Read More]

Wednesday Aug 27, 2008

Reducing Patching Downtimes with Staged Applications Systems


It's axiomatic that everyone wants to minimize maintenance downtimes for their E-Business Suite environments. This is particularly crucial for environments with users in multiple timezones. I've previously summarized some of the most-effective ways of reducing patching downtimes for E-Business Suite environments. As noted in that article, one of the best ways of minimizing your maintenance downtimes is to use a staged Applications system.

The staged approach allows you to perform as many changes as possible in an offline Apps environment, and defers taking down your production environment only for the final database patches tasks. Using this approach, you apply your new patches to an exact clone of your production E-Business Suite environment. This can be done while your production system is still running. The staged Applications environment is then used to run database updates and APPL_TOP changes into your production environment.

[Read More]

Thursday Jun 12, 2008

New Whitepaper: Best Practices for Adopting E-Business Suite Release 12 (First Edition)

A colleague has just pulled off an impressive feat that I wouldn't have attempted myself:  she's collected practical tips and advice on how to do Oracle E-Business Suite Release 12 implementations and upgrades.  She's consolidated input from Oracle's Support, Consulting, IT, and Development groups into a new whitepaper:

This whitepaper is mandatory reading if you're planning -- or in the middle of -- an Oracle E-Business Suite Release 12 deployment.  The whitepaper has a mix of concrete and strategic advice that covers topics such as:

[Read More]

Tuesday May 27, 2008

New Whitepaper: Database Partitioning for the E-Business Suite

Some readers complain that we don't have sufficient documentation to cover all possible scenarios and topics of interest.  This is a valid observation.  As your E-Business Suite deployments grow in complexity and scope, keeping ahead of your questions and new requirements is a constant challenge.

In my position as the editor of this blog, something I do only in my so-called free time, my situation is the odd inverse of yours, namely:  the rate at which we release new Metalink Notes far-outstrips my capacity to read and announce them to the world.  Here's a much-belated announcement about a database partitioning whitepaper produced by our Applications Performance Group.

What Is Database Partitioning?

Partitioning allows a single database table and its associated indexes to be broken into smaller components depending on the table and choice of index partitioning methods.  Several E-Business Suite modules take advantage of database partitioning right out of the box, and custom partitioning is also fully supported.  I've covered database partitioning concepts for Apps environments in more depth in this older article.

Database Partitioning Methods:

Best Practices for Partitioning Apps Databases

As I've noted before, we have a group in our Applications Development division that's dedicated to optimizing the E-Business Suite's performance.  As a member of this Applications Performance group, Mohsin Sameen has worked extensively with some of our enterprise-class customers -- including many of the largest companies in the world -- on fine-tuning the performance of of their high-volume Apps environments.

Mohsin has distilled these experiences into an extensive and in-depth paper on database partitioning:
Mohsin's excellent whitepaper covers topics such as:
  • Overview of database partitioning concepts
  • Table partitioning strategies involving range, list, hash, composite, and multi-column partitions
  • Index partitioning methods, including global and local partitioned indexes
  • Step-by-step decision framework for using partitions
  • Partition maintenance operations
  • Partitioning case study
The knee-jerk reaction answer to a performance problem is often to throw more hardware at it.  If you have a large E-Business Suite environment where the growth rate of your historical transactional data is starting to affect performance, I'd strongly recommend reading this whitepaper.  It's entirely possible that you could use it to squeeze some additional performance out of your existing environment without the added expense of new hardware.

Related Articles

Tuesday May 06, 2008

E-Business Suite + Fusion Middleware Best Practices Center Launched

If you've been watching Oracle's ERP strategy, you'll notice that there's been a profound shift in emphasis over the last few years.  The E-Business Suite is now acknowledged to be only one part of your organization's overall software environment, and we're investing heavily in integration technologies such as Service-Oriented Architecture (SOA).

FMW Best Practices Screenshot: Screenshot of Oracle E-Business Suite + Fusion Middleware Best Practices Center

My colleagues in the Fusion Middleware group have just launched a new Oracle E-Business Suite & Fusion Middleware Best Practice Center.  This site has step-by-step tutorials covering topics that include:
They've also started a blog that already has a rich set of deep technical articles covering topics such as:
If you're already developing SOA-based applications involving the E-Business Suite, or just curious about what's now possible with our latest tools, this site is worth a look.

Related Articles

Wednesday Apr 23, 2008

Preparing for Apps 11i AutoConfig Updates

E-Business Suite Release 11i uses the Oracle9i Application Server components in its Applications tier. This includes an 8.0.6 ORACLE_HOME for the Developer 6i Forms and Reports run-time and an 8.1.7 ORACLE_HOME for the Apache and JServ-based run-time components. AutoConfig and its associated templates provide the configuration data for these ORACLE_HOMEs.

The latest version of AutoConfig is always included with the latest ATG Technology Stack rollup patches.  It's also possible to update the AutoConfig Engine and its associated templates separately if certain prerequisites are met.  You can download the latest version of AutoConfig via the TXK Rollup for Oracle E-Business Suite Release 11i:

In preparation for your AutoConfig upgrade, you can run the Technology Stack Rollup Validation Utility to get a report of the current component versions on your Release 11i Applications tier. The report also advises on the components on each tier that need to be upgraded before upgrading AutoConfig itself.


Running the Technology Stack Rollup Validation Utility

The Technology Stack Rollup Validation Utility is a perl script that is run using the perl infrastructure delivered in the APPL_TOP by the TXK team.

Where to run

  • In a single-node install, run the utility on the Applications Tier Node.
  • In a multi-node install, run the utility on every Web/Forms, Admin/CP Node
  • The latest version of the Validation Utility is shipped with the TXK Rollup. Hence the script needs to be run from the location where the patch is unzipped.

Steps to run

  1. Source the Applications environment file as the owner of the application tier file system
  2. From the location where TXK Rollup patch was unzipped, change directory to fnd/patch/115/bin
  3. Run the Validation script as described in the table below
./txkprepatchcheck.pl -script=ValidateRollup -outfile=$APPLTMP/txkValidateRollup.html 
-appspass=<apps database password>
%ADPERLPRG% txkprepatchcheck.pl -script=ValidateRollup


-appspass=<apps database password>

Note: The arguments for the perl script txkprepatchcheck.pl can be entered manually at the prompt also.

Interpreting the Report

The report generated in the location specified by the outfile argument contains:
  1. Overall PASS/FAIL status of the Utility.
  2. Report header table with information about the host, node type and configuration file, versions.
  3. Validation table for each of the HTTP Server, Forms Server, Concurrent Processing Server, and Administration Server Node types.
  4. For each of the node types, the dependent and relevant Technology Component is displayed with its version number and PASS/FAIL status.
Sample Reports

Report Header



Validation for HTTP Server Node


Validation for Other Nodes


Before Applying the TXP RUP

The AutoConfig upgrade, as delivered by TXK Rollup patches, can be be applied only when the report displays PASS for all entries in each table and an ALLPASS at the top. All WARNING/FAIL rows should be corrected with respective actions recommended in the report row before applying the patch.

  1. Using AutoConfig to Manage System Configurations withOracle Applications 11i (OracleMetalink Note 165195.1).
  2. Frequently Asked Questions About Using AutoConfig With Oracle Applications Release 11i (OracleMetalink Note 218089.1)
Related Articles

Tuesday Apr 22, 2008

New Whitepaper: E-Business Suite Development Using OAF & ADF

Customers, partners and system integrators often develop extensions to Oracle E-Business Suite (EBS) applications. Such extensions have traditionally used the same technology stack (Forms or OA Framework) that the original EBS application was built with.  This ensured that the extensions are fully compatible with the rest of the installed EBS applications.

R12 OAF Techstack: Block diagram showing the OA Framework Technology Stack for E-Business Suite Release 12

With the emergence of the next generation Fusion Middleware technology stack, especially the Application Develoment Framework (ADF), an increasingly-common question is whether to use OAF or ADF to develop E-Business Suite Release 12 extensions.

Our E-Business Suite Applications Technology group has released a long-awaited whitepaper addressing this question. This whitepaper discusses the similarities, differences and overlaps between the OAF and ADF stacks. 

If you're considering these technology stacks for your Apps R12 extensions, I'd strongly recommend reviewing this whitepaper:
Related Articles

Friday Mar 28, 2008

Downtime and Apache Restricted Mode in E-Business Suite Release 11i

As I started writing down the steps for my recent post Downtime and Apache Restricted Mode in Release 12,  Steven and I exchanged some conversations which made me realize that there would be questions about the availability of the same feature Release 11i. And, it turns out to be fairly accurate realization. So, this one is for those enthusiastic pals of mine on this community. I will try to do this differently, hence making it relatively shorter in content.

Is this feature available in 11i?

Yes. Check out the documentation in the References section.

Is the procedure to configure the same?

Yes. As you might have understood already, there are two pieces in this puzzle.
  1. Creation of downtime
  2. Managing Apache in Restricted Mode
Answering the question based on these two tasks,
  • Step '1' is pretty much the same in 11i and Release 12. Hence, I am not recreating the screen shots for you here. Check out the steps in the table below.
  • Step '2' is automated in Release using the perl script where as in Release 11i, the administrator has to go through some manual steps and run Autoconfig.

What are the exact steps in Release 11i?.

Thanks to our OAM team, these are well documented. And, they work like charm. Below are the sequence of steps for my DBA friends.

Note: Navigation in 11i for the downtime page creation is slightly different than Release 12.

How to?
Schedule System Downtime and warn your end users of an impending downtime Use OAM to schedule downtime:
  • ( Navigation: Sitemap=>Maintenance=>Patching and utilities=>Schedule Downtime)
Complete the required one-time setups to monitor patch in progress
  • Use OAM Autoconfig editor to edit the variable < s_trusted_admin_client_nodes > to include the list of hosts that can access OAM in restricted mode. Run autoconfig to ensure that the new settings take effect.
  • Ensure that you have enabled the the monitoring user account by unlocking the ad_monitor user account and setting the password by using the following commands:
    • alter user ad_monitor account unlock;
    • Login to SQL*Plus as user ad_monitor. Default password is lizard. Reset the password.
Shutdown all Applications services
Use the standard ad script from $COMMON_TOP/admin/scripts/<context_name> directory:
  • adstpall.sh <apps user>/<password>
Enable maintenance mode for your Applications system
Run adadmin, and:

  1. Select option 5 'Change Maintenance Mode'
  2. Select option 1 'Enable Maintenance Mode'
Start OAM in restricted mode to monitor patching in progress
From the $COMMON_TOP/admin/scripts/<context_name> directory, run:

  • adaprstctl.sh start
Begin applying patch
Run adpatch (hotpatch=n)
Monitor patching in progress
  1. Access OAM in restricted mode from the following URL : http://host:port/servlets/weboamLocal/oam/oamLogin
  2. Login into OAM using the ad_monitor user
  3. Navigation: Sitemap=>Maintenance=>Patching and utilities=>Timing Reports
Confirm the end of scheduled downtime upon patch completion
From within OAM in restricted mode:
  • Navigation: Sitemap=>Maintenance=>Patching and utilities=>Manage Downtime Schedules=>Select "Complete" button.
Bring your Applications System to normal mode.
Run adadmin, and:
  1. Select option 5 'Change Maintenance Mode'
  2. Select option 2 'Disable Maintenance Mode'
Shutdown Apache in restricted mode and Restart all services
From the $COMMON_TOP/admin/scripts/<context_name> directory, run:
  • adaprstctl.sh stop
  • adstrtall.sh <apps user>/<password>


  1. About Oracle Applications Manager Mini-pack 11i.OAM.H
  2. Chapter 6, Section 'Managing Downtime in Restricted Mode' of Oracle Applications System Administrator's Guide - Maintenance - Release 11i (Part No B13924-04)

Tuesday Jan 29, 2008

Sharing Apps R12 File Systems Across Multiple Databases

One of the challenges an IT manager faces today is managing complex application configurations from development to production. This becomes a monster challenge with the proliferation of environments with multiple application and database tier nodes.

Oracle E-Business Suite Release 11i and 12 provide flexibility in laying out file systems via a Shared File System option. A Shared File System stores application tier files on a shared disk storage that can be accessed by multiple application tier nodes. Shared File Systems have the potential to lower your Total Cost of Ownership (TCO) by reducing your maintenance and hardware requirements.

Note: Shared File System is  not supported on MS Windows platform

New Features in E-Business Suite R12

Release 12 Shared File System configurations allow several deployment options. Some of the key features include:
  • The ability to have multiple nodes running each of the following service types:
    • Forms
    • Web - in a load balancing configured option
    • Concurrent Processing (Batch) - in a Parallel Concurrent Processing configured option
  • Read-only node implementation for Forms, Web, Concurrent processing
In addition, a new configuration option introduced in Release 12 has the potential to save significant time and resources for an Applications DBA:  the ability to share applications tier file systems across multiple database instances.

Sharing Applications Tier File Systems Across Multiple Database Instances

Release 12 customers can now install and configure the application tier file system on a central machine that is used by two or more database instances. This configuration option is useful in cases where you would like your test and development environments to share the same applications tier file system. 

Shared File System across DB:

Minimum Requirements
  1. This option is only valid for the Applications tier file system and not for the database tier file system

  2. All the database instances are patched up to the same level

For a walkthrough and other details required for this configuration and deployment option,  see: [Editor: Sharing application tier file systems between multiple environments has the potential to reduce your overhead but is the IT equivalent of placing all of your eggs in a single basket.  If the shared file system goes down, multiple Apps environments are impacted.  Accordingly, prudent system administrators should put backup and recovery procedures in place for shared servers.]

Related Articles

Tuesday Dec 11, 2007

Performance Tuning for the E-Business Suite

It's said that death and taxes are the only certainties.  I beg to differ.  There's at least one other for IT professionals:  regardless of the amount of network bandwidth you've dedicated to E-Business Suite traffic, no matter how many application servers you have in your load-balancing pool, no matter the turbocharged fire-breathing database server you've got in your data center, it's inevitable that your users will complain about performance. 

Apps DB Console Screenshot: Screenshot of E-Business Suite Database Console showing Host queues,Waiting and Working Sessions, Instance Throughput, and other performance metrics

In my promised but much-belated recap of OpenWorld 2007 highlights for Apps DBAs, here's a presentation from Isam Alyousfi and Lester Gutierrez that you much check out:
Isam and Lester are key members of our Oracle Applications Performance Group -- the same group that publishes all of the official Oracle Apps benchmarks, white papers, and performance metrics.

In this sprawling presentation, they cover the following topics:
  •  E-Business Suite Architecture and techstack from a performance perspective
  • Tuning the Applications Tier, including:
    • Forms & Reports tracing and tuning
    • Concurrent Manager tracing and tuning
    • PL/SQL Profiler usage and tips
    • Apache JServ/OC4J optimization and configuration tips
    • How to debug Java memory and connection leaks
    • OA Framework tracing
    • AOL/J Connection Pool monitoring
    • Discoverer tuning
  • Tuning the Database Tier, including:
    • I/O considerations
    • Automatic Workload Repository (AWR) data interpretation
    • Use of the DB Console
    • How to gather and interpret statistics
    • Use of the Applications Tablespace Migration Utility
    • 10g DB upgrades from a performance perspective
  • Client and Network Tuning, including:
    • Memory usage and profiles for Apps 11i and R12
    • Network performance and bandwidth considerations
  • Application Tuning, including specific tips for:
    • Workflow
    • Order Management
    • Payroll
    • Financials
    • TCA/DQM
    • Logging
    • Profile usage
    • Purging and archiving
  • And much, much more
This presentation is chock full of tips, pointers, and hard-won knowledge.  It represents the distillation of countless performance-related Service Requests and customer escalations.  If you're grappling with performance issues in your environment, or simply trying to squeeze more performance out of existing hardware, I'd strongly recommend downloading this presentation.


Tuesday Nov 27, 2007

Which Are Better: Family Packs or Consolidated Updates?

Boy, is my inbox going to get hammered because of this article.  But when have I shied from controversy?  (Always, a cynical voice mutters).  Nevertheless, into the breach I charge.

11.5.10.CU2 Download screenshot: Screenshot of Metalink download site for E-Business Suite Release 11i 11.5.10 Consolidated Update #2 download

A recurring question raised at OpenWorld came from customers debating between applying individual Family Packs or the much larger Consolidated Updates to their E-Business Suite environments.  Which is better?

If I Were An IT Manager

I understand your dilemma:  maintenance windows grow tighter by the day.  Staffing levels don't keep pace with rising workloads.  Risk-averse stakeholders pressure you to keep the system unchanged -- all the while clamoring for their own personally-reported bugs to be fixed immediately.  Given this climate, anything that looks like it makes your life more complex is going to be summarily rejected.

If I had the choice of applying a random and arbitrary combination of individual family packs (e.g. FWK.H + AD.I + XDO.H) or applying a single Consolidated Update (e.g. 11.5.10.CU2), I'd choose the latter without hesitation.  If I had had the choice, Consolidated Updates are unquestionably what I'd choose.

Pulling Back the Curtain

The recommendation to apply a much larger patchset with broader impact might seem irrational, especially given what I just acknowledged is the typical IT environment.  But bear with me:  this may make more sense if I give you a glimpse into our development and testing processes.

How Consolidated Updates are Tested

Consolidated Updates (CU) are tested by the entire E-Business Suite development division in a central testing environment.  In fact, Consolidated Updates are repeatedly tested in multiple iterations of these centralized testing environments, some built for automated regression testing, others for cross-product integration flows (e.g. "Order to Cash"), and even others built for internationalization testing with NLS character sets and localizations. 

We also test multiple migration paths to a given Consolidated Update.  For example, the 11.5.10.CU2 Consolidated Update was tested on top of the 11.5.9 Rapid Install, 11.5.9.CU2, the 11.5.10 Rapid Install, and 11.5.10.CU1.

On top of all that, Consolidated Updates are tested in a variety of so-called "advanced architectural configurations" that include load-balancers, DMZs, Single Sign-On, Discoverer, SSL, RAC, different JInitiator versions, different native Sun Java (JRE) plug-ins, different desktop platforms (e.g. Windows and Mac OS X), and so on. 

And then, just because we have too much free time on our hands, the Consolidated Updates are tested on various operating system platforms, including Solaris, HP-UX, IBM AIX, and so on.

How Individual Family Packs are Tested

Individual Family Packs are tested by their respective product teams.  They receive product team manual testing and automated regression testing.  Depending on the product, they may receive some additional central testing by multiple product teams or in with advanced configurations, but this is relatively rare.

You can see that there are fundamental and profound differences in the depth and range of Consolidated Updates vs. Family Pack testing.  A Consolidated Update receives massive, intensive, coordinated testing across all E-Business Suite products.

A reasonable analogy might be:  Family Packs are to Consolidated Updates as emergency one-time patches are to Family Packs. 

The Sum of the Parts Does Not Equal the Whole

Here's something else to consider:  let's say that FIN_PF.G is in 11.5.10.CU2 (this is just an example -- I don't know if this is actually true).  Even if you install FIN_PF.G on top of 11.5.9, you aren't officially on 11.5.10.CU2 since you haven't applied the actual 11.5.10.CU2 patchset.  If you haven't explicitly applied 11.5.10.CU2 itself, Oracle Support does not consider you to be on that code level, regardless of the sum of the individual Family Packs that you might have applied.

So, when 11.5.9 is desupported, your calls to Oracle Support will prompt the usual  discussions about being on a desupported release, even though your individual family packs may be up-to-date to 11.5.10.CU2. 

Sometimes "More" is Better

Knowing all that, I always recommend applying the latest Consolidated Update instead of individual Family Packs.  This guarantees an end-run around the hassles of possible family pack incompatibilities and desupport issues.  In my view, the initial larger overhead of applying a bigger patch is far outweighed by the benefits.

I know that this will be a controversial position for some of you, so I welcome your comments. Let the debate begin...


Tuesday Jul 24, 2007

10 Tips for Protecting your APPS Password

E-Business Suite security is a huge topic, as there are many different facets to consider. This article will consider a small but essential part of the security model: protecting your APPS user password.

The APPS user in the E-Business Suite is the master of its world. I have gathered together some thoughts on steps you can take and things to consider to help you protect your APPS password from being compromised:

  1. Stay current with our latest Security Best Practices

    Regularly review the latest version of Best Practices for Securing Oracle E-Business Suite (Note 189367.1). This note is regularly updated and will give security advice covering many different aspects of Applications 11i.    For Release 12, see Best Practices For Securing Oracle E-Business Suite Release 12  (Note 403537.1)

  2. Regularly change your APPS password

    This is an essential activity from a security perspective and needs to be part of your routine operating procedures. Same applies for other schema passwords and SYSADMIN user. As an aside, don't use predicable passwords, or have a system to create passwords, such as using "0ct0ber" for the password in October as this will make it easier to guess

  3. Always change passwords as part of a clone process from PROD

    It is recomended to change ALL schema passwords and ALL eBiz user passwords in a cloned instance.   You can use Removing Credentials from a Cloned EBS Production Database (Note 419475.1) to achieve this.    Similarly, you don't want to have any relation in the passwords used for PROD compared to any other instances. Data masking and obfuscation is a large topic in its own right, but is also something you may need to consider doing for the cloned instance to protect sensitive data generally. With Release 12, EM plugin provides some data scrambling facilities.

  4. Perform data masking on any files sent to outside parties from the PROD system

    When you need to send any log files or configuration files, ensure that you scan for any sensitive data before packing the files to be sent. In this article we are concerned about the APPS password, but this applies equally well for other data as well. For example, a crude mechanism would be to use "ed" or "sed" on all files to globally change any occurrences of the APPS password before creating a tar archive to email or upload. You may be uploading files to Oracle Support, or just emailing them within your Organization. Whenever the files are going to someone who cannot access them directly you should always check the files before sending.

  5. Create separate schemas with minimal access required for direct database access  

    If anyone requires direct access to the E-Business Suite database, ensure that you create a new unique schema with the specific permissions required for them to perform their job role. Except for a very few Apps DBAs, there should be no reason that anyone else needs the APPS user password. Sometimes pressures of work make it easier to just give someone APPS access, but this should be resisted and the time taken to provide only the minimum access absolutely required. Every person should also have their own unique login (but this is digressing into a separate area that I'll address in a later article).   When considering permissions to allocate, don't be tempted to give read only access to everything, as being able to read sensitive information may be just as damaging as being able to change it.

  6. Protect Apps 11i middle tier file system files

    These days, there is little need to give anyone UNIX-level access to the servers, but it is still important to ensure the "applmgr" operating system user password is well protected. Also consider whether any of your own startup scripts or monitoring scripts have the APPS password hard coded in them, and protect these scripts with chmod 700 permissions, or remove them if no longer needed

  7. Ensure no processes are running with APPS username/password in command line

    Generally the APPS password is not listed in "ps" output, but there may be some manual scripts or other processes intermittently running with the APPS password in clear text or trivially encoded. Ensure these scripts are changed to hide the APPS password. In addition, ensure operating system access is restricted to only those who really need it

  8. Protect OID access

    If you have integrated the E-Business Suite with Oracle Application Server 10g, Single Sign-On, and Oracle Internet Directory, then the Apps user password is stored in the OID database, as it is required for Provisioning to function. The OID administrator and anyone with ldapsearch rights in the Provisioning Profiles will be able to extract the APPS password from OID. This in turn implies the "AppsDN" OID password should be protected in the same way as the APPS password itself.    For assistance in security hardening OID, refer to the Oracle Internet Directory Administrator's Guide 10g ( - Part III Directory Security

  9. Encrypt SQLNET traffic from Middle Tier to RDBMS

    In a previous article, Steven highlighted that ANO is certified with the E-Business Suite. Use encryption to protect the APPS password from network sniffers tracing SQLNET connection packets and deciphering the APPS password on the wire.

  10. Allow only specific IP addresses to access RDBMS via SQLNET

    Slightly off topic, but if someone has acquired the APPS password they still have to be able to gain access to a tool that can use it. Restricting the IP addresses that can access your Apps database will help minimise this risk. If you are still using "fat" clients (Discoverer or ADI for example) then you will have to weigh up the risks against the administrative overhead. Oracle recommends upgrading to server-based equivalent tools or shared desktop technologies such as Citrix so desktop clients no longer need direct access.  This topic is discussed further in E-Business Suite Recommended Set Up for Client/Server Products (Note 277535.1)


Defence in depth is generally considered the best approach so hopefully these recommendations will give you some food for thought when you are reviewing how well your own system is protected.

Sound password policies are critical to enforce access policies and enforce individual accountability.  You need to jealously guard your passwords, particularly for the APPS user.


Thursday Jun 28, 2007

OA Framework or ADF?

[Editor Update 4/22/2008:  A new whitepaper from the Oracle Appliations Technology Group comparing OAF and ADF is now available.  See this article for more details.]

[July 9, 2007 - see "One more thing to consider", below - Sara]

A question I've been getting recently is "Should I use OA Framework (OAF) or the Oracle Application Development Framework (ADF) to develop extensions to E-Business Suite (EBS) products?  I know we are moving away from Oracle Forms, and I want to 'future-proof' my extensions as much as possible."

With all the buzz about Fusion (both the coming Fusion Applications and the current Fusion Middleware technology stack) and JDeveloper with ADF, it's hard not to get excited about using all the features of the latest versions of Fusion Middleware.  So would we recommend using OAF or ADF?

My general advice is to stick with OAF so long as you are working with the E-Business Suite, and wait until you move to the Fusion Applications before moving to ADF.  OAF is the development platform for the HTML EBS applications, so it makes sense to use the same development platform for custom extensions to the EBS applications.  OAF provides you with automatic, seamless integration with the EBS applications in terms of EBS menus, security, look and feel, flexfields, personalization, attachments, and so on.  These integrated EBS features aren't part of ADF.

Another part of the integrated OAF/EBS technology stack is Oracle Application Object Library (AOL), which provides many services to OAF, including security, messages, flexfields, and so on.  Developers using OAF also use the OA Extension, which is a JDeveloper add-in bundled with JDeveloper in specific EBS development environment patches.  OA Extension is an extension to JDeveloper 9.0.3 for EBS 11i, and to JDeveloper 10g (10.1.3) for EBS 12.  AOL and OA Extension are not parts of the ADF technology stack for the Fusion Applications.

A terminology note: "ADF UIX" is the same UIX that is used by OAF as the HTML UI display component.  Developers using OAF use ADF UIX indirectly by using OAF components.  ADF UIX is not part of the "ADF" Fusion Applications technology stack that I am discussing here. "ADF Faces" is the UI component of the ADF technology stack. 

Going It Alone

If you intend to build a completely standalone application that just happens to use the EBS tables, you can certainly use ADF, but you'd need to deal with things like security, look and feel, and so on yourself.  You also wouldn't have the personalization, flexfields, attachments, or other infrastructure that OAF provides.

Using ADF to build integrated extensions to the E-Biz Suite is an untested combination, so you won't find much experienced help available if you take that path.  Also, ADF isn't certified for use with E-Business Suite (EBS), and there are no plans for this certification.  ADF is a standalone development tool and is expected to be a key part of the Fusion Applications platform.  It's a very different technology stack from OAF, other than the business components.  Business Components for Java (BC4J) becomes ADF Business Components (ADF BC). 

Your Business Logic Moves Forward

Business logic that you create now with BC4J with the OAF technology stack will be transferable to the Fusion Applications technology stack later (possibly with minor changes).  If you follow our OAF coding standards, build your pages declaratively as much as possible, and avoid putting much code into controllers if you can, you'll have a much easier time moving to Fusion later.  All your business logic, validation, and so on should be in your BC4J objects.

Technology Directions

One further note: it's part of the general EBS Applications Technology and OAF direction (as part of Applications Unlimited), that we want to bring as much "Fusion technology" into the EBS applications as possible in our ongoing releases.  So over time, the technology stack gap will likely get smaller, and customers will get to use some of the newer technology within the E-Biz Suite until they are ready to move to the Fusion Applications. 

One example of this direction is that we now use JDeveloper 10g (10.1.3) for Release 12, instead of JDeveloper 9.0.3, which is used with Release 11.5.10.  So customers can now take advantage of new JDeveloper 10g features such as a much better Java coding/editing environment.  JDeveloper 10.1.3 can also dynamically recognize which files are in your path for a project.  One of my correspondents thinks that feature alone is a good reason to use 10.1.3!

Happy extending!

[July 9, 2007 - One more thing to consider: how big is the project?  If you are embarking on small extensions, a few pages here and there that you want to integrate into the existing applications, it's better to use the existing tool for that product line (EBS, Peoplesoft, etc.).  If you are building entirely new, monolithic applications (large applications with many pages),  where seamless integration with the current application pages isn't so important, you may want to use ADF (with ADF BC and ADF Faces).  Then you won't need to convert your custom applications when you move to the Fusion Applications later.]


Tuesday Jun 19, 2007

Top 7 Ways of Reducing Patching Downtimes for Apps

[June 20 Update:  Our blogging software temporarily lost its mind and somehow misnumbered the list at the heart of this article.  There are only seven tips here, not ten.]

I've previously discussed the Top 5 Myths About Patching Apps Environments.  One of the most commonly-cited reasons for not patching E-Business Suite environments is that it takes too much downtime.  If you're relatively new to Apps system administration, here's a primer on the key techniques compiled by our Applications Release Engineering group for reducing patching downtimes.  Although some of the linked documents are written specifically for Release 11i, these techniques may be used for both Apps 11i and 12.

  1. Use a staged applications system

    This major time-saver hinges on a key principle:  all of your applications filesystem patches are applied to a clone of your production Apps environment.  This can be done while your production system is still running.  Your production system is down only for the time needed to apply database patches.  For details, see:
  2. Use a shared application-tier file system

    If you have a pool of application-tier servers set up for load-balancing, make sure that all of the individual servers share a single application filesystem.  Patches applied to this central shared filesystem are instantly available to all application-tier servers.  I've previously given an overview of this technique in this article.

  3. Distribute worker processes across multiple servers

    When applying a patch that includes a large number of processes, you can reduce the downtime even further by distributing the worker processes across multiple servers on multiple nodes. Using the Distributed AD feature of AutoPatch and AD Controller, you can assign workers to run on the primary node and on other nodes that share the filesystem. See:

    Distributed AD (Metalink Note 236469.1)

  4. Merge multiple patches using AD Merge Patch

    Merging patches saves time because the AutoPatch overhead of starting a new session is eliminated for those patches that are consolidated.  Duplicate linking, generating or database actions are run once only.  If two patches update the same file, AD Merge Patch will save time by applying only the latest one.  Patches can -- and should -- be merged with their listed prerequisite patches. 

    For more details about this AD utility, see the Oracle Applications Maintenance Procedures guide for your Apps release.

  5. Run AD Patch in non-interactive mode

    Applying a set of patches using AD Patch in non-interactive mode eliminates the delay between successive tasks.

  6. Defer system-wide database tasks until the end

    Using adpatch options=nocompiledb,nomaintainmrc defers system-wide database tasks such as "Compile APPS schema" and "Maintain MRC" until after all patches have been applied.  As of AD.H, AutoPatch automatically compiles the APPS schema and maintains MRC when applying standard patches.
  7. Avoid resource-related bottlenecks

    Patching can grind to a halt if you bump into the ceiling on your system.  Before patching, make sure that you've enabled automatic tablespace management, and that you have sufficient hardware and free disk and temp space.
The cumulative downtime reductions of all of these techniques can be quite significant.  I've touched on some of the biggest timesavers, but this short article isn't comprehensive, by any means.  The linked Notes in this article and below discuss a number of other tips for shrinking your patching downtimes.  A small investment in learning these techniques can pay off in large reductions in patching times, and is well worth your time.


Thursday Jun 07, 2007

Top 5 Myths About Patching Apps Environments

When I was younger, I thought I could change this world.  Now I no longer think so but for emotional reasons I must keep on fighting a holding action.
~ Robert Anson Heinlein

A rational person might contend that actions follow attitudes which follow beliefs.  Bad news:  there's a substantial amount of psychological literature that suggests that this isn't how we really tick.

There's an equally large body of empirical data that suggest that beliefs are relatively intractable.  Once established, certain beliefs don't change, regardless of the best data, clear reasoning, and eloquence that can brought to bear on the subject.  Recent fMRI studies suggest that brain structures may develop in such a way that people with strongly-held beliefs are actually unable to process new information that contradicts those beliefs.

The broader implications of this are kind of depressing to contemplate.  But I digress.

For the narrow purposes of this discussion, I'm compelled to make an admittedly-quixotic attempt to persuade you that the benefits of keeping your E-Business Suite environment up-to-date far outweigh the costs.

"We Can't Upgrade Because..."
  1. It requires too much downtime
  2. Testing is too expensive for end-users
  3. It's too complicated
  4. We don't have enough staff
  5. It ain't broken; why fix it?
Myth #1:  It Requires Too Much Downtime

This seems rational on the surface.  After all, a downtime for patching seems worse than no downtimes.  

Remember that we issue patches for four major reasons: 
  1. To add new functionality
  2. To improve stability
  3. To improve performance
  4. To improve security
The corollary is that an unpatched system may have fewer or less-sophisticated features, and be slower, less stable, and less secure than a fully-patched one.  Questions to consider if you believe this myth:
  • How much downtime is currently caused by unplanned outages due to unpatched stability bugs?
  • How much downtime is needed to bounce your application servers due to unpatched JVM memory leaks?
  • How much downtime would be required if an attacker takes advantage of an unpatched security risk?
There are a number of key ways to reduce downtimes.  One of these is to use shared filesystems in environments with multiple application servers.  There are a number of other ways, too.  These are beyond the scope of this article, so I've covered the top seven way of reducing downtimes in this article.

Myth #2:  Testing is Too Expensive For End-Users

Many organizations recruit business users to participate in User Acceptance Tests for patches.  If you do this, it's true that this may reduce participants' productivity for the duration of your testing phase. 

If you don't keep your environment up-to-date, the key consideration are: 
  • How much productivity is lost for all end-users -- not just the testers -- due to unpatched performance, stability, or security bugs?
  • How much could productivity be improved by new features?

Myth #3:  It's Too Complicated

All DBAs know the drill:  before patching, print all patch READMEs and spread them in neat piles across a big surface.  Read them, highlight them, then reread them.  Download more patches.  Repeat. 

Every patch has some set of prerequisites.  These prerequisites lead to other prerequisites.  The key thing to remember is that it's always easier to patch an up-to-date system than something that's much older.  The bigger the gap between your current system and target patch level, the more complex the upgrade will be. 

The best way of reducing the complexity of upgrades is to keep your system up-to-date.

Myth #4:  We Don't Have Enough Staff

I need to be candid here:  there's a kernel of truth in this one.  This is a direct outcome of believing Myth #3.  If you haven't patched your E-Business Suite Release 11i environment since it was installed in 2001, then you may have a big and complex upgrade project on your hands.  This might take more staff than you have.

So, the best way of avoiding the truth of this one is to keep up-to-date with your patching.  If you're up-to-date, applying a given patch is less complex and requires fewer staff.

Myth #5:   It Ain't Broken; Why Fix It?

With apologies to English teachers everywhere, that old axiom has a seductive ring of truth to it.  After all, your system seems to be running just fine, thank you.  Why bother potentially destabilizing something that everyone's happy with?

Repeating the key points made in Myth #1:  new releases are issued to provide new functionality and improve stability, performance, and security.  If you're running an older release, it has -- by definition -- issues in these areas that you may not have noticed yet.

The Worst-Case Scenario

If I were an IT manager running an older E-Business Suite release, say 11.5.2, this would be the nightmare scenario that would keep me awake at night:

My business requirements changes due to an acquisition or a change in business practices.  These changes trigger a different load or usage profile on my E-Business Suite environment.  This triggers a Severity 1 outage.  My production system is down.

I call Oracle Support for help.  Good news: they can reproduce the problem on 11.5.10.CU2.  Bad news: due to technical dependencies (and supported by the fact that 11.5.2 is out of Premier Support status), they can only issue patches on top of 11.5.10.CU2.  There's no way of getting that patch backported to 11.5.2.

Now I'm faced with a huge upgrade from 11.5.2 to 11.5.10.CU2.  My production system is still down, and it will be down for the duration.  Users are burning me in effigy in the parking lot.  Business impact analysis, user acceptance testing, load-testing, fail-over testing -- all of those activities are jettisoned in my panicked attempts to get my environment running again.

It's Never Too Late

At this point, I suspect I've lost most of my readers.  By now, they've clicked on the latest YouTube video or something more cheerful. 

For the handful of my remaining readers:  if you're in a hole, it's never too late to stop digging.  Less metaphorically:  just because you're behind in patching doesn't mean that you should just give up entirely. 

We can help you put together a strategy for getting up-to-date.  Your first call should be to your Oracle account manager.  He or she can engage specialists in Sales Consulting or Consulting to help.  There are also excellent groups in our Advanced Customer Services organization (a.k.a. Field Support, On-Site Support, Premium Support) who offer a number of of packaged and customizable support offerings in the patching realm. Alternately, there are a number of excellent third-party consultants who can help you with this.  Or, if you're really intent on doing it yourself, there are forums on the Oracle Technology Network and elsewhere where you can brainstorm possible upgrade strategies.

Whatever path you choose:  start now.  It's never too late.


Tuesday Jun 05, 2007

Comparing Bandwidth Requirements between Release 11i and 12

[June 8, 2007 Update:  Clarified background on the second set of benchmarks]

A few questions have been raised in our new Release 12 Upgrade Forum about the differences in network bandwidth requirements between Release 11i and 12.

The usual disclaimers apply:  benchmark tests are conducted by the Applications Performance Group on reference E-Business Suite environments.  These environments and the selected transactions will not necessarily match up with your own transaction mix, so your mileage will vary.  If you're in need of precise benchmarks, it's always best if you perform your own tests with your own mix of transactional data.

Differences Between JInitiator and the native Sun JRE Plug-in

Release 11i currently requires Jinitiator to display Oracle Forms-based content.  Release 12 requires the native Sun Java Runtime Engine (JRE) to display Forms-based content. 

Our Applications Performance Group has published a full whitepaper comparing bandwidth requirements between these two configurations:
Differences in Page Sizes

As for our web-based applications, many of the products changed their pages and flows in Release 12.  There are actually two comparison points: 
  1. 11.5.10.CU2 vs. 11.5.10.CU2 with on the Release 12 ATG Rollup
  2. 11.5.10.CU2 vs. Release 12
Here's the first comparison of page loads:

Comparison of page load sizes - Part 1: Comparison of page load sizes between Release 11i 11.5.10.CU2 vs. 11.5.10.CU2 with the Release 12 ATG Rollup applied

[June 8, 2007 Update:  The section above is generating a lot of similar questions, perhaps understandably.  Here's the scoop:  the comparison above shows the additional overhead associated with the new R12 technology stack running the same Applications code.  This
would be the equivalent of running the 11i versions of these screens in Release 12.  Our Applications Performance team is very meticulous about comparing equivalent things.  No "apples to oranges" comparisons for them.

They deemed it important to distinguish between added network bandwidth overhead in Release 12 due to new code at the Apps layer vs. new R12 technology stack requirements.

So, their baseline for network bandwidth was the Release 11i.5.10.CU2 environment.  They then ran benchmark tests with the same Applications code and the new Release 12 ATG components.  Important note:  this configuration isn't documented, supported, or recommended -- it was strictly a Development-only configuration used for benchmarking purposes.  You cannot apply the Release 12 techstack to an E-Business Suite Release 11i environment, and we don't provide any patches to do so.]

Here's a thumbnail of the second comparison (click on it to see the full-size version, which is admittedly low resolution):

Thumbnail - Full page load comparison: Thumbnail of table comparing Release 11i and 12 page load sizes

The table above compares a Release 11i environment with a full E-Business Suite Release 12 environment.  The Release 12 environment includes not only the ATG changes but also all of the Apps code changes.  You'll see that the page sizes did increase, but these increases were mainly due to Apps code changes and user interface changes.  The new OA Framework (Swan) changes accounted for less than 4%.

New Functionality Trumps All

As Apps technologists, it's tempting for us to get caught up in these types of discussions.  "Look -- the login page is X bytes larger!" 

It's always worth remembering that your end-users don't really care much about such things, however.  They're more likely to remark on the fact that the login page now has two new capabilities:  reminding them of their password, and reminding them of their userid, too. 

It's been my experience that new functionality trumps all, at least in the eyes of your users.  My recommendation:  go ahead and consider these benchmarks as part of your Release 12 evaluations, but more meaningful comparisons will come from spending some quality time with your end-users in assessing and prioritizing the new functional benefits in R12.


Friday May 25, 2007

Release 12 Upgrade Forum Now Open

A group of users at the recent OAUG Collaborate conference in Las Vegas pooled business cards after a Release 12-related session.  Their goal:  "to share their thoughts, upgrade feedback, and driving forces for moving to R12." 

We'd like to help that group -- and you, of course -- collaborate and share experiences on this upgrade.  You're not alone.  As of early April, over 225 customers were actively working with Release 12, and almost a thousand downloads had been logged (at 41 GB, not a task for the only-mildly-curious).  Those numbers have certainly increased since then.

We've created a new discussion forum on the Oracle Technology Network:
I'm not an upgrade specialist, but I've (rather nervously) volunteered to monitor that forum and jump in where I can.  I suspect I'll be acting like a human router between forum participants and Oracle development.  This is in addition to my regular responsibilities (like this blog), so I'm bracing myself...

Monday May 21, 2007

Preventing Apps 11i Performance Issues in Four Steps

In my previous article, I talked about what to look for once you have a performance issue with Apps 11i.  In this article, I'll discuss four maintenance activities that you can do proactively to reduce the chances of encountering certain types of performance issues:

  1. Check performance against a baseline
  2. Follow a regular purging schedule
  3. Gather schema statistics regularly
  4. Follow a systematic pinning strategy


1.  Check Performance Against a Baseline

  • Create a baseline so you can monitor performance in response to changes or over time. For example, you may create 6-10 repeatable short transactions of at least 10 seconds each, which represent common functions and/or areas of particular concern.
  • Always execute this baseline test from the same PC and the same location in order to get consistent results.
  • Rerun the test as part of your normal User Acceptance Testing for any system changes.
  • Review and update your performance baseline as part of any upgrade project.

2.  Follow a Regular Purging Schedule

You need to ensure that any data that needs to be purged is scheduled on a regular basis. The data that can (or should) be purged will vary between different products, so confirm the recommendations for the specific products you are using.  All customers will generally need to schedule purging activities for the FND and Workflow products.

Concurrent Jobs to Purge Data

Most customers will need to schedule these Concurrent Purge processes:

  • Purge Obsolete Workflow Runtime Data (FND)
  • Purge Debug Log and System Alerts (FND)
  • Purge Signon Audit data (FND)
  • Workflow Control Queue Cleanup (FND)
  • Delete Data from Temporary Tables (ICX)
Naturally, your business requirements may be unique; review our purging documentation for suitability and establish your own list of jobs to run.

Workflow Specific Purging Tips

Review the following documents for Workflow-specific tips on purging:
Also keep track of number of rows in underlying Workflow tables to ensure they are not continually increasing, to ensure the data really is being purged

3.  Gather Schema Statistics Regularly

In general, running this monthly to bi-weekly should be sufficient with 10%, unless there is any known data skew. As with any generic suggestion, this would need to be proven for suitability on your own environment. For example, it is more important to run this when your data distribution changes, rather than when the amount of data changes.

If your environment is a 24x7 system, you should pick "N" for "Invalidate Dependent Cursors" to prevent fragmentation of the shared pool

For more details about gathering statistics, see: 4. Follow a Systematic Pinning Strategy

Despite new 10g features making ORA-4031 errors a rare occurrence, it is still recommended to have a pinning strategy, even with Apps 11.5.10 running on 10gR2 databases.

General guidance

  • Monitor X$KSMLRU for candidates to pin (> 4100 bytes)
  • Do not pin more than 20% of the Shared Pool
  • Review your Pinning Strategy for changing business cycle, such as month end or overnight batch runs
  • No need to pin objects used only for batch jobs
  • Also have an "un-pinning" strategy
For more information about pinning in Apps environments, see:


Running regular maintenance tasks and ongoing monitoring are essential activities to ensure that your system is performing to the best of its ability. This article highlights a few of the areas that are sometimes overlooked in Apps DBA schedules.


Friday May 18, 2007

Debugging General Performance Issues with Oracle Apps

Identifying performance bottlenecks can sometimes be a black art with any distributed computing system.  It is sometimes difficult to know where to start.  This article gives some high-level guidance on the sort of information you may need to gather in order for Oracle Support to assist you with this task for the E-Business Suite.

Performance issues can potentially occur across any or several different areas of the Technology Stack, or may be restricted within Functional Code. For example (but not restricted to)

  • Architecture issue (e.g. high latency WAN, firewall)
  • Operating System problem or resource constraint
  • SQL or general RDBMS configuration issue
  • Database Deadlock
  • Apache Listener


Types of Performance Issues

"Simpler" Problems

These issues will hopefully be relatively straightforward to define and investigate.  For example: -
Single report consistently slow, SQL trace discovers one or more SQL statement(s) taking the majority of the elapsed time

More complex issues

These can take more work to be able to confidently define the real issue and often involve complex investigations involving different parties and much data gathering and analysis

For example :-
Any intermittent issue, System wide issues, Issues where SQL trace time represents only a small proportion of the elapsed time

Gathering Data for Performance Issues

It's helpful to follow a systematic process for investigating performance issues.  First steps include identify the nature, scope and extent of the problem. 

1. Identify the Extent of the Problem

Determine whether the extent of the problem is:

  • System Wide
  • Confined to one Technology Are. For example does it only happen in one of the Self Service, Forms or Reports areas
  • For a specific Product Area(s) For example, were it to only occur in HR, GL or iStore
  • Single Report/Form/Page
  • Which Instance or instances is the problem observed. Can it be reproduced in UAT/TEST instances

2. Narrow the Scope Further

Determine whether the problem occurs for:

  • All users
  • Only one geographic location
  • Only for users with certain Browser/type of PC
  • Only during peak periods

3. Qualify the Nature of the Problem

  • Is the problem intermittent or reproducible?
  • Is the problem due to slow performance or a process hanging or spinning?
  • When was the last time the process was completed without experiencing poor performance?  Document any changes since then.
  • Is there a workaround available?   For example, restarting Browser or restarting Apache.
  • What is the frequency of occurrence?

4.  Capture Additional Clues

  • Document and differentiate factual information from user perceptions. Example - users may say "it's always slow" but reality may be that it takes 10 seconds most of the time, but 40 seconds between 10am and 11am (peak load).
  • How long does it take for the process to complete when running slowly?  How long did it previously take (before having the performance issue)?  What is the expected performance for this process?
  • What is load on server(s) when issue occurs (CPU/Memory/Network)?  Is there any unusual activity when problems occur?  For example, memory used suddenly growing or CPU at 100%.
  • Is the problem reproducible with other browsers, such as Firefox, as well as with Internet Explorer?

5.  Identify likely causes and eliminate other possible causes

  • Are there any customizations?  If so, can they be eliminated for testing purposes?
  • If the problem only occured since patches have been applied or any configuration changes have been made, can these be reverted?
  • Do you have any Resource Limits enabled on the database that may be effecting the Applications users runtime or Concurrent Manager?
  • Does the number or frequency of certain Concurrent Requests correlate to performance issues occurring?
  • Have the database parameters been configured as per the current recommendations in Database Initialization Parameters and Configuration for Oracle Applications 11i (Metalink Note 216205.1)?
  • Does AWR or Statspack show anything unusual for Top waits, Top buffer gets, etc?
  • Have you searched Metalink generally, but specifically reviewed Recommended Performance Patches for Oracle E-Business Suite (Metalink Note 244040.1) for any known performance issues, tuning guidelines and/or patches?

Getting Help from Oracle Support

Once you understand the issue and reach the point where you need Oracle Support involved, it may be useful for you to review the "Performance Tuning" section of:

The first key decision you need to make is selecting a product code for your Service Request (SR): 

  1. If the issue is with an individual Form/Report/Page or only in one product area then log the SR for that particular product support team
  2. If issue seems to be with the Technology Area or System wide then log the SR with the AOL team (Oracle Application Object Library)

It is very important to provide a good problem definition (nature, scope, extent) so be as verbose as needed to give a good description of the issue.

It is also very important to reproduce the issue on a non-Production instance. If you are able to reproduce the problem outside of your Production instance, then any recomended patches or changes can be quickly assessed for their impact and any detailed debug or tracing required to identify the issue can be easily implemented.

A Quick Aside:  One issue per Service Request

You may need several issues investigated simultaneously and may be reluctant to raise different SRs or have different people investigating. Unfortunately, even similar-looking issues can have different root causes, which means they will need to go to different support teams or have different SR statuses. This is why it is important to ensure each issue is raised as a separate SR.

Performance Issues with Specific Forms, Reports, or Pages

For these "simpler" types of issues, we can generally track down the issue with the following information:

  • Form, Report or Page name, version and navigation path
  • Full versions of Forms, Reports or Framework as well as the iAS and Database versions, including rollup or interop patches applied
  • Relevant Family Packs or Maintenance Packs applied
  • Description of symptoms, what have you tried, your investigations, conclusions and/or thoughts/ideas
  • Does it reproduce constantly, in other environments as well?
  • When did you last run "Gather Schema Statistics" and at what percentage?
  • When did you last run any relevant purging processes?
  • SQL trace with binds and waits (raw file and TKPROF output)
  • Wall clock time elapsed from user perspective
  • What are the target and acceptable times for this process?

Performance Issues for Other Areas

These are the more complex issues and will normally require additional information (and patience) to resolve.  We will need the same information as listed above, but also:

  • Description of your System Architecture and network diagram
  • Technology Stack configuration files (Apache, Forms, Reports and Database, as relevant to the problem)
  • List of any Metalink notes or product documentation you have reviewed already, and what steps you implemented from this documentation
  • List of any patches applied specifically to try to address this issue
  • Details of profile options or configuration settings you have tried changing, explaining why you tried these settings and what effect they had (if any)
  • AWR or Statspack output for "good" and "bad" performing time periods
  • Details about your pinning, purging and gather schema statistics strategy
  • Relevant Log files
  • Debug and trace files

Good starting points for collating all this information are:

Additional Tips for Troubleshooting for Apache / JVM problems

Additional Tips for OA Framework-related issues

  • It is often useful to enable "STATEMENT" level logging (for ONE user only!) using the FND:Debug% profile options if you have a reproducible test case


Performance issues can sometimes be tricky to isolate, particularly those having more than one root cause.  This article has presented some ideas about the approach to take, in addition to the sort of information Oracle Support would likely be asking for if you need to log a Service Request.

If there is sufficient demand, I can write further articles in future, expanding on the topics introduced here.


Thursday May 17, 2007

Performance Tuning the Apps Database Layer

Performance tuning is an art that should be executed systematically.  I assume almost everyone has heard this from others talking about tuning, but it's often repeated because of its fundamental truth.

Whenever there's a response time issue for the E-Business Suite, it is initially treated as a performance problem.  I would go a step further and say that poor performance is not a problem by itself.  It is a result of a root cause which lies somewhere else.

Let's dig into each of the possible places for this root cause in the E-Business Suite's technology stack.  Oracle Applications has the following layers:

  • Operating system
  • Database
  • Techstack components (Concurrent tier, Forms tier and iAS techstack)
  • Application code (Forms, Jsp etc)

This article will touch on performance-related issues for the database layer. 

Possible Causes for Database Layer Performance Issues

Possible issues include:

  • Core optimizer issues
  • Known performance-related database bugs
  • Incorrect statistics to the cost-based optimizer (CBO), which is responsible for SQL optimization
  • Incorrect System Global Area or Program Global Area sizing

Other areas like locking and latching affect performance, too.  For the limited purposes of this article, we'll focus on these four setup-related areas.

Core Optimizer Issues

Most of the E-Business Suite database's recommended parameters are listed in:

Setting the values of memory-related parameters as is not a one-time job.  We recommend that you periodically review your Statspack and AWR reports to find if the your current settings meet your current load requirements.

For recommended database init.ora parameters, refer to Note 216205.1.  Pay attention to mandatory parameters.  Based on our benchmarks, we recommend that your Apps setup adhere to the init.ora parameters listed in the note.

Known Performance-Related Database Bugs

For issues with the Oracle optimizer and database, most of the known performance-related bugs and their recommended patches for E-Business Suite environments are listed in:

Incorrect Statistics to the Cost-Based Optimizer (CBO)

Now comes the most complex -- and paradoxically, the easiest to solve -- area:  ensuring that the cost-based optimizer works with accurate metadata.

Before going into the details, I'll remind you that the rule-based optimizer (RBO) is no longer supported.  Don't use it in your E-Business Suite environments.  If you have an optimizer-related issue that appears in the cost-based optimizer but not the rule-based optimizer, log a Service Request with Oracle Support.

Maximizing the Benefits of Cost-Based Optimization

The cost-based optimizer is not infallible, but it's a DBA's responsibility to ensure that correct metadata is available to the optimizer.  There are two things that system administrators should do:

    1. Set all of the optimizer-related parameters, following Note 216205.1
    2. Gather statistics for all database objects

Gathering Statistics for Database Objects

There are many different ways of gathering statistics:

  • Use the Analyze command
  • Use Dbms_stats
  • Use Fnd_stats

For Oracle E-Business Suite environments, we recommend using fnd_stats.

A Short Digression:  What is fnd_stats?

Fnd_stats is a wrapper around dbms_stats that suits most of the E-Business Suite's requirements.

We recommend using fnd_stats over dbms_stats for Apps environments because of the former's support for restarts.  Starting with the 10g version of the database, dbms_stats has also this feature.

If the Gather Schema Statistics concurrent program is used, fnd_stats does the bookkeeping for the run.  Should the run fail for any reason, the next run of the program starts from where the previous run was stopped. This saves lot of time.

We also recommend  fnd_stats because of its support for histograms.  Histograms are useful when:

  1. A table's column is used in an equality or equi-join predicate AND there are  skews in the column.
  2. A table's column is used in a range or like predicate AND there are  skews in the column.

Oracle Applications' data distribution is dependent on the functionality of the specific product modules.  Based on our benchmarks, a number of columns are useful for histograms.  These columns are listed in the FND_HISTOGRAM_COLS table.

When Gather Schema Statistics is executed, it reads FND_HISTOGRAM_COLS and builds the histograms.  

gather schema/table statistics:

Back to Gathering Database Statistics

When gathering the statistics for the entire applications database, we must use the Gather Schema Statistics concurrent program.  Only the Gather Schema Statistics and Gather Table Statistics should be used. Do not use the Gather Column Statistics program.

When Should You Gather Statistics?

There is again no hard and fast rule for the interval between gathering statistics.  A general rule-of-thumb is is to run the statistics collection after a 10% increase in the database size. 

Having said that, other factors may come into play.  From 11.5.10 onwards, fnd_stats was enhanced to gather only the statistics for those objects, which have undergone a predefined percentage of data increase, or for objects that have no statistics.  The latest versions of fnd_stats identifies STALE and EMPTY statistics and gather the statistics for those objects only.  This saves lot of time and allows sysadmins to set flexible statistics gathering intervals. 

gather auto:

Although there are a number of options that you can set, we recommend selecting  the GATHER AUTO option.

Incorrect System Global Area and Program Global Area Sizing

SGA_TARGET sets the maximum size of the System Global Area (SGA) for your E-Business Suite instance.  Parameters like db_cache_size and shared_pool_size affect the database buffer cache and shared pool for that instance.   PGA_AGGREGATE_TARGET determines the maximum value for the Program Global Area (PGA) for the instance.

Note 216205.1 lists the minimum values for these parameters. These values might need to be varied depending on the actual load on your environment.  In the same way.  If these are values are:

  • Set too high:   memory on the server is not utilised to the maximum
  • Set too low:  there will be performance issues

To find out whether these parameters are set properly on your system, you can run a Statspack or AWR report.

Interpreting Statspack or AWR Reports

Covering this area comprehensively is beyond the scope of this article, but here's some things to remember about interpreting your Statspack report:

Total Response time = Service time + Wait time

Service Time is generally the amount of "CPU used by this session".  From the 9.2 database version and upwards, this is reported in the Statspack.  You can use this to derive the Wait Time.

For example:

Using a Statspack report with a 9.2 or higher database, examine the figures reported for CPU time.  If it's 70% of the total time, then the wait time is 100-70=30%.

Total Response time = 70% CPU time + 30% wait time

Once we have identified the time taking component, look into the specific component.  For memory-related areas, look at the Advisory Statistics section.  This shows details about the sizing of SGA (cache and shared pool) and PGA (M-pass executes should be as small as possible for a optimal PGA). 

Based on this type of analysis, you can vary your SGA and PGA settings accordingly.

Getting Support for Performance Issues

As you can see, investigating performance issues can be tricky, especially if you find the topics briefly described here to be daunting.  Take heart:  you're not on your own.  We have teams that specialize in these areas, so if you run into any performance-related issues, log a Service Request via Metalink and we'll jump in to help.

In Summary

A quick recap of database setup-related issues and recommendations for tracking down performance-related issues:

  • Database Parameters:  Refer to Note 216205.1 for mandatory settings, and then check your Statspack/AWR reports periodically to see whether your memory-related parameters are set correctly.

  • Apply the Recommend Performance-Related Database Patches:    Refer to note 244040.1

  • Keep Your Statistics Fresh: Use the Gather Schema/Table Statistics concurrent program regularly.  The GATHER AUTO option can be of great help here.

It's possible to dive much deeper into database or instance tuning topics.  If you'd like me to go into more details, feel free to post a comment here.


Wednesday May 09, 2007

Reducing Patching Downtimes via Shared Apps File Systems

I had a chance conversation with an Apps sysadmin late in the evening at the Collaborate conference.  He wearily noted that he had to somehow cover increasing maintenance costs with reduced levels of staffing.  The conversation suggested that our support for shared E-Business Suite application tier file systems may be one of our better-kept secrets.  If you haven't come across this yet, here's a way of reducing the amount of time that you spend patching.

Scaling Up with Load-Balancers

When your E-Business Suite user base grows beyond a certain size, it's likely that you'll look into deploying multiple application servers in a load-balanced pool of nodes.  Your deployment topology might look like this:

Generic Apps Load-balancing:

Regardless of whether you're running Release 11i or 12, each of these nodes would need its own disk space and would to be maintained and patched separately.  What a pain.

Different File System Structures in Release 11i and 12

In Release 11i, the Applications tier file system includes the APPL_TOP, the COMMON_TOP, and the Applications technology stack (8.0.6 and iAS ORACLE_HOMEs).  In a traditional deployment, each one of these application servers would have its own Applications file system, like this:

Distributed Application Tier Filesystem:

In Release 12, the Applications tier file system includes the APPL_TOP, the COMMON_TOP, and the Applications technology stack (10.1.2 and 10.1.3 ORACLE_HOMEs), plus a new INST_TOP.  Each node would have the following:

Release 12 application tier structure 2:

Enter the Shared Application Tier File System

For Release 11i, starting with 11.5.10, it's possible to put the Applications tier file system on a shared disk resource mounted to each Application tier server node in the system, like this:

Shared Application tier file system:

Similarly, in Release 12, you could put the Applications tier file system on a shared disk resource mounted to each Application tier server node, like this:

Release 12 shared filesystem:

Migrating to Shared File Systems

There are a few prerequisites:
  • Different nodes must be running the same operating system and the same O/S patches
  • You must be on a UNIX platform (Windows doesn't support shared file systems)
Certification of Shared File System Solutions

If you've gotten to this point, you're probably wondering, "Is my _____ SAN/NAS shared file system software certified with the E-Business Suite?"  File system solutions that customers have recently asked about include:
Supported but not Certified

The short answer is that your shared file system solution is supported but not certified with the E-Business Suite, for either Release 11i or 12.  Remember that there's a key distinction between support and certification, which I've covered in detail in this article:
The complexities of whatever shared disk resource management solution that you're using must be transparent to the E-Business Suite.  Aside from that, there aren't any special requirements for shared disk resources.  They can be local to the server or on a standalone disk array.

Almost Irresistible

Regardless of which E-Business Suite release you're running, the main advantage of using an Application shared file system is simplicity and ease of patching:  when you apply patches or changes to the shared disk resource, they're immediately visible on all application tier server nodes.  You patch in a single place and deploy those changes across multiple servers.  You save disk space for each additional application node you deploy, and it's easier to deploy additional nodes, too.

If you've been wondering how to squeeze more productivity out a packed roster of administration activities, I'd recommend setting up a testbed to try this out.  Feel free to post a comment with your experiences with this; I'll make sure your feedback gets back our team.

Related Articles

Tuesday May 01, 2007

Java Caching for Oracle Applications 11i: Part 1

In this article I will try to explain some of the mysteries around the "Java Caching Framework," as this is a commonly misunderstood technology (which is probably due to it being well-hidden from the casual observers view).

This article will be an introduction to the topic, rather than trying to talk about all aspects.  I intend to write a follow up article to talk in more detail about common issues, configuration and troubleshooting for Java Caching, but let's start with the basics...

What is a Cache?

In general computing terms a cache is simply a copy of some data, sometimes prefetched or perhaps precalculated data, stored away from the original data.  The purpose of the cache is to provide faster access to data, as it will be stored either closer to the user and/or in a faster access medium.

There are a multitude of caches, some which have been around since computing began.  To name a few that immediately spring to mind:

What is eBiz Java Caching?

Data from the eBiz database is stored (cached) in the OACoreGroup Java Virtual Memory (JVM) memory on the eBiz Middle Tier server.  Normally this is done only when data is first accessed by the JVM, although the technology allows data to be preloaded if the Developer chooses.

It is also important to remember that every OACoreGroup JVM has its own local Java Cache, separate from the others.  We'll talk more about that later in this article.


Why Use Java Caching?

Java caching improves Application performance because data is retrieved from JVM memory.  This saves a network trip to the database, and in turn saves the time taken for a SQL command to extract data.

Characteristics of data that is most suitable to be cached:

  • Expensive SQL query
  • Does not change very often
  • Is frequently used

Challenges of Distributed Caching

Oracle E-Business uses the DISTRIBUTED cache model, where updates to objects in one JVM are propagated through to other JVMs using the same database.  If you have multiple eBiz middle tier servers and/or multiple OACoreGroup JVMs configured, there needs to be a mechanism to keep these separate JVM Java Caches synchronized.  A developer must specify which of the following two synchronization methods to use: 

  1. JVMs talk to each other and send the updated object to the other JVMs
  2. A JVM will just send an invalidation message for a specified object.  This is more commonly used.

An additional consideration is how to handle updates to data that happens outside of the JVMs, such as with updates via the Forms interface or a batch process.  For this case, an invalidation business event is triggered from the database, sending the JVMs an invalidation message for any records that have been updated.

If an object is invalidated in a JVM, it will be reread from the database when next accessed.  The new version will then be stored in the JVM's cache.


A Brief History of Java Caching

Java Caching has been around in the E-Business Suite for quite a long time.  It was not used by many Apps products until 11.5.10, as this was the first release where Database Invalidation was introduced.  I believe iStore was the first team to utilize Java Caching functionality in any depth.  Since 11.5.10 CU2, more and more product teams are using Java Caching.

Understanding How to Patch Java Caching

There are several different dependent layers to Java Caching, which is implemented as Java Classes stored on the Apps Web Tier:

Application Server packages (oracle.ias.cache)
eBiz leverages Java Caching mechanism implemented by the iAS Technology itself.   It is sometimes required to apply iAS Technology patches for Oracle9i Application Server  You should only apply patches that are released by the eBiz Applications Technology Group (ATG) team.

At the time of writing, the latest patch for this layer is:


JTF and FND packages (oracle.apps.jtf.cache and oracle.apps.fnd.cache)
Updates to these packages are shipped via normal Apps patches.  Originally released via JTT product patchsets, these patches are now rolled into FND patchsets such as ATG RUP4.   The eBiz ATG development team is responsible for this layer.

At the time of writing, two patches are recommended on top of ATG RUP3 and ATG RUP4

Functional layer

It is up to the individual Applications product teams to utilise the Java Caching Framework provided by ATG.  Product teams are responsible for creating and maintaining their own "buckets" (more properly called a "Component Cache") within the Java Cache, and to control the invalidation mechanism for their own objects.   

Updates to code on this layer are released via product-specific patches.   Unfortunately, a list of product-specific patches is beyond the scope of this article.

Troubleshooting Common Problems

Updates to data not being shown immediately

A commonly reported symptom is that changes to users' responsibilities are not taking effect immediately.  Customers report this problem after applying the ATG RUP3 patch.  The Applications Technology Group implemented Java Caching for responsibilities (amongst other things) in ATG RUP3, so this is often the first time customers see Java Caching in action (or perhaps I should say "inaction" in this case).

If you're encountering this behavior, the following Note should help:

Java Caching through a firewall
Java Caching uses a separate set of port numbers for communication between JVMs.  These ports need to be opened through a firewall for Java Caching to function.   These ports are defined by the following profile options:
These profile options are configured in AutoConfig via these CONTEXT.xml file variables:
  • s_fnd_cache_port_range
  • s_java_object_cache_port

Run the Java Cache Diagnostic Tests

If you are experiencing any symptoms that could be attributed to Java Cache problems, it is important to gather the information from the Diagnostic Tests.  For example, one test will prove if database invalidation is functioning correctly.

These tests were introduced in JTT.E (11.5.10) and can be accessed via the "CRM HTML Administration" responsibility:

  • Diagnostics--> CRM Foundation--> Caching Framework

Review the javacache.log

This log file is located in $COMMON_TOP/rgf/<Instance >_<Hostname> and provides error messages specific to Java Caching.


Java Caching is simple in concept but can be complex to troubleshoot.  This article has hopefully given you some insight into the fundamental concepts.  We'll delve deeper into some of these concepts in my next article.



Tuesday Apr 10, 2007

Getting Personal with OA Framework Pages

I often receive questions similar to the following, regarding personalization of E-Business Suite (EBS) HTML-based pages:

"How can we quickly determine if particular E-Biz pages support personalizations?"

First of all, I assume we're talking about the Oracle Application Framework (OAF) Administrator personalization, not Personalized Views (Saved Searches).

All pages built using OA Framework are personalizable by default--the page developer has to do something specific to make the page be non-personalizable.  One thing a developer can do to make a page (usually just a region or field) non-personalizable is to set a specific property to false during development.  The other is to create regions or fields programmatically (so they don't exist in the page definition that is stored for using with personalizations).  These are generally the exception cases.

Some HTML Products Don't Support OAF Personalization

In 11.5.10 (11i), many E-Business Suite CRM products and some other products were built using a different technology stack (JTT/JTF), so they couldn't use OAF personalization.  In EBS Release 12, most of these have been rebuilt using OAF. 

EBS Business Intelligence products have their own version of personalization, and they do not use the OA Framework feature even though they are built with OAF.  Business Intelligence products have somewhat different implementations of their personalization depending on whether you are looking at a dashboard or a report page.  But the way to tell if it's a BI page is to look in the upper corners.  If you see an Actions dropdown, it's a BI page.

BI Page screenshot:

But Which Ones Are Built Using OA Framework?

Sometimes people have trouble telling which pages are created using OA Framework.  Here are several ways I use to tell if a page is built using OAF:

The best and easiest way I know of is to look for "OA.jsp" in the URL of the page.

Is OA.jsp in the URL screenshot:

You can also use the About this Page feature to confirm if a page is an OA Framework page.  If you don't have it already, you'll need the FND: Diagnostics profile option set to Yes (you may need your administrator to set this).  Go to a product page (I'm using iProcurement here) and click on the About this Page link:

About this page screenshot:

If you see that the first line of the page definition is "pageLayout", it's an OAF page.

Page definition starts with pageLayout screenshot:

Beyond looking at each page individually, you can use the About this Page feature to get a whole list of pages built with OAF, all at once.  Since you're already there, simply go look at the page context (menu), as follows:

About this Page > Page Context > Menu (Expand All)

Most of the functions in the menu display a URL, so you can see at a glance which ones start with "OA.jsp":

Lots of OA Framework Pages screenshot:

Another way to check for a single page is to use View Page Source in the browser.  Early in the source, you will see the following for an OAF page:


You can also list the pages (and more) using the JDR_UTILS package.  It is described in the "Inspecting the MDS Repository Content" chapter of the Oracle Application Framework Developer's Guide for your release version (available on MetaLink).

Finally, you can set the "Personalize Self-Service Defn" profile option to Yes to allow personalization, and then look for the Personalize Page link on the top of each page.  Of course you can also use the link to see the personalization hierarchy page (or the context page if you are using 11.5.9 up to 11.5.10 CU1--it changed in 11.5.10 CU2).

Note that all of this applies to both Release 11.5.10 and Release 12 of the E-Business Suite.  It also applies to EBS 11.5.9 if you applied OAF 11.5.10 to it as part of a patch.

To learn more about OA Framework Personalization, see the Oracle Application Framework Personalization Guide in the Oracle Applications Online Documentation Library.

Here's an Extra Tip: Hidden Fields Between the Fields 

Some product teams build features into their pages that are hidden by default because they expect most customers won't use them.  These include extra fields for special purposes, hidden Additional Information regions containing descriptive flexfields, and so on.  We recommend to product teams that they explicitly document places in their applications where customers are expected to personalize the pages.  Check your product's documentation for such updates.  I know that iProcurement, at least, mentions such hidden fields directly in the text of their implementation manual.  I found one on page 2-17 of the Oracle iProcurement Implementation and Administration Guide, Release 12, Part No. B31402-01.  Also, the Oracle Workflow Administrator's Guide and Oracle Workflow User's Guide both have appendices highlighting some of the personalizations for Workflow pages that customers may be most interested in using.

Happy personalizing!


Wednesday Apr 04, 2007

Fully Automated Cloning for Release 11i and 12

Cloning is the process of creating an identical copy of an already existing Oracle Applications system.  Cloning is a regular part of an Apps DBA's responsibilities, so you're probably already familiar with the latest Rapid Clone utility and its predecessor,  adclone.pl, the original AD Clone Utility.  If you've used those tools in the past, you likely also know that overall cloning process requires some user intervention for things like the APPS password and so on. 

Enterprise Manager Screenshot:

A customer recently asked whether the following cloning process can be fully automated, so that no user intervention is required at all:
  1. Shut down the PROD environment for both the application server and database tier
  2. Do an offline preclone of both tiers
  3. Copy both tiers to another server (the sandbox environment)
  4. Restart the PROD environment
  5. Do an offline postclone of both tiers for the sandbox environment
  6. Start the sandbox environment
In researching that, I was pleased to learn that this kind of automation is a new feature in the latest Application Management Pack for Oracle E-Business Suite for Enterprise Manager Grid Control 10gR3. 

The management pack automates the cloning of both Release 11i (11.5.10 with ATG RUP4) as well as Release 12 environments.  You go through a step-by-step interview process to configure the clone routine once, and then you can reuse the cloning routine as many times as you wish.  In addition, the cloning routing can send out a notification if it failed.

Very slick.  If you've had a chance to try this, I'd be interested in hearing about your experiences with this new feature.


Monday Mar 05, 2007

Security Best Practices for Release 12

If you're working with Release 12, you'll be pleased to hear that our Applications Technology Group Security team has just published a new document detailing our best practices security recommendations for this release.

Like its Release 11i cousin, this document covers the following topics for Release 12:

  • A framework for securing different segments of your E-Business Suite deployment, starting with the operating system
  • Pointers to essential Apps security reference materials, security alerts to monitor, and recommended patches
  • Guidance for securing internal deployments, including the database at the schema level and for database net access, the Apps web tier, and end-user PCs
  • Tips for hardening your EBS security setup
  • Monitoring security through Oracle Applications Manager (OAM)
  • Guidance for securing externally-facing deployments in DMZs, including the use of Responsibility filters, URL filters via the URL firewall, noise
    filters via mod_security
For details, see:


« June 2016