Sunday Jan 10, 2010

Mini-Review: Oracle E-Business Suite Development & Extensibility Handbook

Cover of Oracle E-Business Suite Development and Extensibility Handbook
I tend to read technical books, so I was pleased to be given the opportunity to review a book co-authored by an old colleague: 

Despite the personal relationship, I will try to not let this cloud my opinion of this book.

This Oracle Press book sets out to provide a single resource to cover all aspects of developing in an EBiz environment.   It is aimed at developers or professionals who are either starting out with eBiz, or have some Apps knowledge but may not have experience of all the technologies discussed.

The first four chapters are a brief introduction to eBusiness Suite itself and some key concepts that developers would need to understand, such as Multi-Org, Flexfields, Auditing and Logging.

The following chapters takes a topic at a time and gives a brief introduction into what the technology provides and a brief overview of how it works and what it does.   The meat of each chapter is a detailed step-by-step guide on how to create a simple "Hello World" type of customization in a Release 12.0 VISION environment, interspersed with handy hints of any "gotchas" you may come across.  Finally there are some best practises and general comments for the topic, such as coding standards or implementation tips.  

The chapters cover:

  •  Concurrent Programs
  •  Forms
  •  Reports
  •  BI Publisher
  •  OAF
  •  CLAF
  •  Workflow
  •  XML Gateway
  •   Moving AOL Objects between instances
  •  SOA
  •  SQL performance

Although I did not try out the step-by-step examples myself, I did read them through and they made sense to me.  In the main, I found the writing style to be straightforward and easy to read.  The authors do not assume prior knowledge and soon get you into the interesting part of the chapter, so there is not too much preamble.   Concepts are explained concisely with examples where needed, so a newbie will be able to grasp the knowledge that is being imparted

I was a little disappointed in the Workflow chapter, in that the chapter introduction seemed a bit confusing, but the examples were quite interesting.  I also thought the SOA chapter was very short.  It would also have been nice if the code used in the book was available online.

My personal highlights of the book were the OAF chapter, which was the longest chapter and covered a multitude of useful areas, and the CLAF chapter, which was very illuminating even for me, as this topic is a poorly documented area.

The main drawback is also the main advantage of this book, in that they do try and cover everything in one place.   The authors themselves acknowledge that some of the topics covered deserve a book in their own right (and many of them do have such books available).  

Overall, if you are looking for a simple overview of the main development techniques with eBusiness Suite, then this book would certainly be worth investing in.  Although the topics are covered fairly briefly, I believe there is enough discussed in each chapter to allow the basics to be picked up and give a good grounding for further detailed reading into the topic

Related Articles

Saturday Mar 15, 2008

New Sun Java JRE Plug-In Certification Policy for Apps 11i & 12

Our Oracle E-Business Suite Technology Integration team has just announced a significant change in its certification and support policy for the use of the native Sun Java Runtime Engine (JRE) with the E-Business Suite.  Effective earlier this month, this new policy reflects a switch from certifying specific JRE versions for the E-Business Suite to specifying minimum versions, instead.  This permits E-Business Suite users to run any JRE release above the minimum certified level, even later ones that Oracle hasn't explicitly tested with the E-Business Suite.


This policy change applies to the use of Sun's Java plug-in to run Oracle Forms-based content in both E-Business Suite Release 11i and 12.  This change to Oracle's certification and support policy reflects our confidence in Sun's continued ability to preserve compatibility with Oracle Forms in future JRE releases in both the 1.5 and 1.6 codelines. 

This significant policy change should make life easier for Apps DBAs and end-users, as it widens the scope of JRE plug-in versions that can be used on your desktops.  This should help minimize the risk of experiencing conflicts between multiple JRE plug-ins, as described in this previous article.

Which JRE Versions Are Certified With Which Browsers?

This depends on your Windows operating version and the browser that your firm has deployed.  Various combinations of Windows XP, Windows Vista, Microsoft Internet Explorer (IE) 6 and 7, and Firefox are now certified with JRE 1.5 and 1.6. To review the new minimum certification levels, see:

Compatibility With Later Sun JRE Releases

You should review the documentation above and check Certify to confirm the current situation on a regular basis, as there may be specific warnings or exclusions in the future.  Both documents state:-

Oracle will continue to test and certify the Oracle E-Business Suite with selected future versions of Sun JRE releases in advance of their general availability to the public. Oracle will update this documentation with known compatibility issues or workarounds as needed.

Why Switch From JInitiator to the Sun Java Plug-In?

Remember that Oracle has started the obsolescence process for JInitiator!  For an overview of why you might be interested in switching your users to the Sun Java plug-in, and discussions about supported architectures, prerequisites, and other deployment considerations, see:

Wednesday Feb 27, 2008

JInitiator 1.3 To Be Desupported for Apps 11i in July 2009

With the onward march of progress, I would like to draw your attention to the newly-published Obsolescence notice for JInitiator 1.3 for E-Business Suite 11i (Metalink Note 552692.1).


As of July 31, 2009 Error Correction Support for JInitiator 1.3 will end.  In other words, the Oracle Forms group will no longer issue bug fixes via new versions of JInitiator 1.3.x as of July 31, 2009.

What Do You Do Now?

You need to start the upgrade process for your end-users' desktops to the native Sun Java plug-in as soon as possible.  This extract from note 552692.1 sums it up well by stating :-

All E-Business Suite customers using Oracle Jinitiator 1.3.x need to upgrade to the Sun JRE (Native Plug-in). Oracle highly recommends that EBS customers upgrade to the latest certified version of the Sun plug-in by following Metalink document 290807.1, "Upgrading Sun JRE(Native Plug-in) with Oracle Applications 11i"

Why Choose the Native Sun JRE Over JInitiator?

Nearly every business desktop has multiple Java clients installed.  These Java clients often clash; see this article for a more in-depth discussion about JRE conflicts.  The main benefit of switching from JInitiator to the native Sun Java plug-in lies in reducing  conflicts between these clients on your desktop.

In addition, Steven's previous blog article elegantly describes the reasons why Oracle is moving to using the native Sun Java plug-in in favour of JInitiator:

Oracle JInitiator was originally released as a licenced version of Sun's Java client back when specific features were required to support Oracle Forms.  This was particularly necessary for E-Business Suite Forms-based functionality, since the E-Business Suite pushed the envelope of what Forms and Java could do collectively from a user interface perspective.

Sun has since incorporated all of the enhancements needed to support Oracle Forms into their native Sun Java plug-in.  As a result, the Oracle JInitiator team is pleased that they can get out of the business of maintaining and repackaging Sun's Java client code. 

Oracle's emphasis from this point forward will be to certify future versions of the Sun Java client with the E-Business Suite.


Friday Feb 01, 2008

Diagnosing Sun Java Plug-in Issues with Oracle Apps


As discussed in previous articles, the native Sun Java Runtime Engine (JRE) is certified for Windows-based desktop clients accessing E-Business Suite Releases 11i and 12.   The native Sun JRE is used to run Forms-based content in the E-Business Suite.  It is the only plug-in certified for Release 12, and can be used in place of the older Oracle JInitiator 1.3 plug-in certified with Release 11i.


This article spotlights Metalink notes which guide you through the tracing options and some gotchas when using the native Sun JRE with the Oracle E-Business Suite.

Enabling JRE Tracing and Logging

There are several notes available which explain how to enable these options for the JRE:-

I like this note the best as it just describes the simplest way to enable logging, tracing and java console (OK, yes I admit I wrote it...)

Both of these notes have a "JRE Tracing options" section describing the steps, however the method described to enable the logging is the more long-winded method (in my view).

This note describes all the different methods to enable logging, tracing and the Java Console and has lots of pictures. The slight drawback is that it doesn't recommend a particular method.

Potential "Gotchas"

  • Rnning Multiple JRE Versions on the Same Desktop Client

This issue is fully described in the "Upgrading Sun J2SE" notes referenced above as well as in this article, but is worth spotlighting here.  If your organisation uses runs Java-based applications that depend on different JRE releases, you will need to consider static versioning issues when deploying the native JRE required by the E-Business Suite.

  • Potential Character Set Issues


There are different dependencies with Native JRE compared to Jinitiator, as the following note highlights.  In this case one Client PC using Japanese language did not have the necessary Language code pages installed.   This Note could apply to other languages also:

  • Potential Issues When Connecting via VPN

This issue applies equally to JInitator, but thought I would mention the following Note as it seems to be becoming more popular for customers to run
the E-Business Suite over a secure VPN connection:

Related Articles

Thursday Sep 20, 2007

Java Caching for Oracle Applications 11i: Part 2

In my previous article "Java Caching for Oracle Applications 11i: Part 1" I talked about the basics of Java Caching and how it works conceptually.   


My original intention with "Part 2" was to discuss how to diagnose issues with Java Caching, but I got a bit carried away with scripts and the like so ended up creating some Metalink notes instead. 

Diagnosing Database Invalidation Issues

Here's the first one:

As it says on the tin, this note covers diagnosing issues with Database Invalidation.   The classic symptoms being that when Responsibilities are added to a user, they are not appearing immediately (but do after Apache is bounced).

Although the scripts are listed in the note, you can also download a soft copy via the link mentioned in the note.

Diagnosing Issues with Responsibility Assignments

This issue is not actually a Java Caching problem, but the symptoms initially look similar to those described above.  The key difference here is that the affected responsibilities still do not appear after Apache is bounced.  For details, see:

More on NoClassDefFoundErrors

This one seems to be a bit of a "phantom illness" as I have seen two or three customers hit this issue, but after following the Action Plan in the note the problem is not reproducible anymore.   Even more bizarrely, after reverting the changes introduced by the Action Plan, the problem doesn't come back!    This one has piqued my interest, so if anyone out there has the same symptoms and can still reproduce the problem after reading the following note, then I would be happy for you to email me your SR number.

Hopefully you will find these notes useful towards understanding and diagnosing any Java Cache issues you find yourself facing.


Tuesday Jul 24, 2007

10 Tips for Protecting your APPS Password

E-Business Suite security is a huge topic, as there are many different facets to consider. This article will consider a small but essential part of the security model: protecting your APPS user password.

The APPS user in the E-Business Suite is the master of its world. I have gathered together some thoughts on steps you can take and things to consider to help you protect your APPS password from being compromised:

  1. Stay current with our latest Security Best Practices

    Regularly review the latest version of Best Practices for Securing Oracle E-Business Suite (Note 189367.1). This note is regularly updated and will give security advice covering many different aspects of Applications 11i.    For Release 12, see Best Practices For Securing Oracle E-Business Suite Release 12  (Note 403537.1)

  2. Regularly change your APPS password

    This is an essential activity from a security perspective and needs to be part of your routine operating procedures. Same applies for other schema passwords and SYSADMIN user. As an aside, don't use predicable passwords, or have a system to create passwords, such as using "0ct0ber" for the password in October as this will make it easier to guess

  3. Always change passwords as part of a clone process from PROD

    It is recomended to change ALL schema passwords and ALL eBiz user passwords in a cloned instance.   You can use Removing Credentials from a Cloned EBS Production Database (Note 419475.1) to achieve this.    Similarly, you don't want to have any relation in the passwords used for PROD compared to any other instances. Data masking and obfuscation is a large topic in its own right, but is also something you may need to consider doing for the cloned instance to protect sensitive data generally. With Release 12, EM plugin provides some data scrambling facilities.

  4. Perform data masking on any files sent to outside parties from the PROD system

    When you need to send any log files or configuration files, ensure that you scan for any sensitive data before packing the files to be sent. In this article we are concerned about the APPS password, but this applies equally well for other data as well. For example, a crude mechanism would be to use "ed" or "sed" on all files to globally change any occurrences of the APPS password before creating a tar archive to email or upload. You may be uploading files to Oracle Support, or just emailing them within your Organization. Whenever the files are going to someone who cannot access them directly you should always check the files before sending.

  5. Create separate schemas with minimal access required for direct database access  

    If anyone requires direct access to the E-Business Suite database, ensure that you create a new unique schema with the specific permissions required for them to perform their job role. Except for a very few Apps DBAs, there should be no reason that anyone else needs the APPS user password. Sometimes pressures of work make it easier to just give someone APPS access, but this should be resisted and the time taken to provide only the minimum access absolutely required. Every person should also have their own unique login (but this is digressing into a separate area that I'll address in a later article).   When considering permissions to allocate, don't be tempted to give read only access to everything, as being able to read sensitive information may be just as damaging as being able to change it.

  6. Protect Apps 11i middle tier file system files

    These days, there is little need to give anyone UNIX-level access to the servers, but it is still important to ensure the "applmgr" operating system user password is well protected. Also consider whether any of your own startup scripts or monitoring scripts have the APPS password hard coded in them, and protect these scripts with chmod 700 permissions, or remove them if no longer needed

  7. Ensure no processes are running with APPS username/password in command line

    Generally the APPS password is not listed in "ps" output, but there may be some manual scripts or other processes intermittently running with the APPS password in clear text or trivially encoded. Ensure these scripts are changed to hide the APPS password. In addition, ensure operating system access is restricted to only those who really need it

  8. Protect OID access

    If you have integrated the E-Business Suite with Oracle Application Server 10g, Single Sign-On, and Oracle Internet Directory, then the Apps user password is stored in the OID database, as it is required for Provisioning to function. The OID administrator and anyone with ldapsearch rights in the Provisioning Profiles will be able to extract the APPS password from OID. This in turn implies the "AppsDN" OID password should be protected in the same way as the APPS password itself.    For assistance in security hardening OID, refer to the Oracle Internet Directory Administrator's Guide 10g ( - Part III Directory Security

  9. Encrypt SQLNET traffic from Middle Tier to RDBMS

    In a previous article, Steven highlighted that ANO is certified with the E-Business Suite. Use encryption to protect the APPS password from network sniffers tracing SQLNET connection packets and deciphering the APPS password on the wire.

  10. Allow only specific IP addresses to access RDBMS via SQLNET

    Slightly off topic, but if someone has acquired the APPS password they still have to be able to gain access to a tool that can use it. Restricting the IP addresses that can access your Apps database will help minimise this risk. If you are still using "fat" clients (Discoverer or ADI for example) then you will have to weigh up the risks against the administrative overhead. Oracle recommends upgrading to server-based equivalent tools or shared desktop technologies such as Citrix so desktop clients no longer need direct access.  This topic is discussed further in E-Business Suite Recommended Set Up for Client/Server Products (Note 277535.1)


Defence in depth is generally considered the best approach so hopefully these recommendations will give you some food for thought when you are reviewing how well your own system is protected.

Sound password policies are critical to enforce access policies and enforce individual accountability.  You need to jealously guard your passwords, particularly for the APPS user.


Monday May 21, 2007

Preventing Apps 11i Performance Issues in Four Steps

In my previous article, I talked about what to look for once you have a performance issue with Apps 11i.  In this article, I'll discuss four maintenance activities that you can do proactively to reduce the chances of encountering certain types of performance issues:

  1. Check performance against a baseline
  2. Follow a regular purging schedule
  3. Gather schema statistics regularly
  4. Follow a systematic pinning strategy


1.  Check Performance Against a Baseline

  • Create a baseline so you can monitor performance in response to changes or over time. For example, you may create 6-10 repeatable short transactions of at least 10 seconds each, which represent common functions and/or areas of particular concern.
  • Always execute this baseline test from the same PC and the same location in order to get consistent results.
  • Rerun the test as part of your normal User Acceptance Testing for any system changes.
  • Review and update your performance baseline as part of any upgrade project.

2.  Follow a Regular Purging Schedule

You need to ensure that any data that needs to be purged is scheduled on a regular basis. The data that can (or should) be purged will vary between different products, so confirm the recommendations for the specific products you are using.  All customers will generally need to schedule purging activities for the FND and Workflow products.

Concurrent Jobs to Purge Data

Most customers will need to schedule these Concurrent Purge processes:

  • Purge Obsolete Workflow Runtime Data (FND)
  • Purge Debug Log and System Alerts (FND)
  • Purge Signon Audit data (FND)
  • Workflow Control Queue Cleanup (FND)
  • Delete Data from Temporary Tables (ICX)
Naturally, your business requirements may be unique; review our purging documentation for suitability and establish your own list of jobs to run.

Workflow Specific Purging Tips

Review the following documents for Workflow-specific tips on purging:
Also keep track of number of rows in underlying Workflow tables to ensure they are not continually increasing, to ensure the data really is being purged

3.  Gather Schema Statistics Regularly

In general, running this monthly to bi-weekly should be sufficient with 10%, unless there is any known data skew. As with any generic suggestion, this would need to be proven for suitability on your own environment. For example, it is more important to run this when your data distribution changes, rather than when the amount of data changes.

If your environment is a 24x7 system, you should pick "N" for "Invalidate Dependent Cursors" to prevent fragmentation of the shared pool

For more details about gathering statistics, see: 4. Follow a Systematic Pinning Strategy

Despite new 10g features making ORA-4031 errors a rare occurrence, it is still recommended to have a pinning strategy, even with Apps 11.5.10 running on 10gR2 databases.

General guidance

  • Monitor X$KSMLRU for candidates to pin (> 4100 bytes)
  • Do not pin more than 20% of the Shared Pool
  • Review your Pinning Strategy for changing business cycle, such as month end or overnight batch runs
  • No need to pin objects used only for batch jobs
  • Also have an "un-pinning" strategy
For more information about pinning in Apps environments, see:


Running regular maintenance tasks and ongoing monitoring are essential activities to ensure that your system is performing to the best of its ability. This article highlights a few of the areas that are sometimes overlooked in Apps DBA schedules.


Friday May 18, 2007

Debugging General Performance Issues with Oracle Apps

Identifying performance bottlenecks can sometimes be a black art with any distributed computing system.  It is sometimes difficult to know where to start.  This article gives some high-level guidance on the sort of information you may need to gather in order for Oracle Support to assist you with this task for the E-Business Suite.

Performance issues can potentially occur across any or several different areas of the Technology Stack, or may be restricted within Functional Code. For example (but not restricted to)

  • Architecture issue (e.g. high latency WAN, firewall)
  • Operating System problem or resource constraint
  • SQL or general RDBMS configuration issue
  • Database Deadlock
  • Apache Listener


Types of Performance Issues

"Simpler" Problems

These issues will hopefully be relatively straightforward to define and investigate.  For example: -
Single report consistently slow, SQL trace discovers one or more SQL statement(s) taking the majority of the elapsed time

More complex issues

These can take more work to be able to confidently define the real issue and often involve complex investigations involving different parties and much data gathering and analysis

For example :-
Any intermittent issue, System wide issues, Issues where SQL trace time represents only a small proportion of the elapsed time

Gathering Data for Performance Issues

It's helpful to follow a systematic process for investigating performance issues.  First steps include identify the nature, scope and extent of the problem. 

1. Identify the Extent of the Problem

Determine whether the extent of the problem is:

  • System Wide
  • Confined to one Technology Are. For example does it only happen in one of the Self Service, Forms or Reports areas
  • For a specific Product Area(s) For example, were it to only occur in HR, GL or iStore
  • Single Report/Form/Page
  • Which Instance or instances is the problem observed. Can it be reproduced in UAT/TEST instances

2. Narrow the Scope Further

Determine whether the problem occurs for:

  • All users
  • Only one geographic location
  • Only for users with certain Browser/type of PC
  • Only during peak periods

3. Qualify the Nature of the Problem

  • Is the problem intermittent or reproducible?
  • Is the problem due to slow performance or a process hanging or spinning?
  • When was the last time the process was completed without experiencing poor performance?  Document any changes since then.
  • Is there a workaround available?   For example, restarting Browser or restarting Apache.
  • What is the frequency of occurrence?

4.  Capture Additional Clues

  • Document and differentiate factual information from user perceptions. Example - users may say "it's always slow" but reality may be that it takes 10 seconds most of the time, but 40 seconds between 10am and 11am (peak load).
  • How long does it take for the process to complete when running slowly?  How long did it previously take (before having the performance issue)?  What is the expected performance for this process?
  • What is load on server(s) when issue occurs (CPU/Memory/Network)?  Is there any unusual activity when problems occur?  For example, memory used suddenly growing or CPU at 100%.
  • Is the problem reproducible with other browsers, such as Firefox, as well as with Internet Explorer?

5.  Identify likely causes and eliminate other possible causes

  • Are there any customizations?  If so, can they be eliminated for testing purposes?
  • If the problem only occured since patches have been applied or any configuration changes have been made, can these be reverted?
  • Do you have any Resource Limits enabled on the database that may be effecting the Applications users runtime or Concurrent Manager?
  • Does the number or frequency of certain Concurrent Requests correlate to performance issues occurring?
  • Have the database parameters been configured as per the current recommendations in Database Initialization Parameters and Configuration for Oracle Applications 11i (Metalink Note 216205.1)?
  • Does AWR or Statspack show anything unusual for Top waits, Top buffer gets, etc?
  • Have you searched Metalink generally, but specifically reviewed Recommended Performance Patches for Oracle E-Business Suite (Metalink Note 244040.1) for any known performance issues, tuning guidelines and/or patches?

Getting Help from Oracle Support

Once you understand the issue and reach the point where you need Oracle Support involved, it may be useful for you to review the "Performance Tuning" section of:

The first key decision you need to make is selecting a product code for your Service Request (SR): 

  1. If the issue is with an individual Form/Report/Page or only in one product area then log the SR for that particular product support team
  2. If issue seems to be with the Technology Area or System wide then log the SR with the AOL team (Oracle Application Object Library)

It is very important to provide a good problem definition (nature, scope, extent) so be as verbose as needed to give a good description of the issue.

It is also very important to reproduce the issue on a non-Production instance. If you are able to reproduce the problem outside of your Production instance, then any recomended patches or changes can be quickly assessed for their impact and any detailed debug or tracing required to identify the issue can be easily implemented.

A Quick Aside:  One issue per Service Request

You may need several issues investigated simultaneously and may be reluctant to raise different SRs or have different people investigating. Unfortunately, even similar-looking issues can have different root causes, which means they will need to go to different support teams or have different SR statuses. This is why it is important to ensure each issue is raised as a separate SR.

Performance Issues with Specific Forms, Reports, or Pages

For these "simpler" types of issues, we can generally track down the issue with the following information:

  • Form, Report or Page name, version and navigation path
  • Full versions of Forms, Reports or Framework as well as the iAS and Database versions, including rollup or interop patches applied
  • Relevant Family Packs or Maintenance Packs applied
  • Description of symptoms, what have you tried, your investigations, conclusions and/or thoughts/ideas
  • Does it reproduce constantly, in other environments as well?
  • When did you last run "Gather Schema Statistics" and at what percentage?
  • When did you last run any relevant purging processes?
  • SQL trace with binds and waits (raw file and TKPROF output)
  • Wall clock time elapsed from user perspective
  • What are the target and acceptable times for this process?

Performance Issues for Other Areas

These are the more complex issues and will normally require additional information (and patience) to resolve.  We will need the same information as listed above, but also:

  • Description of your System Architecture and network diagram
  • Technology Stack configuration files (Apache, Forms, Reports and Database, as relevant to the problem)
  • List of any Metalink notes or product documentation you have reviewed already, and what steps you implemented from this documentation
  • List of any patches applied specifically to try to address this issue
  • Details of profile options or configuration settings you have tried changing, explaining why you tried these settings and what effect they had (if any)
  • AWR or Statspack output for "good" and "bad" performing time periods
  • Details about your pinning, purging and gather schema statistics strategy
  • Relevant Log files
  • Debug and trace files

Good starting points for collating all this information are:

Additional Tips for Troubleshooting for Apache / JVM problems

Additional Tips for OA Framework-related issues

  • It is often useful to enable "STATEMENT" level logging (for ONE user only!) using the FND:Debug% profile options if you have a reproducible test case


Performance issues can sometimes be tricky to isolate, particularly those having more than one root cause.  This article has presented some ideas about the approach to take, in addition to the sort of information Oracle Support would likely be asking for if you need to log a Service Request.

If there is sufficient demand, I can write further articles in future, expanding on the topics introduced here.


Tuesday May 01, 2007

Java Caching for Oracle Applications 11i: Part 1

In this article I will try to explain some of the mysteries around the "Java Caching Framework," as this is a commonly misunderstood technology (which is probably due to it being well-hidden from the casual observers view).

This article will be an introduction to the topic, rather than trying to talk about all aspects.  I intend to write a follow up article to talk in more detail about common issues, configuration and troubleshooting for Java Caching, but let's start with the basics...

What is a Cache?

In general computing terms a cache is simply a copy of some data, sometimes prefetched or perhaps precalculated data, stored away from the original data.  The purpose of the cache is to provide faster access to data, as it will be stored either closer to the user and/or in a faster access medium.

There are a multitude of caches, some which have been around since computing began.  To name a few that immediately spring to mind:

What is eBiz Java Caching?

Data from the eBiz database is stored (cached) in the OACoreGroup Java Virtual Memory (JVM) memory on the eBiz Middle Tier server.  Normally this is done only when data is first accessed by the JVM, although the technology allows data to be preloaded if the Developer chooses.

It is also important to remember that every OACoreGroup JVM has its own local Java Cache, separate from the others.  We'll talk more about that later in this article.


Why Use Java Caching?

Java caching improves Application performance because data is retrieved from JVM memory.  This saves a network trip to the database, and in turn saves the time taken for a SQL command to extract data.

Characteristics of data that is most suitable to be cached:

  • Expensive SQL query
  • Does not change very often
  • Is frequently used

Challenges of Distributed Caching

Oracle E-Business uses the DISTRIBUTED cache model, where updates to objects in one JVM are propagated through to other JVMs using the same database.  If you have multiple eBiz middle tier servers and/or multiple OACoreGroup JVMs configured, there needs to be a mechanism to keep these separate JVM Java Caches synchronized.  A developer must specify which of the following two synchronization methods to use: 

  1. JVMs talk to each other and send the updated object to the other JVMs
  2. A JVM will just send an invalidation message for a specified object.  This is more commonly used.

An additional consideration is how to handle updates to data that happens outside of the JVMs, such as with updates via the Forms interface or a batch process.  For this case, an invalidation business event is triggered from the database, sending the JVMs an invalidation message for any records that have been updated.

If an object is invalidated in a JVM, it will be reread from the database when next accessed.  The new version will then be stored in the JVM's cache.


A Brief History of Java Caching

Java Caching has been around in the E-Business Suite for quite a long time.  It was not used by many Apps products until 11.5.10, as this was the first release where Database Invalidation was introduced.  I believe iStore was the first team to utilize Java Caching functionality in any depth.  Since 11.5.10 CU2, more and more product teams are using Java Caching.

Understanding How to Patch Java Caching

There are several different dependent layers to Java Caching, which is implemented as Java Classes stored on the Apps Web Tier:

Application Server packages (oracle.ias.cache)
eBiz leverages Java Caching mechanism implemented by the iAS Technology itself.   It is sometimes required to apply iAS Technology patches for Oracle9i Application Server  You should only apply patches that are released by the eBiz Applications Technology Group (ATG) team.

At the time of writing, the latest patch for this layer is:


JTF and FND packages (oracle.apps.jtf.cache and oracle.apps.fnd.cache)
Updates to these packages are shipped via normal Apps patches.  Originally released via JTT product patchsets, these patches are now rolled into FND patchsets such as ATG RUP4.   The eBiz ATG development team is responsible for this layer.

At the time of writing, two patches are recommended on top of ATG RUP3 and ATG RUP4

Functional layer

It is up to the individual Applications product teams to utilise the Java Caching Framework provided by ATG.  Product teams are responsible for creating and maintaining their own "buckets" (more properly called a "Component Cache") within the Java Cache, and to control the invalidation mechanism for their own objects.   

Updates to code on this layer are released via product-specific patches.   Unfortunately, a list of product-specific patches is beyond the scope of this article.

Troubleshooting Common Problems

Updates to data not being shown immediately

A commonly reported symptom is that changes to users' responsibilities are not taking effect immediately.  Customers report this problem after applying the ATG RUP3 patch.  The Applications Technology Group implemented Java Caching for responsibilities (amongst other things) in ATG RUP3, so this is often the first time customers see Java Caching in action (or perhaps I should say "inaction" in this case).

If you're encountering this behavior, the following Note should help:

Java Caching through a firewall
Java Caching uses a separate set of port numbers for communication between JVMs.  These ports need to be opened through a firewall for Java Caching to function.   These ports are defined by the following profile options:
These profile options are configured in AutoConfig via these CONTEXT.xml file variables:
  • s_fnd_cache_port_range
  • s_java_object_cache_port

Run the Java Cache Diagnostic Tests

If you are experiencing any symptoms that could be attributed to Java Cache problems, it is important to gather the information from the Diagnostic Tests.  For example, one test will prove if database invalidation is functioning correctly.

These tests were introduced in JTT.E (11.5.10) and can be accessed via the "CRM HTML Administration" responsibility:

  • Diagnostics--> CRM Foundation--> Caching Framework

Review the javacache.log

This log file is located in $COMMON_TOP/rgf/<Instance >_<Hostname> and provides error messages specific to Java Caching.


Java Caching is simple in concept but can be complex to troubleshoot.  This article has hopefully given you some insight into the fundamental concepts.  We'll delve deeper into some of these concepts in my next article.



Thursday Mar 01, 2007

Latest JVM Tuning Recommendations for Apps 11i

In my previous article "Configuring Middle-Tier JVMs for Applications 11i" I provided some suggestions in the paragraph "Rough Guidelines for JVMs" as to how many JVMs could be implemented as an initial starting point.  Subsequent to that blog article,  Guidelines to setup the JVM in Apps 11i"
(Metalink Note 362851.1) has been released which includes the ATG Performance Group's official recommendations, some of which are different than stated in my original article. 


In this article I would like to go into a little bit more depth about the newly published recommendations.  I will focus only on the OACoreGroup JVM and not consider the Forms Servlet, Discoverer or XML Service JVM settings.  I will also assume the reader has some familiarity with Java and eBiz 11i already and is also using eBusiness version 11.5.10 with JDK 1.4 or 1.5.   


From a CPU utilization perspective, there is no need to run more than 1 JVM per CPU because of the following reasons:

  • The JVM uses native threads and the native threads can be scheduled on any of the available cores/CPUs. 
  • Multiple parallel Garbage Collection (GC)  threads -- introduced in 1.4 -- are spawned.  Running multiple JVMs per CPU could result in an excessive number of background and GC threads. 
  • Multiple JVM increases the number of JDBC connections required and there is a memory and CPU overhead for each JVM process. 

For the majority of our web applications, more than 90% of the memory allocations are transient (for the life of the request or page), so new space GCs dominate much more than old generation GCs.  Old generation GCs are usually only a problem with older versions of Applications code or modules such as Configurator which load a large amount of objects (e.g. config model) in the old space.

Why increase the number of JVMs

There are several reasons why you would want to increase the number of JVMs:

  1. Minimize GC pause overhead
  2. Reduce the JVM heap size
  3. Minimize contention due to hot synchronized methods (i.e. monitor contention).
Based on Oracle's performance testing, benchmarking, and our experience with working directly with enterprise-class customers, we have determined that 100 users per OACore JVM has generally proven to be a practical and performant number.   Increasing the session.timeout parameter in above the 30 minute default may increase JVM memory requirements.

Recomendations in summary

Guideline #1:   For the OACoreGroup JVMs start with the lower of the following two values:

  1. Number of cores on the middle tier
  2. Peak Concurrent users / 100
For example: 

If you have 2 x Dual Core CPUs in your server and have 500 peak users, then 4 JVMs is the recommended starting point, since the number of cores is the lower number.  However, if you only had 300 peak users, then you would configure 3 JVMs as a starting point as this is now the lower figure.

Guideline #2:  Size your maximum heap size as either 512 MB or 1 GB.  If you start with 512 MB and find that more memory is required, increase to a maximum of 1 GB.   If more memory than 1 GB is required, then add another JVM instead (free physical memory allowing) to increase the total memory available overall. 

For example:

You are using 1 x JVM with 1 GB heap size and find you need to increase memory.  Configure your system for 2 JVMs, each with 750 MB heap size, thus providing 1.5 GB in total.

Your mileage will vary

As always, these recommendations are generalised and can only act as a suggested starting point for your system.  As mentioned in my previous blog article, there are many factors that effect how the JVM is utilized, so you will need to undertake representative testing to prove that the number of JVMs you select is suitable for your specific user load and profile.


Thursday Oct 19, 2006

Investigating java.lang.OutOfMemoryError with Apps 11i Middle Tier JVMs

If you had to guess what the error "java.lang.OutOfMemoryError" indicates when seen in your Apache log files, you would probably be right...

In most cases this message is telling you that the Java process has exhausted available memory and cannot process some requests successfully.   In many cases additional symptoms will be:

  • Poor web page performance
  • User requests being timed out
  • Java processes taking 100% CPU on your server.  
The Java Virtual Machine (JVM) will sometimes be restarted by oprocmgr as performance is so poor it presumes the JVM has died.

There can be other causes.  For example, this message can sometimes be given where there is the lack of some other resource such as threads per process (kernel parameter "max_thread_proc" on HP-UX) but this case should be easily identifiable from the exact message written out.

Where has all the memory gone?

Every object created in Java takes some memory.  Once all java code that references the java object has completed, then the Garbage Collector (GC) automatically removes the object and claims back the memory.

You can run out of memory due to:

  • Insufficient memory allocated to the JVM to cope with the amount of work
  • Suboptimal GC processing
  • A memory leak -- Java code is not releasing an object when it should
  • A Java code or operating system defect
How are JVMs configured with Apps 11i

Two important configuration files for Java in an Apps environment are:
    This file controls the JVM parameters in the "wrapper.bin.parameters=" entry.  When you install Apps originally or update the JDK version, the default settings are updated in your CONTEXT.xml file ("s_jvm_options" for the OACoreGroup JVM) and controlled by AutoConfig thereafter.

What can I do about OutOfMemoryError?

  • Are you using Oracle Configurator?
Configurator can sometimes require huge chunks of memory for a relatively long period of time, so Oracle now recommends setting up separate JVMs specifically for Configurator.  For more details see Setting Up a Dedicated JServ for Running Only Oracle Configurator" (Metalink Note 343677.1).

  • Increase available memory
An obvious choice, you may think, and this will often at least delay the effect of the issue, but may not always solve it.   You can achieve this by either increasing the number of JVMs or the memory settings ("-Xmx512M" for example).  In general, I would not recommend going much over 640MB per JVM, in order to keep the Garbage Collection time to a reasonable level.

  • Tuning GC

You can influence how Java performs Garbage Collection.   I would normally recomend reviewing and possibly backing out any non-default JVM parameters first, as these are sometimes just carried forward because they were used with previous JDK versions.  As always, any tuning should be undertaken in a TEST environment to ensure the changes have a positive impact.   

  • Using Concurrent Mark Sweep collector

With multi-CPU servers using JDK 1.4 and higher, the following parameters have been seen to improve performance and GC throughput on some environments:

-XX:+UseConcMarkSweepGC -XX:+UseParNewGC  

  • Identify the components that are using memory

Not always as easy as it sounds.  Generating class histograms or java profiling may be the only options. 

What data should I gather to investigate further ?

The initial investigation should focus on trying to identify the pattern of memory usage building up and how frequent GC is occurring.  The tools and information described below may help to achieve this

  • Review stdout files

Basic Garbage Collection information is normally written to the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm/*.stdout file for each JVM.  Additional Garbage Collection information can be logged, for example:

-XX:+PrintGCTimeStamps -XX:+PrintGCDetails

  • Add more verbose GC logging


This allows you to see how much of each memory pool is being used.  For example you may find it is the "Permanent Generation" space that is running out of memory, rather than the "Tenured Generation".  If this turns out to be the case, the "-XX:MaxPermSize" parameter could be increased to provide more memory.

  • Collect class histograms


Implemented in JDK 1.4 and higher by some vendors.  Can be very useful if available.  Beware of using this on HP-UX platforms running JDK 1.4 as it seems to crash the JVM rather than output the data!

  • Use JConsole

Discussed in a previous article: Using JConsole to monitor Apps 11i JVMs

  • Java Profiling


This should tell you exactly where the memory is being used, but has a huge impact on performance and is therefore not likely to be possible to gather this data, except for a lightly used test environment.

  • Other clues

Does the DMS output for the JVM show increasing active connections or threads?

Do JDBC connections to the database keep increasing? What module names do they relate to?

Does temporarily disabling "AM Pool" reduce memory consumption (Profile option "FND: Application Module Pool Enabled")?

Are you seeing PL/SQL "cursor leaks"?  See, "Understanding and Tuning the Shared Pool" (Metalink Note 62143.1) in section "Checking hash chain lengths" for SQL to check this.

  • Any Java Thread Deadlocks or Database Deadlocks?

Are Java processes spinning to 100% CPU or is CPU use very low when the error occurs?

  • Look for platform references

Review your operating system vendor's web site to look for known issues or required platform patches for your Java version.

My Production system is failing every few hours, what can I do right now?

A possible quick fix would be to increase memory and/or number of JVMs.  Be sure to ensure that this won't cause the operating system to start swapping!

You could also consider a scheduled bounce of Apache every 12 or 24 hours if possible. 

These steps may allow you to minimise the immediate impact on the business so you can investigate and implement changes in a methodical, safe, controlled and tested manner.

Where can I read more about Java memory and tuning?

There are many resources on the Internet to describe how Java uses memory and what steps to take for performance tuning.  The best place to start is your hardware vendors web site as the options available depend on their Java implementation and also the specific Java version you are using.  

For example:


Dealing with Java processes running out of memory can sometimes be as simple as increasing the memory available, or may require detailed investigation to identify the root cause.     

Although each case will have its own unique considerations, I hope that the information in this article has given you some ideas as to the general approach to take should the need arise.

Additional References

Tuesday Aug 01, 2006

Configuring Middle-Tier JVMs for Applications 11i

When you call Oracle Support with a problem like apj12 errors in your mod_jserv.log, middle-tier Java Virtual Machines (JVMs) crashing, or poor middle-tier performance, then it will often be suggested to increase the number of JVM processes.  So, the key question that likely occurs to you is, "How many JVMs are required for my system?"

Processing Java Traffic in Groups

First, some quick background:  web requests received by Oracle HTTP Server (Apache) to process Java code is sent to one of four different types of JVM groups to be processed.   You can see this in the jserv.conf file:

 ApJServGroup OACoreGroup 2 1 /usr/.../
 ApJServGroup DiscoGroup  1 1 /usr/.../
 ApJServGroup FormsGroup  1 1 /usr/.../
 ApJServGroup XmlSvcsGrp  1 1 /usr/.../

The number of JVMs for each group is signified by the first number on each line.
  • OACoreGroup is the default group.  This is where most Java requests will be serviced
  • DiscoGroup is only used for Discoverer 4i requests
  • FormsGroup is only used for Forms Servlet requests
  • XmlSvcsGrp is for XML Gateway, Web Services, and SOAP requests

In the example above, I have two JVMs configured for OACoreGroup and one JVM configured for each of the other groups.

Factors Affecting Number of JVMs Required

Determining how many JVMs to configure is a complex approximation, as many factors need to be taken into account.  These include:
  • Hardware specification and current utilization levels
  • Operating system patches and kernel settings
  • JDK version and tuning
  • Applications code versions, especially JDBC and oJSP
  • JServ configuration file tuning ( and
  • Applications modules being used
  • How many active users
  • User behaviour

Rough Guidelines for JVMs

Luckily, Oracle Development have undertaken various performance tests to establish some rules of thumb that can be used to configure the initial number of JVMs for your system.

    •    1 JVM per 100 active users
    • Use the capacity planning guide from Note 236124.1 "Oracle 9iAS Discoverer 4i: A Capacity Planning Guide"
    •    1 JVM per 50 active forms users
    •    1 JVM is generally sufficient

In addition to this, Oracle generally recommends no more than 2 JVMs per CPU.    You also need to confirm there are enough operating system resources (e.g. physical memory) to cope with any additional JVMs.

Your Mileage Will Vary

The general guidelines above are just that -- they're very broad estimates, and your mileage will vary.  As I write this, the Applications Technology Group is working on a JVM Sizing whitepaper that will provide guidelines based on whether your E-Business Suite deployment is small, medium, or large.  I'll profile this whitepaper here as soon as it's released publicly.

Until then, it's critical that you test your environment under load, using transactional tests that closely mirror what your users will be doing.  It's useful to use automated testing tools for this, as you create your benchmarks. 

Here are a couple of quick-and-dirty tools that might be useful in sizing your JVMs.

Script to determine "active users" for OACoreGroup

REM SQL to count number of Apps 11i users
REM Run as APPS user
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate   
and counter < limit_connects;

How to determine "active forms users" for FormsGroup

Check the number of f60webmx processes on the Middle Tier server.  For example:

ps -ef | grep f60webmx | wc -l


  • The number of required JVMs is extremely site-specific and can be complex to predict
  • Use the rules of thumb as a starting point, but benchmark your environment carefully to see if they're adequate
  • Proactively monitor your environment to determine the efficiency of the existing settings and reevaluate if required

More on Java Memory Tuning Later

Once you've established the right number of JVMs to use, it's then time to optimize them.  I'm intending to discuss Java memory tuning and OutOfMemory issues in a future article.  Stay tuned.




« June 2016