Friday Mar 23, 2012

ATG Live Webcast March 29: Diagnosing E-Business Suite JVM and Forms Performance Issues (Performance Series Part 4 of 4)

The next webcast in our popular EBS series on performance management is going to be a showstopper.  Dave Suri, Project Lead, Applications Performance and Gustavo Jimenez, Senior Development Manager will discuss some of the steps involved in triaging and diagnosing E-Business Suite systems related to JVM and Forms components.

Please join us for our next ATG Live Webcast on Mar. 29, 2012:

Triage and Diagnostics for E-Business Suite JVM and Forms

Debugging JVM logs

The topics covered in this webcast will be:

  • Overall Menu/Sections
    • Architecture
    • Patches/Certified browsers/jdk versions
    • JVM Tuning
    • JVM Tools (jstat,eclipse mat, ibm tda)
    • Forms Tools (strace/FRD)
    • Java Concurrent Program options location
    • Case studies
  • Case Studies
    • JVM Thread dump case for Oracle Advanced Product Catalog
    • Forms FRD trace relating to Saving an SR
    • Java Concurrent Program for BT

Date:               Thursday, March 29, 2012
Time:              8:00 AM - 9:00 AM Pacific Standard Time
Presenters:  Dave Suri, Project Lead, Applications Performance
                        Gustavo Jimenez, Senior Development Manager

Webcast Registration Link (Preregistration is optional but encouraged)

To hear the audio feed:
   Domestic Participant Dial-In Number:            877-697-8128
    International Participant Dial-In Number:      706-634-9568
    Additional International Dial-In Numbers Link:
    Dial-In Passcode:                                              99342

To see the presentation:
    The Direct Access Web Conference details are:
    Website URL: https://ouweb.webex.com
    Meeting Number:  597073984

If you miss the webcast, or you have missed any webcast, don't worry -- we'll post links to the recording as soon as it's available from Oracle University.  You can monitor this blog for pointers to the replay. And, you can find our archive of our past webcasts and training here.

If you have any questions or comments, feel free to email Bill Sawyer (Senior Manager, Applications Technology Curriculum) at BilldotSawyer-AT-Oracle-DOT-com. 

Tuesday Nov 24, 2009

Tuning the Oracle E-Business Suite Environment (OpenWorld 2009 Recap)

In addition to their benchmarking efforts, our Applications Performance Group works with some of our largest E-Business Suite customers in the world.  These customers have thousands of concurrent end-users, sprawling multi-terabyte databases, enormous clusters of application-tier servers, and massive RAC architectures.  The Applications Performance Group is called in whenever a customer experiences severe performance issues. 

Every year, our performance architects take all of their hard-won experience with these customers and distill it into a single OpenWorld presentation.  Wondering how many JVMs to use per CPU?  Curious about tuning the garbage collection parameters on your app tier?  Want some tips for optimizing your Concurrent manager setup?

Isam Alyousfi and Lester Gutierrez cover these topics, and much, much more, in their densely-packed presentation:

db_console_screenshot.png

It's difficult to convey the depth and range of the performance-tuning tips that they capture, but here's an outline of the topics that they cover:
  • Tuning the applications tier
    • Pointers to resources and performance-related patches
    • Tips for tuning and debugging Forms
    • OC4J and JVM sizing guidelines (e.g. number of JVMs per CPU, number of users per JVM)
    • Tools for tuning the application tier
    • Diagnostic framework for investigating response time / CPU issues
    • Garbage collection (GC) tuning tips
    • Common causes of OutOfMemoryError conditions
    • Symptoms of memory leaks
    • Using JDBC connection identification to map JDBC sessions to a JVM process
    • Performance implications of end-user training on web application resource consumption
    • Using the Pool Monitor to check Framework applications and JVM utilization
    • Using EM Monitoring and Application Diagnostics for Java (AD4J) to monitor JVMs on different hosts
    • Using JConsole to analyze heap dump issues
    • Performance-related patches to consider
  • Tuning the Concurrent Manager
    • General tips for maximizing job throughput
    • Tips for workload management, including the number of target processes per CPU
    • Tips for Transaction Managers (TMs)
    • Tips for tracing and speeding up Concurrent Reports
    • Using Parallel Concurrent Processing (PCP) in RAC deployments
  • Tuning the client tier and network
    • Pointers to a new whitepaper, and tips for optimizing desktop client configurations
    • Tips for reducing browser memory footprints
    • Guidance on network latency and co-locating database and application tier servers
    • Network profiling and packet analysis techniques
    • Use of caching tools to reduce network traffic between client and middle-tiers
  • Tuning the database tier
    • init.ora recommended settings
    • Consequences of undersized buffer caches or shared pools
    • I/O optimization techniques
    • Recommended performance features
    • Using Automatic Workload Repository (AWR) data and baselines
    • Linking AWR with ASH data
    • Tips for using DB Console
    • Techniques for gathering statistics
    • Recommendations for OATM Tablespace Model conversions
    • SQL issue triage notes, and common problems
    • Benefits of upgrading to 11g Database
    • Notable performance-related features in the 11g Database
    • Advanced Compression performance benchmarks
    • 11g Automatic SQL monitoring and tuning
    • SQL tracing enhancements in 11g DB
    • 11g Optimizer improvements
  • Tuning the applications
    • Pointers to application module tuning resources
    • Recommended performance-related patches and configuration guidance for specific products: Create Accounting XLA, Accounts Receivables AR, Accounts Payables AP, iPayment IBY, Incentive Compensation CN, Time & Labor, Workflow, Order Management OM, Payroll
    • Logging, purging, and archiving
    • Accessing the Purge Portal
  • Upgrade performance tips
This presentation is chock full of tips, pointers, and hard-won knowledge. It represents the distillation of countless performance-related Service Requests and customer escalations. If you're grappling with performance issues in your environment, or simply trying to squeeze more performance out of existing hardware, I'd strongly recommend downloading this presentation.

Listening to the Session

If you registered for OpenWorld, here's a link to the OpenWorld On Demand page where you can download the presentation or listen to the live recording of this session.

Related Articles

Saturday Mar 29, 2008

Two Essential Tools for Diagnosing E-Business Suite Network Issues

Network problems seem to be on the rise again, either due to the increase in service demands, integration of different technologies such as Voice Over IP (VoIP), and the inevitable increases in uptake of technologies at new locations. Not only do network administrators have to concern themselves with various WAN optimization techniques, but they also have to deal with ad-hoc problems as they occur. As we all know too well, intermittent problems are the most difficult and resource intensive to address.

Network Diagnostic Tools for Oracle E-Business Suite

There are two diagnostic tools available in the Oracle E-Business Suite:
  • The Network test Form - available in all versions
  • The Client Analyzer - introduced in 11.5.10
Commonly, many people have found it difficult to understand the output of these tools and how they can be used to troubleshoot network problems. The Oracle E-Business Suite Network Utilities white paper explains:
  • How these two utilities work - they actually test a "form round trip"
  • How to understand the figures and what they tell you 
  • How end-users can help narrow down the source of problems and even identify which part of the system is not performing
Getting Into the Details

This last point needs a little more explanation, as this previously underused form now becomes an additional tool in a holistic approach to system performance troubleshooting. The techniques in the paper show the following:
  • How end users can gather diagnostic information saving the time and expense of deploying specialized personnel
  • How to compare the E-Business Suite measurements with standard network utilities such as ping
  • How to interpret the information to isolate a problem to the network, middle tier, or other system component.
The techniques do not require specialized skills and therefore much progress can be made towards identifying the root cause of the problem by junior system administration staff and suitably experienced end-users. Understanding the purpose of the tools and how to interpret their output enables end-users to collect the requisite diagnostic information and perform rudimentary diagnosis when performance problems occur. Inevitably, this means that this reduces the load on the support infrastructure as well as freeing up having to wait for hours for a problem to occur.

In addition to showing example usage of the tools, the paper presents sample output for users to compare with their own measurements, and thereby draw conclusions more quickly and effectively.

The chart below shows an example of how the Network Test form measurement maps to the network ping time.

Form Measured Latency Chart: Chart showing simulated Forms Measured Latency vs Actual Network Latency

By comparing the Network Test form measurement with the network ping time, you can identify:
  • If there is a problem in the network
  • If there is a problem in the middle tier
  • If there is a problem in the network and middle tier
  • Or Whether there is a problem in another system component
A couple of important points ......
  • Understand the limitations of these tools
  • Know which figures to use and how much they can be expected to fluctuate in a given scenario
  • If ping (ICMP) traffic is blocked for security reasons, the paper describes two Linux utilities that could be used to measure the network ping time
References
Related Articles

`

Friday Feb 01, 2008

Diagnosing Sun Java Plug-in Issues with Oracle Apps

 

As discussed in previous articles, the native Sun Java Runtime Engine (JRE) is certified for Windows-based desktop clients accessing E-Business Suite Releases 11i and 12.   The native Sun JRE is used to run Forms-based content in the E-Business Suite.  It is the only plug-in certified for Release 12, and can be used in place of the older Oracle JInitiator 1.3 plug-in certified with Release 11i.


mzFormsArch:


This article spotlights Metalink notes which guide you through the tracing options and some gotchas when using the native Sun JRE with the Oracle E-Business Suite.

Enabling JRE Tracing and Logging

There are several notes available which explain how to enable these options for the JRE:-



I like this note the best as it just describes the simplest way to enable logging, tracing and java console (OK, yes I admit I wrote it...)



Both of these notes have a "JRE Tracing options" section describing the steps, however the method described to enable the logging is the more long-winded method (in my view).



This note describes all the different methods to enable logging, tracing and the Java Console and has lots of pictures. The slight drawback is that it doesn't recommend a particular method.

Potential "Gotchas"

  • Rnning Multiple JRE Versions on the Same Desktop Client

This issue is fully described in the "Upgrading Sun J2SE" notes referenced above as well as in this article, but is worth spotlighting here.  If your organisation uses runs Java-based applications that depend on different JRE releases, you will need to consider static versioning issues when deploying the native JRE required by the E-Business Suite.

  • Potential Character Set Issues

 


There are different dependencies with Native JRE compared to Jinitiator, as the following note highlights.  In this case one Client PC using Japanese language did not have the necessary Language code pages installed.   This Note could apply to other languages also:

  • Potential Issues When Connecting via VPN

This issue applies equally to JInitator, but thought I would mention the following Note as it seems to be becoming more popular for customers to run
the E-Business Suite over a secure VPN connection:


Related Articles

Friday Dec 28, 2007

Essential Debugging Tools For Apps 12

E-Business Suite Release 12 includes Oracle Application Server 10g components for Forms and Java, each hosted in their own ORACLE_HOMEs.  A couple of colleagues in my team have published a pair of complementary Metalink Notes about debugging, diagnostic, and tracing tools in E-Business Suite Release 12.  It's high time that I got around to profiling those notes here.

Release 12 Techstack Overview: Overview of three-tier logical architecture for E-Business Suite Release 12, including the database, application server, and desktop tiers. 

Debugging Java Components in Release 12

The first Note highlights a series of Java-related problem scenarios that you might encounter with Release 12:
This useful document highlights possible symptoms, log files, useful scripts and other tools that can be used to change logging levels to track down issues with:
  • Oracle Process Manager and Notification Server (OPMN)
  • Oracle HTTP Server (Apache)
  • Oracle Containers for J2EE (OC4J)
  • Configured Application Modules, including:
    • OACORE - Core Application Modules
    • OAFM - Oracle Transport Agent, MapViewer
    • FORMS - Forms (using Servlet Mode)
Some common problem scenarios highlighted:
  • OPMN errors at startup
  • HTTP Server fails at startup
  • OC4J Applications Modules (e.g. oacore, forms, oafm) fail at startup
  • HTML requests complete but Java requests fail
  • HTML and Java requests complete but Applications Login page fails
  • Applications Login page is displayed but login fails
It also provides pointers to ways of monitoring or debugging the following components:
  • OPMN
  • Java Object Cache Monitor
  • Applications Server Forms Servlets
  • Class Loaders
The Note wraps up with useful summaries of all of the relevant log files used by OPMN, Oracle HTTP Server (OHS), and the various J2EE Applications modules.

Debugging Forms Components in Release 12

The second Note covers the Forms part of that territory:
This document highlights techniques for using Forms Trace as part of your diagnostic toolkit.  Forms Trace offers detailed data collection and other features to help drill down into Forms runtime problems.  The Note covers:
  • Activating Forms Trace via Application profiles
  • Defining Trace Groups to isolate problems more efficiently
  • Tips on interpreting Trace Files
  • Tips on using the legacy Forms Runtime Diagnostics (FRD) utility
Both of these documents are useful additions to any R12 sysadmin's toolkit.  If you have feedback or questions about these documents, feel free to post your thoughts here.  If we're lucky, you might get a chance to interact directly with the Notes' authors.

Related Articles

Tuesday Dec 11, 2007

Performance Tuning for the E-Business Suite

It's said that death and taxes are the only certainties.  I beg to differ.  There's at least one other for IT professionals:  regardless of the amount of network bandwidth you've dedicated to E-Business Suite traffic, no matter how many application servers you have in your load-balancing pool, no matter the turbocharged fire-breathing database server you've got in your data center, it's inevitable that your users will complain about performance. 

Apps DB Console Screenshot: Screenshot of E-Business Suite Database Console showing Host queues,Waiting and Working Sessions, Instance Throughput, and other performance metrics

In my promised but much-belated recap of OpenWorld 2007 highlights for Apps DBAs, here's a presentation from Isam Alyousfi and Lester Gutierrez that you much check out:
Isam and Lester are key members of our Oracle Applications Performance Group -- the same group that publishes all of the official Oracle Apps benchmarks, white papers, and performance metrics.

In this sprawling presentation, they cover the following topics:
  •  E-Business Suite Architecture and techstack from a performance perspective
  • Tuning the Applications Tier, including:
    • Forms & Reports tracing and tuning
    • Concurrent Manager tracing and tuning
    • PL/SQL Profiler usage and tips
    • Apache JServ/OC4J optimization and configuration tips
    • How to debug Java memory and connection leaks
    • OA Framework tracing
    • AOL/J Connection Pool monitoring
    • Discoverer tuning
  • Tuning the Database Tier, including:
    • I/O considerations
    • Automatic Workload Repository (AWR) data interpretation
    • Use of the DB Console
    • How to gather and interpret statistics
    • Use of the Applications Tablespace Migration Utility
    • 10g DB upgrades from a performance perspective
  • Client and Network Tuning, including:
    • Memory usage and profiles for Apps 11i and R12
    • Network performance and bandwidth considerations
  • Application Tuning, including specific tips for:
    • Workflow
    • Order Management
    • Payroll
    • Financials
    • TCA/DQM
    • Logging
    • Profile usage
    • Purging and archiving
  • And much, much more
This presentation is chock full of tips, pointers, and hard-won knowledge.  It represents the distillation of countless performance-related Service Requests and customer escalations.  If you're grappling with performance issues in your environment, or simply trying to squeeze more performance out of existing hardware, I'd strongly recommend downloading this presentation.

Related

Thursday Sep 20, 2007

Java Caching for Oracle Applications 11i: Part 2

In my previous article "Java Caching for Oracle Applications 11i: Part 1" I talked about the basics of Java Caching and how it works conceptually.   


mzJavaD1:


My original intention with "Part 2" was to discuss how to diagnose issues with Java Caching, but I got a bit carried away with scripts and the like so ended up creating some Metalink notes instead. 

Diagnosing Database Invalidation Issues

Here's the first one:




As it says on the tin, this note covers diagnosing issues with Database Invalidation.   The classic symptoms being that when Responsibilities are added to a user, they are not appearing immediately (but do after Apache is bounced).


Although the scripts are listed in the note, you can also download a soft copy via the link mentioned in the note.

Diagnosing Issues with Responsibility Assignments

This issue is not actually a Java Caching problem, but the symptoms initially look similar to those described above.  The key difference here is that the affected responsibilities still do not appear after Apache is bounced.  For details, see:




More on NoClassDefFoundErrors

This one seems to be a bit of a "phantom illness" as I have seen two or three customers hit this issue, but after following the Action Plan in the note the problem is not reproducible anymore.   Even more bizarrely, after reverting the changes introduced by the Action Plan, the problem doesn't come back!    This one has piqued my interest, so if anyone out there has the same symptoms and can still reproduce the problem after reading the following note, then I would be happy for you to email me your SR number.


Hopefully you will find these notes useful towards understanding and diagnosing any Java Cache issues you find yourself facing.

Related

Understanding JDBC Connections From the eBusiness Middle Tier

 

In this article I will describe the basics of configuring and monitoring JDBC connections between the eBusiness Web Tier and the Database, then cover some common issues to help identify root causes for both Release 11i and Release 12 of eBusiness Suite.


11i Architecture:


Brief overview


In general, whenever a functional page requires data from the database, it makes a call to a lower code level (the Java layer in the Application Object Library, also known as AOL/J) which handles the database connectivity.  The AOL/J code provides a JDBC connections to the database through the Database Connection Pool.


You should note that the Java Connection Pool mechanism for eBiz is completely provided by eBiz code and does not use the mechanisms provided to pool connections through Java language directly, nor through the Application Server configuration.


Configuring JDBC connection pooling


The JDBC connection pool is created using the settings in the DBC file. The name, location and contents of this DBC file is controlled through AutoConfig. To modify any of the JDBC connection pool parameters, you should therefore use the techniques described in the following document to ensure changes are maintained in a supportable and consistent way:



The "s_dbc_file_name" variable in the CONTEXT.xml file provides the dbc filename and is located in the $FND_TOP/admin/< INSTANCE>_<HOST> directory.


JDBC connection pool parameters and their use are covered in the following documents:



When considering any changes, you should also take into account that every JVM has its own JDBC connection pool. For example, if you have one Web Node with three OACoreGroup JVMs plus one XmlSvcsGrp JVM configured, then you will have a total of four JDBC connection pools with connections to your eBiz database


Monitoring the JDBC connection pool


It is always a good idea to understand how your environment looks when things are going well, to give you a baseline to compare against if you need to investigate any issues. 


You will most certainly need to review the JDBC connection data if you are experiencing issues.



Monitoring JDBC Connections through Oracle Applications Manager (OAM)

Login to OAM directly or via the "System Administration" responsibility.


  1. Select the "JServ Usage" under the Monitoring section in OAM
  2. Click the "Expand all" link to list the Servers and all the JServ processes for OACoregroup.  This shows memory usage, connections (including "Potentially Leaked") and Application Module information.  You can click the "Add to support cart" to capture this page if Oracle Support are assisting your investigation.
  3. If there are any non zero values for "Potentially Leaked" then click on this number to drill down into the details
  4. Select "Yes" for the "Filter by Potentially Leaked" option and click "Go" button
  5. Click "Show all details" to display the Java Thread dump for all the potentially leaks Java connections

mz_jdbc_oam:


The "old" way of gathering this data was to use the URL http://host.domain:port/OA_HTML/jsp/fnd/AoljDbcPoolStatus.jsp but this will only give data for the one JVM you happen to connect to, so may not be so useful in multi-JVM environments.


Run SQL scripts to monitor database connections


Using SQL scripts will not give so much information as OAM, but can be used to provide useful summary information on a periodic basis. For example you may wish to include the information from the SQL below as part of your baseline data:


REM
REM START OF SQL
REM
REM Connections by machine and instance
select s.machine, s.username, s.module, s.inst_id, count(*) how_many
from (select distinct PROGRAM, PADDR, machine, username, module, inst_id from gV$SESSION) s,
 gv$process p
where s.paddr = p.addr
and p.inst_id = s.inst_id
group by s.machine,s.username, s.module, s.inst_id
/
REM
REM END OF SQL
REM


NOTE - when looking at V$SESSION, the STATUS of JDBC connections tend to show as INACTIVE, this is normal and does not indicate a specific problem


Where Could It Go Wrong?


Issues with the JDBC connection pool tend to be of a nature whereby the number of database connections increase over time and do not seem to be released again. If this continues unchecked, then you may find the database will run out of sessions/processes and/or the Middle Tier JVM will run out of connections or memory.


A "quick fix" would normally be to restart Apache, but the underlying cause would need to be investigated further.


Issues can potentially occur at five different levels:



  1. Core Java Code
  2. AOL/J JDBC Code
  3. OA Framework
  4. Functional Code
  5. External Influences

I'll discuss each of these areas next.



Core Java code


Although eBiz does not use native Java connection pooling methods, we do rely on the underlying Java APIs generally.   Any issues at this level will generally require the latest Java version to be installed


If you need to upgrade your Java version, see:



AOL/J JDBC code


As this is the code that handles JDBC connection, it is often the first area to be blamed, but the JDBC connection pool can only drop database connections where the calling application has released the JDBC connection it holds in the pool, so it often turns out to be an issue higher up the code stack.


Number of JDBC connections increase after ATG Rup 5 because jdbc parameters are lower case (Metalink Note 459072.1) describes a known issue with Apps 11i.


It is prudent to be on the latest JDBC driver patch, but should have at least applied one of the following patches:-



The latest JDBC patch can be found in:



You should note that the JDBC driver version has no relation to the Database version, as it is installed on the eBiz Middle Tier.    For example, the latest version of JDBC drivers provided by patch 4899697 (9.2.0.8) is the same patch for all RDBMS versions.


OA Framework (OAF)


OA Framework calls AOL/J when it needs a database connection, and it is up to OAF to release any such connection when it has finished with it. There is an added complexity, in that OAF also has its own pooling mechanism for the OAF Pages, which is the "Application Module pool" (AM pool).  This means that although a user may have finished with a page, the page and its associated database connection are retained for a period of time.


The AM pool is controlled by profile options, which are described in :-



Issues at this code level would tend to be either:



1. Issue with AM Pooling


You can test the effect of disabling AM pooling by setting the profile option "FND: Application Module Pool Enabled" to "No".  Use this technique with caution if considering this for a production environment.


2. Specific bug where database connection is not released.


This would generally require patching.


Functional Code


Issues at this code level would tend to be a specific bug where a connection is not released.


External influences


Firewall timeouts are known to potentially cause an issue for JDBC connections.  For details, see:



Although this note is for Apps 11i, the technical issue can also apply to Release 12 as well


Configuring eBiz to minimize JDBC connections


If investigating issues with JDBC connections increasing, it may be useful to minimise the database connections as much as possible by de-tuning the JDBC pool. This may reduce end user performance, so should be used with caution if considering this for a production environment.


To do so, you need to do both these steps:



1. Disable Application Module (AM) Pooling


This is necessary as the AM objects hold a JDBC connection whilst they are alive, even if they are not currently used.


Set the profile option "FND: Application Module Pool Enabled" (AMPOOL_ENABLED) at SITE level to a value of "No"

2. Set the JDBC connection pool parameters to release connections:


    FND_JDBC_BUFFER_DECAY_INTERVAL=180
    FND_JDBC_BUFFER_MIN=0
    FND_JDBC_BUFFER_MAX=0
    FND_JDBC_BUFFER_DECAY_SIZE=50


Conclusion


Identifying issues with JDBC connections can sometimes be a frustrating process, as the investigations may need to consider multiple failure points and complex architectures. I hope this article has given you a better understanding of JDBC Pooling and where to start looking for issues.


References



Related


Wednesday Sep 05, 2007

Troubleshooting DMZ Setups for Apps

It's possible to expose selected Oracle E-Business Suite applications such as iStore or iRecruitment to users outside of your corporate intranet.  As part of our security best practices recommendations, we recommend the use of reverse proxies in demilitarized zones (DMZ) for these types of deployments.

DMZ Reverse Proxy:

While simple in concept, the actual execution is sometimes a little trickier.  These projects are often complicated by the separation between different groups that manage network operations, enterprise security, and the E-Business Suite environments themselves.  Coordinating all three organizational groups can be a project in itself.  Even small missteps can result in some of the following issues:
  • Misconfigured firewalls and other networking components
  • Incorrectly configured reverse proxies
  • Incomplete or incorrect E-Business Suite setups
  • Inconsistencies between testbeds and production setups
One Step at a Time

Debugging environments with lots of complex moving parts can be frustrating.  The best strategy is to take a systematic approach and test the critical components in sequence.  To help you with that, our hardworking Oracle Support team has assembled some of the best tips for debugging these types of configurations here:
They've also published a companion document with a crisp walkthrough:
These documents are written specifically with Release 11i in mind but the principles and techniques apply equally to Release 12, too.  Great stuff and highly recommended if you're working on implementing a DMZ in your Apps environment.

Related

Wednesday Aug 15, 2007

Latest Support Diagnostics for Apps 11i

Our hard-working Service Engineering team has just released more updates to their Support Diagnostics Patch for E-Business Suite Release 11i, hard on the heels of their recent updates to the Release 12 version of those tests.

Support Diagnostics Screenshot: Screenshot of sample diagnostic report output for Financials - Receivables

This patch contains new support diagnostic tools along with critical updates to existing tests.  The patch is available for download on Metalink > Patches & Updates.  It's also directly accessible from:
A description of all tests included in the Support Diagnostics patch can be found in:
For a quick summary of Apps products with new an updated tests, see:

Thursday Aug 02, 2007

Latest Support Diagnostics for Apps Now Available

[Read More]

Tuesday Jun 05, 2007

Comparing Bandwidth Requirements between Release 11i and 12

[June 8, 2007 Update:  Clarified background on the second set of benchmarks]

A few questions have been raised in our new Release 12 Upgrade Forum about the differences in network bandwidth requirements between Release 11i and 12.


The usual disclaimers apply:  benchmark tests are conducted by the Applications Performance Group on reference E-Business Suite environments.  These environments and the selected transactions will not necessarily match up with your own transaction mix, so your mileage will vary.  If you're in need of precise benchmarks, it's always best if you perform your own tests with your own mix of transactional data.

Differences Between JInitiator and the native Sun JRE Plug-in

Release 11i currently requires Jinitiator to display Oracle Forms-based content.  Release 12 requires the native Sun Java Runtime Engine (JRE) to display Forms-based content. 

Our Applications Performance Group has published a full whitepaper comparing bandwidth requirements between these two configurations:
Differences in Page Sizes

As for our web-based applications, many of the products changed their pages and flows in Release 12.  There are actually two comparison points: 
  1. 11.5.10.CU2 vs. 11.5.10.CU2 with on the Release 12 ATG Rollup
  2. 11.5.10.CU2 vs. Release 12
Here's the first comparison of page loads:

Comparison of page load sizes - Part 1: Comparison of page load sizes between Release 11i 11.5.10.CU2 vs. 11.5.10.CU2 with the Release 12 ATG Rollup applied

[June 8, 2007 Update:  The section above is generating a lot of similar questions, perhaps understandably.  Here's the scoop:  the comparison above shows the additional overhead associated with the new R12 technology stack running the same Applications code.  This
would be the equivalent of running the 11i versions of these screens in Release 12.  Our Applications Performance team is very meticulous about comparing equivalent things.  No "apples to oranges" comparisons for them.


They deemed it important to distinguish between added network bandwidth overhead in Release 12 due to new code at the Apps layer vs. new R12 technology stack requirements.

So, their baseline for network bandwidth was the Release 11i.5.10.CU2 environment.  They then ran benchmark tests with the same Applications code and the new Release 12 ATG components.  Important note:  this configuration isn't documented, supported, or recommended -- it was strictly a Development-only configuration used for benchmarking purposes.  You cannot apply the Release 12 techstack to an E-Business Suite Release 11i environment, and we don't provide any patches to do so.]

Here's a thumbnail of the second comparison (click on it to see the full-size version, which is admittedly low resolution):

Thumbnail - Full page load comparison: Thumbnail of table comparing Release 11i and 12 page load sizes

The table above compares a Release 11i environment with a full E-Business Suite Release 12 environment.  The Release 12 environment includes not only the ATG changes but also all of the Apps code changes.  You'll see that the page sizes did increase, but these increases were mainly due to Apps code changes and user interface changes.  The new OA Framework (Swan) changes accounted for less than 4%.

New Functionality Trumps All

As Apps technologists, it's tempting for us to get caught up in these types of discussions.  "Look -- the login page is X bytes larger!" 

It's always worth remembering that your end-users don't really care much about such things, however.  They're more likely to remark on the fact that the login page now has two new capabilities:  reminding them of their password, and reminding them of their userid, too. 

It's been my experience that new functionality trumps all, at least in the eyes of your users.  My recommendation:  go ahead and consider these benchmarks as part of your Release 12 evaluations, but more meaningful comparisons will come from spending some quality time with your end-users in assessing and prioritizing the new functional benefits in R12.

Related

Friday May 18, 2007

Debugging General Performance Issues with Oracle Apps

Identifying performance bottlenecks can sometimes be a black art with any distributed computing system.  It is sometimes difficult to know where to start.  This article gives some high-level guidance on the sort of information you may need to gather in order for Oracle Support to assist you with this task for the E-Business Suite.

Performance issues can potentially occur across any or several different areas of the Technology Stack, or may be restricted within Functional Code. For example (but not restricted to)



  • Architecture issue (e.g. high latency WAN, firewall)
  • Operating System problem or resource constraint
  • SQL or general RDBMS configuration issue
  • Database Deadlock
  • Apache Listener


mzPerfIssues:


Types of Performance Issues



"Simpler" Problems

These issues will hopefully be relatively straightforward to define and investigate.  For example: -
Single report consistently slow, SQL trace discovers one or more SQL statement(s) taking the majority of the elapsed time


More complex issues

These can take more work to be able to confidently define the real issue and often involve complex investigations involving different parties and much data gathering and analysis


For example :-
Any intermittent issue, System wide issues, Issues where SQL trace time represents only a small proportion of the elapsed time


Gathering Data for Performance Issues


It's helpful to follow a systematic process for investigating performance issues.  First steps include identify the nature, scope and extent of the problem. 


1. Identify the Extent of the Problem

Determine whether the extent of the problem is:



  • System Wide
  • Confined to one Technology Are. For example does it only happen in one of the Self Service, Forms or Reports areas
  • For a specific Product Area(s) For example, were it to only occur in HR, GL or iStore
  • Single Report/Form/Page
  • Which Instance or instances is the problem observed. Can it be reproduced in UAT/TEST instances

2. Narrow the Scope Further

Determine whether the problem occurs for:



  • All users
  • Only one geographic location
  • Only for users with certain Browser/type of PC
  • Only during peak periods

3. Qualify the Nature of the Problem


  • Is the problem intermittent or reproducible?
  • Is the problem due to slow performance or a process hanging or spinning?
  • When was the last time the process was completed without experiencing poor performance?  Document any changes since then.
  • Is there a workaround available?   For example, restarting Browser or restarting Apache.
  • What is the frequency of occurrence?

4.  Capture Additional Clues


  • Document and differentiate factual information from user perceptions. Example - users may say "it's always slow" but reality may be that it takes 10 seconds most of the time, but 40 seconds between 10am and 11am (peak load).
  • How long does it take for the process to complete when running slowly?  How long did it previously take (before having the performance issue)?  What is the expected performance for this process?
  • What is load on server(s) when issue occurs (CPU/Memory/Network)?  Is there any unusual activity when problems occur?  For example, memory used suddenly growing or CPU at 100%.
  • Is the problem reproducible with other browsers, such as Firefox, as well as with Internet Explorer?

5.  Identify likely causes and eliminate other possible causes



  • Are there any customizations?  If so, can they be eliminated for testing purposes?
  • If the problem only occured since patches have been applied or any configuration changes have been made, can these be reverted?
  • Do you have any Resource Limits enabled on the database that may be effecting the Applications users runtime or Concurrent Manager?
  • Does the number or frequency of certain Concurrent Requests correlate to performance issues occurring?
  • Have the database parameters been configured as per the current recommendations in Database Initialization Parameters and Configuration for Oracle Applications 11i (Metalink Note 216205.1)?
  • Does AWR or Statspack show anything unusual for Top waits, Top buffer gets, etc?
  • Have you searched Metalink generally, but specifically reviewed Recommended Performance Patches for Oracle E-Business Suite (Metalink Note 244040.1) for any known performance issues, tuning guidelines and/or patches?

Getting Help from Oracle Support


Once you understand the issue and reach the point where you need Oracle Support involved, it may be useful for you to review the "Performance Tuning" section of:



The first key decision you need to make is selecting a product code for your Service Request (SR): 



  1. If the issue is with an individual Form/Report/Page or only in one product area then log the SR for that particular product support team
  2. If issue seems to be with the Technology Area or System wide then log the SR with the AOL team (Oracle Application Object Library)

It is very important to provide a good problem definition (nature, scope, extent) so be as verbose as needed to give a good description of the issue.


It is also very important to reproduce the issue on a non-Production instance. If you are able to reproduce the problem outside of your Production instance, then any recomended patches or changes can be quickly assessed for their impact and any detailed debug or tracing required to identify the issue can be easily implemented.



A Quick Aside:  One issue per Service Request


You may need several issues investigated simultaneously and may be reluctant to raise different SRs or have different people investigating. Unfortunately, even similar-looking issues can have different root causes, which means they will need to go to different support teams or have different SR statuses. This is why it is important to ensure each issue is raised as a separate SR.


Performance Issues with Specific Forms, Reports, or Pages


For these "simpler" types of issues, we can generally track down the issue with the following information:



  • Form, Report or Page name, version and navigation path
  • Full versions of Forms, Reports or Framework as well as the iAS and Database versions, including rollup or interop patches applied
  • Relevant Family Packs or Maintenance Packs applied
  • Description of symptoms, what have you tried, your investigations, conclusions and/or thoughts/ideas
  • Does it reproduce constantly, in other environments as well?
  • When did you last run "Gather Schema Statistics" and at what percentage?
  • When did you last run any relevant purging processes?
  • SQL trace with binds and waits (raw file and TKPROF output)
  • Wall clock time elapsed from user perspective
  • What are the target and acceptable times for this process?

Performance Issues for Other Areas


These are the more complex issues and will normally require additional information (and patience) to resolve.  We will need the same information as listed above, but also:



  • Description of your System Architecture and network diagram
  • Technology Stack configuration files (Apache, Forms, Reports and Database, as relevant to the problem)
  • List of any Metalink notes or product documentation you have reviewed already, and what steps you implemented from this documentation
  • List of any patches applied specifically to try to address this issue
  • Details of profile options or configuration settings you have tried changing, explaining why you tried these settings and what effect they had (if any)
  • AWR or Statspack output for "good" and "bad" performing time periods
  • Details about your pinning, purging and gather schema statistics strategy
  • Relevant Log files
  • Debug and trace files

Good starting points for collating all this information are:



Additional Tips for Troubleshooting for Apache / JVM problems



Additional Tips for OA Framework-related issues



  • It is often useful to enable "STATEMENT" level logging (for ONE user only!) using the FND:Debug% profile options if you have a reproducible test case

Conclusion


Performance issues can sometimes be tricky to isolate, particularly those having more than one root cause.  This article has presented some ideas about the approach to take, in addition to the sort of information Oracle Support would likely be asking for if you need to log a Service Request.


If there is sufficient demand, I can write further articles in future, expanding on the topics introduced here.


Related


Tuesday May 01, 2007

Java Caching for Oracle Applications 11i: Part 1

In this article I will try to explain some of the mysteries around the "Java Caching Framework," as this is a commonly misunderstood technology (which is probably due to it being well-hidden from the casual observers view).


This article will be an introduction to the topic, rather than trying to talk about all aspects.  I intend to write a follow up article to talk in more detail about common issues, configuration and troubleshooting for Java Caching, but let's start with the basics...


What is a Cache?


In general computing terms a cache is simply a copy of some data, sometimes prefetched or perhaps precalculated data, stored away from the original data.  The purpose of the cache is to provide faster access to data, as it will be stored either closer to the user and/or in a faster access medium.

There are a multitude of caches, some which have been around since computing began.  To name a few that immediately spring to mind:


What is eBiz Java Caching?


Data from the eBiz database is stored (cached) in the OACoreGroup Java Virtual Memory (JVM) memory on the eBiz Middle Tier server.  Normally this is done only when data is first accessed by the JVM, although the technology allows data to be preloaded if the Developer chooses.


It is also important to remember that every OACoreGroup JVM has its own local Java Cache, separate from the others.  We'll talk more about that later in this article.




mzJavaD1:



Why Use Java Caching?

Java caching improves Application performance because data is retrieved from JVM memory.  This saves a network trip to the database, and in turn saves the time taken for a SQL command to extract data.

Characteristics of data that is most suitable to be cached:


  • Expensive SQL query
  • Does not change very often
  • Is frequently used

Challenges of Distributed Caching


Oracle E-Business uses the DISTRIBUTED cache model, where updates to objects in one JVM are propagated through to other JVMs using the same database.  If you have multiple eBiz middle tier servers and/or multiple OACoreGroup JVMs configured, there needs to be a mechanism to keep these separate JVM Java Caches synchronized.  A developer must specify which of the following two synchronization methods to use: 

  1. JVMs talk to each other and send the updated object to the other JVMs
  2. A JVM will just send an invalidation message for a specified object.  This is more commonly used.

An additional consideration is how to handle updates to data that happens outside of the JVMs, such as with updates via the Forms interface or a batch process.  For this case, an invalidation business event is triggered from the database, sending the JVMs an invalidation message for any records that have been updated.


If an object is invalidated in a JVM, it will be reread from the database when next accessed.  The new version will then be stored in the JVM's cache.



mzJavaD2:



A Brief History of Java Caching

Java Caching has been around in the E-Business Suite for quite a long time.  It was not used by many Apps products until 11.5.10, as this was the first release where Database Invalidation was introduced.  I believe iStore was the first team to utilize Java Caching functionality in any depth.  Since 11.5.10 CU2, more and more product teams are using Java Caching.


Understanding How to Patch Java Caching

There are several different dependent layers to Java Caching, which is implemented as Java Classes stored on the Apps Web Tier:


Application Server packages (oracle.ias.cache)
eBiz leverages Java Caching mechanism implemented by the iAS Technology itself.   It is sometimes required to apply iAS Technology patches for Oracle9i Application Server 1.0.2.2.2.  You should only apply patches that are released by the eBiz Applications Technology Group (ATG) team.


At the time of writing, the latest patch for this layer is:

    • Patch 5639951 "REHOST ORACLE JAVA OBJECT CACHE (FORMERLY OCS4J) FOR ORACLE APPLICATIONS 11i"

JTF and FND packages (oracle.apps.jtf.cache and oracle.apps.fnd.cache)
Updates to these packages are shipped via normal Apps patches.  Originally released via JTT product patchsets, these patches are now rolled into FND patchsets such as ATG RUP4.   The eBiz ATG development team is responsible for this layer.


At the time of writing, two patches are recommended on top of ATG RUP3 and ATG RUP4



Functional layer

It is up to the individual Applications product teams to utilise the Java Caching Framework provided by ATG.  Product teams are responsible for creating and maintaining their own "buckets" (more properly called a "Component Cache") within the Java Cache, and to control the invalidation mechanism for their own objects.   


Updates to code on this layer are released via product-specific patches.   Unfortunately, a list of product-specific patches is beyond the scope of this article.



Troubleshooting Common Problems


Updates to data not being shown immediately

A commonly reported symptom is that changes to users' responsibilities are not taking effect immediately.  Customers report this problem after applying the ATG RUP3 patch.  The Applications Technology Group implemented Java Caching for responsibilities (amongst other things) in ATG RUP3, so this is often the first time customers see Java Caching in action (or perhaps I should say "inaction" in this case).


If you're encountering this behavior, the following Note should help:


Java Caching through a firewall
Java Caching uses a separate set of port numbers for communication between JVMs.  These ports need to be opened through a firewall for Java Caching to function.   These ports are defined by the following profile options:
  • FND_CACHE_PORT_RANGE
  • JTF_DIST_CACHE_PORT
These profile options are configured in AutoConfig via these CONTEXT.xml file variables:
  • s_fnd_cache_port_range
  • s_java_object_cache_port


Run the Java Cache Diagnostic Tests

If you are experiencing any symptoms that could be attributed to Java Cache problems, it is important to gather the information from the Diagnostic Tests.  For example, one test will prove if database invalidation is functioning correctly.


These tests were introduced in JTT.E (11.5.10) and can be accessed via the "CRM HTML Administration" responsibility:

  • Diagnostics--> CRM Foundation--> Caching Framework

Review the javacache.log

This log file is located in $COMMON_TOP/rgf/<Instance >_<Hostname> and provides error messages specific to Java Caching.


Conclusion

Java Caching is simple in concept but can be complex to troubleshoot.  This article has hopefully given you some insight into the fundamental concepts.  We'll delve deeper into some of these concepts in my next article.

References

Related


Thursday Mar 01, 2007

Latest JVM Tuning Recommendations for Apps 11i

In my previous article "Configuring Middle-Tier JVMs for Applications 11i" I provided some suggestions in the paragraph "Rough Guidelines for JVMs" as to how many JVMs could be implemented as an initial starting point.  Subsequent to that blog article,  Guidelines to setup the JVM in Apps 11i"
(Metalink Note 362851.1) has been released which includes the ATG Performance Group's official recommendations, some of which are different than stated in my original article. 

JConsole_heap:

In this article I would like to go into a little bit more depth about the newly published recommendations.  I will focus only on the OACoreGroup JVM and not consider the Forms Servlet, Discoverer or XML Service JVM settings.  I will also assume the reader has some familiarity with Java and eBiz 11i already and is also using eBusiness version 11.5.10 with JDK 1.4 or 1.5.   


Background

From a CPU utilization perspective, there is no need to run more than 1 JVM per CPU because of the following reasons:

  • The JVM uses native threads and the native threads can be scheduled on any of the available cores/CPUs. 
  • Multiple parallel Garbage Collection (GC)  threads -- introduced in 1.4 -- are spawned.  Running multiple JVMs per CPU could result in an excessive number of background and GC threads. 
  • Multiple JVM increases the number of JDBC connections required and there is a memory and CPU overhead for each JVM process. 

For the majority of our web applications, more than 90% of the memory allocations are transient (for the life of the request or page), so new space GCs dominate much more than old generation GCs.  Old generation GCs are usually only a problem with older versions of Applications code or modules such as Configurator which load a large amount of objects (e.g. config model) in the old space.


Why increase the number of JVMs

There are several reasons why you would want to increase the number of JVMs:

  1. Minimize GC pause overhead
  2. Reduce the JVM heap size
  3. Minimize contention due to hot synchronized methods (i.e. monitor contention).
Based on Oracle's performance testing, benchmarking, and our experience with working directly with enterprise-class customers, we have determined that 100 users per OACore JVM has generally proven to be a practical and performant number.   Increasing the session.timeout parameter in zone.properties above the 30 minute default may increase JVM memory requirements.

Recomendations in summary

Guideline #1:   For the OACoreGroup JVMs start with the lower of the following two values:

  1. Number of cores on the middle tier
  2. Peak Concurrent users / 100
For example: 

If you have 2 x Dual Core CPUs in your server and have 500 peak users, then 4 JVMs is the recommended starting point, since the number of cores is the lower number.  However, if you only had 300 peak users, then you would configure 3 JVMs as a starting point as this is now the lower figure.


Guideline #2:  Size your maximum heap size as either 512 MB or 1 GB.  If you start with 512 MB and find that more memory is required, increase to a maximum of 1 GB.   If more memory than 1 GB is required, then add another JVM instead (free physical memory allowing) to increase the total memory available overall. 

For example:

You are using 1 x JVM with 1 GB heap size and find you need to increase memory.  Configure your system for 2 JVMs, each with 750 MB heap size, thus providing 1.5 GB in total.


Your mileage will vary


As always, these recommendations are generalised and can only act as a suggested starting point for your system.  As mentioned in my previous blog article, there are many factors that effect how the JVM is utilized, so you will need to undertake representative testing to prove that the number of JVMs you select is suitable for your specific user load and profile.

Related


Friday Feb 16, 2007

Tuning JVMs for Release 11i

Some people consider tuning the performance of Java Virtual Machines (JVMs) in an E-Business Suite environment to be more of an art than science.  These comments usually result because of the large number of factors that may be considered, factors that are sometimes difficult to quantify.

JConsole_heap:

Adding to the existing material available to help you perfect your art, our E-Business Suite Performance Group released a short whitepaper late with some tips:
This short but lucid Note covers topics such as:
  • The number of recommended users per JVM
  • The number of recommended JVMs per CPU
  • Heap configuration tips for OACoreGroup, with guidelines for Forms Servlet mode and Configurator use
  • Recommended techstack patches
Related Articles

Tuesday Oct 31, 2006

Apps 11i Jinitiator Conflicts with Other Java Clients for Discoverer

Today we'll take a short breather from OpenWorld coverage, since I'm still doggedly plowing through what seems like a terabyte's worth of Powerpoint presentations.


If you're still debating the value of participating in our Early Adopter Program for replacing Oracle's Jinitiator with the native Sun Java plug-in for your E-Business Suite users, here's another possible conflict scenario with Oracle Business Intelligence 10g that's recently been uncovered.

If you have both of the following combinations in your environment:
  1. E-Business Suite Release 11i configured with Jinitiator 1.3.x
  2. Discoverer 10g 10.1.2 configured with Sun Java 1.4.x or 1.5.x
Then you may encounter conflicts between the Java Virtual Machines for those two configurations.  These conflicts will trigger one or both of the following errors:
  • 'Runtime Error! Program: C:Program FilesInternet Exploreriexplore.exe abnormal program termination'
  • "Register Failk"
If you run into this situation, Metalink Note 396773.1 lists a number of possible workarounds.  It may also be worth looking into our Early Adopter Program, as well, to see if that configuration eliminates the conflict in a less-painful manner than the workarounds.

Related

Thursday Oct 19, 2006

Investigating java.lang.OutOfMemoryError with Apps 11i Middle Tier JVMs

If you had to guess what the error "java.lang.OutOfMemoryError" indicates when seen in your Apache log files, you would probably be right...

In most cases this message is telling you that the Java process has exhausted available memory and cannot process some requests successfully.   In many cases additional symptoms will be:

  • Poor web page performance
  • User requests being timed out
  • Java processes taking 100% CPU on your server.  
The Java Virtual Machine (JVM) will sometimes be restarted by oprocmgr as performance is so poor it presumes the JVM has died.

There can be other causes.  For example, this message can sometimes be given where there is the lack of some other resource such as threads per process (kernel parameter "max_thread_proc" on HP-UX) but this case should be easily identifiable from the exact message written out.

Where has all the memory gone?


Every object created in Java takes some memory.  Once all java code that references the java object has completed, then the Garbage Collector (GC) automatically removes the object and claims back the memory.

You can run out of memory due to:



  • Insufficient memory allocated to the JVM to cope with the amount of work
  • Suboptimal GC processing
  • A memory leak -- Java code is not releasing an object when it should
  • A Java code or operating system defect
How are JVMs configured with Apps 11i

Two important configuration files for Java in an Apps environment are:
  • jserv.properties
    This file controls the JVM parameters in the "wrapper.bin.parameters=" entry.  When you install Apps originally or update the JDK version, the default settings are updated in your CONTEXT.xml file ("s_jvm_options" for the OACoreGroup JVM) and controlled by AutoConfig thereafter.

What can I do about OutOfMemoryError?


  • Are you using Oracle Configurator?
Configurator can sometimes require huge chunks of memory for a relatively long period of time, so Oracle now recommends setting up separate JVMs specifically for Configurator.  For more details see Setting Up a Dedicated JServ for Running Only Oracle Configurator" (Metalink Note 343677.1).

  • Increase available memory
An obvious choice, you may think, and this will often at least delay the effect of the issue, but may not always solve it.   You can achieve this by either increasing the number of JVMs or the memory settings ("-Xmx512M" for example).  In general, I would not recommend going much over 640MB per JVM, in order to keep the Garbage Collection time to a reasonable level.

  • Tuning GC

You can influence how Java performs Garbage Collection.   I would normally recomend reviewing and possibly backing out any non-default JVM parameters first, as these are sometimes just carried forward because they were used with previous JDK versions.  As always, any tuning should be undertaken in a TEST environment to ensure the changes have a positive impact.   


  • Using Concurrent Mark Sweep collector

With multi-CPU servers using JDK 1.4 and higher, the following parameters have been seen to improve performance and GC throughput on some environments:



-XX:+UseConcMarkSweepGC -XX:+UseParNewGC  



  • Identify the components that are using memory

Not always as easy as it sounds.  Generating class histograms or java profiling may be the only options. 


What data should I gather to investigate further ?

The initial investigation should focus on trying to identify the pattern of memory usage building up and how frequent GC is occurring.  The tools and information described below may help to achieve this



  • Review stdout files

Basic Garbage Collection information is normally written to the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm/*.stdout file for each JVM.  Additional Garbage Collection information can be logged, for example:



-XX:+PrintGCTimeStamps -XX:+PrintGCDetails



  • Add more verbose GC logging


-Xverbosegc


This allows you to see how much of each memory pool is being used.  For example you may find it is the "Permanent Generation" space that is running out of memory, rather than the "Tenured Generation".  If this turns out to be the case, the "-XX:MaxPermSize" parameter could be increased to provide more memory.



  • Collect class histograms


-XX:+PrintClassHistogram


Implemented in JDK 1.4 and higher by some vendors.  Can be very useful if available.  Beware of using this on HP-UX platforms running JDK 1.4 as it seems to crash the JVM rather than output the data!



  • Use JConsole

Discussed in a previous article: Using JConsole to monitor Apps 11i JVMs



  • Java Profiling


-Xrunhprof


This should tell you exactly where the memory is being used, but has a huge impact on performance and is therefore not likely to be possible to gather this data, except for a lightly used test environment.



  • Other clues

Does the DMS output for the JVM show increasing active connections or threads?


Do JDBC connections to the database keep increasing? What module names do they relate to?


Does temporarily disabling "AM Pool" reduce memory consumption (Profile option "FND: Application Module Pool Enabled")?


Are you seeing PL/SQL "cursor leaks"?  See, "Understanding and Tuning the Shared Pool" (Metalink Note 62143.1) in section "Checking hash chain lengths" for SQL to check this.


  • Any Java Thread Deadlocks or Database Deadlocks?

Are Java processes spinning to 100% CPU or is CPU use very low when the error occurs?


  • Look for platform references

Review your operating system vendor's web site to look for known issues or required platform patches for your Java version.


My Production system is failing every few hours, what can I do right now?

A possible quick fix would be to increase memory and/or number of JVMs.  Be sure to ensure that this won't cause the operating system to start swapping!


You could also consider a scheduled bounce of Apache every 12 or 24 hours if possible. 


These steps may allow you to minimise the immediate impact on the business so you can investigate and implement changes in a methodical, safe, controlled and tested manner.


Where can I read more about Java memory and tuning?

There are many resources on the Internet to describe how Java uses memory and what steps to take for performance tuning.  The best place to start is your hardware vendors web site as the options available depend on their Java implementation and also the specific Java version you are using.  


For example:



Conclusion

Dealing with Java processes running out of memory can sometimes be as simple as increasing the memory available, or may require detailed investigation to identify the root cause.     


Although each case will have its own unique considerations, I hope that the information in this article has given you some ideas as to the general approach to take should the need arise.

Additional References


About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
19
20
21
23
24
25
26
27
28
29
30
   
       
Today