Do You Need to Bounce E-Business Suite Application Servers Regularly?

A customer recently reported an issue where they needed to bounce their E-Business Suite application tier (mid-tier) servers once a week to resolve stability issues.  This shouldn't be necessary.  We recommend against bouncing EBS application servers regularly.  We've done a lot of work with the E-Business Suite to ensure that regular bounces are not required with the latest releases.

Bouncing your system can reduce performance

It seems counter-intuitive, but bouncing your EBS application tier servers can actually reduce your system's performance rather than improve it.  Oracle E-Business Suite is explicitly designed to minimize round trips between the application tier and the database.  We make extensive use of caches:

  • JDBC cache buffers
  • Java object cache
  • MSD cache for OA Framework pages
  • ... and others
Every time you bounce your mid-tier, you clear these caches.  The longer your mid-tier remains up and running, the better-populated these caches will be, and the better your overall system performance.

Treat the symptoms but miss the real problem

If you need to bounce your EBS application tier server regularly, there's a deeper problem with your environment. 

Here are some issues that can be deceptively-masked by bouncing your EBS mid-tier:

  1. Improperly-tuned heap space (e.g. insufficient heap space, insufficient space for perm generation)
  2. Memory leaks (usually the result of poorly-designed customizations)
  3. Memory hemorrhage (usually the result of infinite loops or poorly-designed custom queries)
  4. Heap space fragmentation (usually the result of running older Java releases)
  5. JDBC connection leaks (resolved in 11.5.10.2 and all R12 releases)

If you're experiencing problems that seem to go away when you bounce your mid-tier, that's a red flag.  You should log a formal Service Request with Oracle Support so that we can help you identify the root cause.

References

Related Articles


Comments:

Steve,

I'm glad you wrote about this and I hope other customers give their feedback on this topic.

All the issues mention above are all real-world issues faced by customers regularly. We've been trying to get the java heap size right for years now, we still get the occasional java.lang.OutOfMemory errors. My experience working on these issues with Oracle support has been very very frustrating. I recently raised an SR on this, but had to close it after a few weeks as the analyst was kind of shooting in the dark.

If someone in the EBS Performance team in Oracle can clearly lay out some best practices document and settings to avoid these issues, it would tremendously help the Oracle Apps DBA community.

Thanks
Naveen Garg

Posted by Naveen Garg on September 01, 2011 at 01:34 PM PDT #

Naveen,

No single set of settings will meet everyone's needs. Your first step should be to review the presentation linked in the article above to isolate the issues as much as possible. Once you get to that stage, log another SR, send it to me, and I'll have one of our Applications Performance specialists in ATG Development work with your Support engineer on it.

Regards,
Steven

Posted by Steven Chan on September 01, 2011 at 01:59 PM PDT #

Steven,

The screen shot is from JConsole? If so, how do you get it to work? I never could make it work with either 11i, 12.0.x or 12.1.3. I found many doc on metalink and even opened SR and never could get my issue fix.

Whenever I start JConsole, the only process it finds is itself.
Any idea?

Thanks alot!

Posted by guest on September 02, 2011 at 03:46 AM PDT #

What version of java are you using? There can be some issues with JDK lower than 1.5.10. Additionally, ensure that the following system property is set in the server

com.sun.management.jmxremote

Additionally, if you want to connect remotely (let's say from your desktop to the server), you need additional parameters depending on your current configuration. If you have multiple JVMs per instance, you need to connect via process PID locally on the server given that multiple JVMs will compete for the same port in the server. Just execute $INST_TOP/admin/scripts/adopmnctl.sh status to find the OACORE instances PIDs.

Feel free to shoot me an email at gustavo.jimenez@oracle.com for further details.

Regards,
Gustavo

Posted by Gustavo on September 02, 2011 at 07:21 AM PDT #

Naveen,

Complementing Steven's comment above, please review below:

In order to tune your JVM settings in case you run out of heap space in your JVM, ensure that these JVM options are present:

-XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/some_dir/

This will cause the JVM to create an HPROF file for later examination with any heap analysis tool (i.e.: Eclipse Memory Analyzer) to find the source of the leaks and fix or tune heap space accordingly.

Note that the above JVM options are available starting with JDK 1.5.10, so you definitely need to upgrade your JDK if you are below 1.5.10.

If the heap shows signs of memory leaks caused by e-Business Suite code or techstack components, please do let us know right away via a Service Request with Oracle Support.

Thanks,
Gustavo

Posted by Gustavo Jimenez on September 02, 2011 at 07:30 AM PDT #

Hi Steve,

Thanks for nice Sharing. How about the MWA? Refer to this Note "MWA Troubleshooting Tips for Release 11i [ID 269991.1]", it's need regular bounce for MWA.

Regards,
suratman

Posted by Suratman on September 04, 2011 at 01:54 PM PDT #

Suratman,

Since MWA runs in its own dedicated JVM (outside of the J2EE container-namely OC4J or Apache Jserv), it will not affect the entire stability of the container. Your point is valid and should perhaps be addressed via a service request with the MWA team to ensure and/or to provide reasons for this requirement.

The objective of this blog entry was to clear confusion regarding the need of bouncing the main middle tier J2EE container at regular intervals, which is not affected by the MWA telnet server.

Having said the above, regarding the note you mention, I extract the following paragraph which caught my attention:

"3. It is always recommended to be on the latest version of Java for the JDK Caching ability and MWA performance is increased with each newer version. The following Documents can be used for upgrading the JDK. With JDK versions 5 and 6, MWA no longer requires a daily bounce. "

Thanks,
Gustavo

Posted by Gustavo Jimenez on September 06, 2011 at 04:52 AM PDT #

I'd agree with Naveen's initial comments. If you want to avoid doing the daily bounce and find the cause of the issue, then be prepared for weeks of investment in time to troubleshoot the problem. During this time be prepared to hit the problem (for which you have implemented the bounce) numerous times and see how the business like it.
My experience has led me to over allocate JVMs to mitigate against against memory leaks and perform a staggered bounce accross Apps tiers. The cost in doing this has been far better than the cost of the time invested in going backwards and forwards with Oracle.
Until we have an easier method to find the culprits (perhaps when weblogic is introduced) I'm sticking to this.

Posted by Jen Alsop on March 07, 2013 at 05:24 AM PST #

Hello, Jen,

Thanks for sharing your own experiences. Everyone's mileage varies, since everyone's respective tolerance for bounces and investigating SRs is different. I'm glad that you've found an approach that works for you.

If you're finding that bounces are required, we still would prefer to help track down the root issues in EBS environments that cause JVM leaks. It's entirely possible that we've already fixed the issue in a released patch, so it never hurts to call us before resigning yourself to a rolling bounce schedule.

Regards,
Steven

Posted by Steven Chan on March 07, 2013 at 08:00 AM PST #

I would like to add one question, recently we migrated on inv_replenish_detail_pub package to production instance . next day PO receipt started giving error RVTII-060-Subroutine
INV_TXN_MANAGER_PUB.process_transaction() returned Error

we bounce application server and problem got resolved. is this mandatory to bounce the server or any configuration problem in server?

Posted by guest on August 12, 2013 at 08:45 AM PDT #

Hi, Guest,

I'm sorry to hear that you've encountered an issue with this. You shouldn't have to bounce your server to resolve this issue.

If you need to do this on an ongoing basis, you should get our help in identifying the root cause.

Your best bet would be to log a formal Service Request via My Oracle Support (formerly Metalink) to get one of our specialists engaged.

Regards,
Steven

Posted by Steven Chan on August 12, 2013 at 09:08 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« March 2015
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today