Configuring Middle-Tier JVMs for Applications 11i

When you call Oracle Support with a problem like apj12 errors in your mod_jserv.log, middle-tier Java Virtual Machines (JVMs) crashing, or poor middle-tier performance, then it will often be suggested to increase the number of JVM processes.  So, the key question that likely occurs to you is, "How many JVMs are required for my system?"

Processing Java Traffic in Groups

First, some quick background:  web requests received by Oracle HTTP Server (Apache) to process Java code is sent to one of four different types of JVM groups to be processed.   You can see this in the jserv.conf file:

 ApJServGroup OACoreGroup 2 1 /usr/.../jserv.properties
 ApJServGroup DiscoGroup  1 1 /usr/.../viewer4i.properties
 ApJServGroup FormsGroup  1 1 /usr/.../forms.properties
 ApJServGroup XmlSvcsGrp  1 1 /usr/.../xmlsvcs.properties

The number of JVMs for each group is signified by the first number on each line.
  • OACoreGroup is the default group.  This is where most Java requests will be serviced
  • DiscoGroup is only used for Discoverer 4i requests
  • FormsGroup is only used for Forms Servlet requests
  • XmlSvcsGrp is for XML Gateway, Web Services, and SOAP requests

In the example above, I have two JVMs configured for OACoreGroup and one JVM configured for each of the other groups.

Factors Affecting Number of JVMs Required

Determining how many JVMs to configure is a complex approximation, as many factors need to be taken into account.  These include:
  • Hardware specification and current utilization levels
  • Operating system patches and kernel settings
  • JDK version and tuning
  • Applications code versions, especially JDBC and oJSP
  • JServ configuration file tuning (jserv.properties and zone.properties)
  • Applications modules being used
  • How many active users
  • User behaviour

Rough Guidelines for JVMs

Luckily, Oracle Development have undertaken various performance tests to establish some rules of thumb that can be used to configure the initial number of JVMs for your system.

OACoreGroup
    •    1 JVM per 100 active users
DiscoGroup
    • Use the capacity planning guide from Note 236124.1 "Oracle 9iAS 1.0.2.2 Discoverer 4i: A Capacity Planning Guide"
FormsGroup
    •    1 JVM per 50 active forms users
XmlSvcsGrp
    •    1 JVM is generally sufficient

In addition to this, Oracle generally recommends no more than 2 JVMs per CPU.    You also need to confirm there are enough operating system resources (e.g. physical memory) to cope with any additional JVMs.

Your Mileage Will Vary

The general guidelines above are just that -- they're very broad estimates, and your mileage will vary.  As I write this, the Applications Technology Group is working on a JVM Sizing whitepaper that will provide guidelines based on whether your E-Business Suite deployment is small, medium, or large.  I'll profile this whitepaper here as soon as it's released publicly.

Until then, it's critical that you test your environment under load, using transactional tests that closely mirror what your users will be doing.  It's useful to use automated testing tools for this, as you create your benchmarks. 

Here are a couple of quick-and-dirty tools that might be useful in sizing your JVMs.

Script to determine "active users" for OACoreGroup

REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate   
and counter < limit_connects;
REM
REM END OF SQL
REM

How to determine "active forms users" for FormsGroup

Check the number of f60webmx processes on the Middle Tier server.  For example:

ps -ef | grep f60webmx | wc -l

Conclusion

  • The number of required JVMs is extremely site-specific and can be complex to predict
  • Use the rules of thumb as a starting point, but benchmark your environment carefully to see if they're adequate
  • Proactively monitor your environment to determine the efficiency of the existing settings and reevaluate if required

More on Java Memory Tuning Later

Once you've established the right number of JVMs to use, it's then time to optimize them.  I'm intending to discuss Java memory tuning and OutOfMemory issues in a future article.  Stay tuned.

Related


Comments:

Mike,

Thank you very much for this JVM article. Looking forward to the JVM memory tuning one.

Posted by Hongtao Zhu on August 01, 2006 at 09:38 AM PDT #

mike this is a very intersting and new info. form me i have been in the field for 3.5 years now and this is the fisrt time i knew such info. so thank yhou

fadi hasweh

Posted by Fadi Hasweh on August 01, 2006 at 08:12 PM PDT #

mike i have a qustion on my jser.conf i have the #ApJServGroup FormsGroup commeted out is this correct should i remove it or what.

thanks
fadi

Posted by Fadi Hasweh on August 01, 2006 at 09:22 PM PDT #

Mike,

Thank you very much for your article. It really helps.
Your advice is 100% right and useful.
One bit that is not covered in it is why it is advisable to run multiple JVM?
Can you please advice on it?
Bellow is my version.

Please correct me if I am wrong.
In Ideal World there is no single reason to have more then 1 JVM per host (environment).
Even opposite it is advisable to have one JVM from performance and resource utilisation point of view.
As if all JVM threads running in one OS process context they can utilise common memory structures in more efficient ad quicker way. Running just one JVM eliminates the overhead introduced by JVM background threads (you keep just one instance of those). Shared memory structures such as pool of JDBC connections, pool of JVM treads etc. are utilised in more efficient way.

Why Oracle & Co recommends to run multiple JVM then?

My answer is: Because of BUG-s and Developers errors.
I am talking not just about Oracle. Bugs and structures management problems are everywhere.
Os have got some memory management limitation and problems, JDK (JRE) have its own problems. Oracle code isn’t ideal as well.
Because of that and the fact that memory and memory structures problems are likely to occur if there is more users connected to single JVM, we pushed to use multiple JVM-s.
In fact it is only way nowadays to run J2EE processes, otherwise systems are unstable and unreliable.

Can you think about any other reason why to run multiple JVM for single Apache instance?

Thank you in advance,
Yury

I have spend some time to troubleshoot JVM-s previously ;)

https://portal.hotsos.com/events/SYM06/speakers/abstracts/fusion-middleware-performance-tuning-and-troubleshooting

Posted by Yury Velikanov on August 02, 2006 at 01:43 AM PDT #

Thanks you all for your feedback, it really makes it worthwhile to hear you are reading and enjoying these articles

With regard to the FormsGroup being commented out, this is quite normal and indicates you do not use Forms Listener Servlet. If you implement Forms Listener Servlet by following Note 201340.1 "Using Forms Listener Servlet with Oracle Applications 11i" then part of that process is to modify the setup via AutoConfig which will take out the comment and startup this JVM

Posted by Mike Shaw on August 02, 2006 at 06:58 PM PDT #

Yury has made a very interesting observation. With todays powerful CPUs, support for large heap sizes and 64 bit operating systems, why do we need multiple JVM and their associated overhead ? This could be subject to a discussion all on it's own but my view is expressed briefly below....

There are several reasons why you may want to configure multiple JVM.

Historically the problem has been that periodic Full Garbage Collection (GC) in Java is "stop the world", meaning all processing in the JVM stops completely whilst the Full GC is underway. This can take a relatively long time even for small Heap sizes, having a significant impact on user performance. This issue is being overcome with the latest versions of Java through the use of Parallel and incremental GC, but having one JVM with a very large Heap size could possibly re-introduce this as an issue.

Operating system or physical hardware limitations can also come into play. There is a finite amount of resources available per process, so for high volume sites you may not be able to configure enough heap size, TCP sockets or number of threads per process for example. Java also does a lot of work, with background monitoring, creating and managing the huge number of internal objects, as well as the work the users are asking it to do, so believe CPU will eventually become the bottleneck in this scenario

Resiliance and high availability. Why do you have multiple physical Web Servers, RAC nodes (or whatever) rather than 1 big one ? If you have one JVM and it goes down then all users are immediately effected. If you have two JVMs then only half your users are effected if one goes down, these users can immediately reconnect and any new connections will be uneffected as they are routed to the running JVM.

As operating systems, Java itself and Applications code improves, this will eventually negate the arguments against having only one JVM per server but for the moment do not believe we have reached this ideal

Posted by Mike Shaw on August 02, 2006 at 07:46 PM PDT #

thank you mike for taking time to reply to our qustion and to contribute to this great site

Posted by Fadi Hasweh on August 02, 2006 at 09:02 PM PDT #

good and valid info.
With Muliple Users Logged in to Apps , How does app use multiple jvm. what parameters are used to distribute the load.

Thanks

Raghu

Posted by Rajagopalan Raghuraman on August 10, 2006 at 04:54 AM PDT #

Hi Raghu, thanks for your comments

mod_Oprocmgr maintains an internal table of the JVM Groups and the PIDs of the processes started. When a new client request comes in to Apache for a Jserv request, it will be routed to one of the JVM processes in the appropriate group. A session cookie named "JServSessionId&lt;Group>&lt;hostname>" is returned in the response to the browser, which contains a session id. Subsiquent requests from the same browser session will pass back this cookie to Apache, which uses it to route requests through to the same JVM (where session state is maintained for that client)

Unfortunately I don't know the exact algorithm used to choose the JVM in the initial request but presume it is using a simple round robin mechanism. You can see this selection process in action in the mod_jserv.log file which logs messages like :-
balance: new session:
balance: continuing session to host.domain Port

Posted by Mike Shaw on August 10, 2006 at 07:18 PM PDT #

Did Applications Technology Group came with any JVM Sizing whitepaper?

Posted by vijay on February 16, 2007 at 08:36 AM PST #

Vijay,You might wish to check out this recent article:Tuning JVMs for Release 11iRegards,Steven&nbsp;

Posted by Steven Chan on February 19, 2007 at 01:06 AM PST #

Hi Steven
i want add one mor jvm to our middle tier .we have out of memory problem .so we are planning to add one more jvm can you please give what should be consider to add additional JVM

Thanks
Kumar

Posted by kumar on March 20, 2009 at 12:24 AM PDT #

Hello Kumar,

It is not always a simple answer to your question. If you have enough free CPU and RAM capacity then it should be harmless to add another JVM and could be of some benefit. Whether this solves an OOM error is another matter, as if there is a memory leak (rather than a capacity issue) then you will still run out of memory eventually

I would therefore recomend raising a SR with Oracle Support, so we can review your specific configuration and symptoms and make an informed judgement as to what the best plan will be for your specific circumstances

regards

Mike Shaw

Posted by Mike Shaw on March 23, 2009 at 05:21 PM PDT #

Hi!
There is a character “m” missing from command

f60webx = f60webmx

“ps -ef | grep f60webmx | wc -l “

Posted by Gabriel Costache on December 09, 2009 at 06:09 PM PST #

Thanks Gabriel, have modified the command now.

Posted by Mike Shaw on December 09, 2009 at 09:40 PM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today