When you call Oracle Support with a problem like apj12 errors in your mod_jserv.log, middle-tier Java Virtual Machines (JVMs) crashing, or poor middle-tier performance, then it will often be suggested to increase the number of JVM processes. So, the key question that likely occurs to you is, "How many JVMs are required for my system?"
Processing Java Traffic in Groups
First, some quick background: web requests received by Oracle HTTP Server (Apache) to process Java code is sent to one of four different types of JVM groups to be processed. You can see this in the jserv.conf file:
ApJServGroup OACoreGroup 2 1 /usr/.../jserv.properties
ApJServGroup DiscoGroup 1 1 /usr/.../viewer4i.properties
ApJServGroup FormsGroup 1 1 /usr/.../forms.properties
ApJServGroup XmlSvcsGrp 1 1 /usr/.../xmlsvcs.properties
- OACoreGroup is the default group. This is where most Java requests will be serviced
- DiscoGroup is only used for Discoverer 4i requests
- FormsGroup is only used for Forms Servlet requests
- XmlSvcsGrp is for XML Gateway, Web Services, and SOAP requests
In the example above, I have two JVMs configured for OACoreGroup and one JVM configured for each of the other groups.
Factors Affecting Number of JVMs Required
- Hardware specification and current utilization levels
- Operating system patches and kernel settings
- JDK version and tuning
- Applications code versions, especially JDBC and oJSP
- JServ configuration file tuning (jserv.properties and zone.properties)
- Applications modules being used
- How many active users
- User behaviour
Rough Guidelines for JVMs
Luckily, Oracle Development have undertaken various performance tests to establish some rules of thumb that can be used to configure the initial number of JVMs for your system.
- 1 JVM per 100 active users
- Use the capacity planning guide from Note 236124.1 "Oracle 9iAS 1.0.2.2 Discoverer 4i: A Capacity Planning Guide"
- 1 JVM per 50 active forms users
- 1 JVM is generally sufficient
In addition to this, Oracle generally recommends no more than 2 JVMs per CPU. You also need to confirm there are enough operating system resources (e.g. physical memory) to cope with any additional JVMs.
Your Mileage Will Vary
The general guidelines above are just that -- they're very broad estimates, and your mileage will vary. As I write this, the Applications Technology Group is working on a JVM Sizing whitepaper that will provide guidelines based on whether your E-Business Suite deployment is small, medium, or large. I'll profile this whitepaper here as soon as it's released publicly.
Until then, it's critical that you test your environment under load, using transactional tests that closely mirror what your users will be doing. It's useful to use automated testing tools for this, as you create your benchmarks.
Script to determine "active users" for OACoreGroup
REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate
and counter < limit_connects;
REM
REM END OF SQL
REM
How to determine "active forms users" for FormsGroup
Check the number of f60webmx processes on the Middle Tier server. For example:
ps -ef | grep f60webx | wc -l
Conclusion
- The number of required JVMs is extremely site-specific and can be complex to predict
- Use the rules of thumb as a starting point, but benchmark your environment carefully to see if they're adequate
- Proactively monitor your environment to determine the efficiency of the existing settings and reevaluate if required
More on Java Memory Tuning Later
Once you've established the right number of JVMs to use, it's then time to optimize them. I'm intending to discuss Java memory tuning and OutOfMemory issues in a future article. Stay tuned.
Related
- Proactive Monitoring on an Apps 11i Web Tier Server
- In-Depth: Load-Balancing E-Business Suite Environments
Comments (13)
Mike,
Thank you very much for this JVM article. Looking forward to the JVM memory tuning one.
Posted by Hongtao Zhu | August 1, 2006 4:38 PM
Posted on August 1, 2006 16:38
mike this is a very intersting and new info. form me i have been in the field for 3.5 years now and this is the fisrt time i knew such info. so thank yhou
fadi hasweh
Posted by Fadi Hasweh | August 2, 2006 3:12 AM
Posted on August 2, 2006 03:12
mike i have a qustion on my jser.conf i have the #ApJServGroup FormsGroup commeted out is this correct should i remove it or what.
thanks
fadi
Posted by Fadi Hasweh | August 2, 2006 4:22 AM
Posted on August 2, 2006 04:22
Mike,
Thank you very much for your article. It really helps.
Your advice is 100% right and useful.
One bit that is not covered in it is why it is advisable to run multiple JVM?
Can you please advice on it?
Bellow is my version.
Please correct me if I am wrong.
In Ideal World there is no single reason to have more then 1 JVM per host (environment).
Even opposite it is advisable to have one JVM from performance and resource utilisation point of view.
As if all JVM threads running in one OS process context they can utilise common memory structures in more efficient ad quicker way. Running just one JVM eliminates the overhead introduced by JVM background threads (you keep just one instance of those). Shared memory structures such as pool of JDBC connections, pool of JVM treads etc. are utilised in more efficient way.
Why Oracle & Co recommends to run multiple JVM then?
My answer is: Because of BUG-s and Developers errors.
I am talking not just about Oracle. Bugs and structures management problems are everywhere.
Os have got some memory management limitation and problems, JDK (JRE) have its own problems. Oracle code isn’t ideal as well.
Because of that and the fact that memory and memory structures problems are likely to occur if there is more users connected to single JVM, we pushed to use multiple JVM-s.
In fact it is only way nowadays to run J2EE processes, otherwise systems are unstable and unreliable.
Can you think about any other reason why to run multiple JVM for single Apache instance?
Thank you in advance,
Yury
I have spend some time to troubleshoot JVM-s previously ;)
https://portal.hotsos.com/events/SYM06/speakers/abstracts/fusion-middleware-performance-tuning-and-troubleshooting
Posted by Yury Velikanov | August 2, 2006 8:43 AM
Posted on August 2, 2006 08:43
Thanks you all for your feedback, it really makes it worthwhile to hear you are reading and enjoying these articles
With regard to the FormsGroup being commented out, this is quite normal and indicates you do not use Forms Listener Servlet. If you implement Forms Listener Servlet by following Note 201340.1 "Using Forms Listener Servlet with Oracle Applications 11i" then part of that process is to modify the setup via AutoConfig which will take out the comment and startup this JVM
Posted by Mike Shaw | August 3, 2006 1:58 AM
Posted on August 3, 2006 01:58
Yury has made a very interesting observation. With todays powerful CPUs, support for large heap sizes and 64 bit operating systems, why do we need multiple JVM and their associated overhead ? This could be subject to a discussion all on it's own but my view is expressed briefly below....
There are several reasons why you may want to configure multiple JVM.
Historically the problem has been that periodic Full Garbage Collection (GC) in Java is "stop the world", meaning all processing in the JVM stops completely whilst the Full GC is underway. This can take a relatively long time even for small Heap sizes, having a significant impact on user performance. This issue is being overcome with the latest versions of Java through the use of Parallel and incremental GC, but having one JVM with a very large Heap size could possibly re-introduce this as an issue.
Operating system or physical hardware limitations can also come into play. There is a finite amount of resources available per process, so for high volume sites you may not be able to configure enough heap size, TCP sockets or number of threads per process for example. Java also does a lot of work, with background monitoring, creating and managing the huge number of internal objects, as well as the work the users are asking it to do, so believe CPU will eventually become the bottleneck in this scenario
Resiliance and high availability. Why do you have multiple physical Web Servers, RAC nodes (or whatever) rather than 1 big one ? If you have one JVM and it goes down then all users are immediately effected. If you have two JVMs then only half your users are effected if one goes down, these users can immediately reconnect and any new connections will be uneffected as they are routed to the running JVM.
As operating systems, Java itself and Applications code improves, this will eventually negate the arguments against having only one JVM per server but for the moment do not believe we have reached this ideal
Posted by Mike Shaw | August 3, 2006 2:46 AM
Posted on August 3, 2006 02:46
thank you mike for taking time to reply to our qustion and to contribute to this great site
Posted by Fadi Hasweh | August 3, 2006 4:02 AM
Posted on August 3, 2006 04:02
good and valid info.
With Muliple Users Logged in to Apps , How does app use multiple jvm. what parameters are used to distribute the load.
Thanks
Raghu
Posted by Rajagopalan Raghuraman | August 10, 2006 11:54 AM
Posted on August 10, 2006 11:54
Hi Raghu, thanks for your comments
mod_Oprocmgr maintains an internal table of the JVM Groups and the PIDs of the processes started. When a new client request comes in to Apache for a Jserv request, it will be routed to one of the JVM processes in the appropriate group. A session cookie named "JServSessionId<Group><hostname>" is returned in the response to the browser, which contains a session id. Subsiquent requests from the same browser session will pass back this cookie to Apache, which uses it to route requests through to the same JVM (where session state is maintained for that client)
Unfortunately I don't know the exact algorithm used to choose the JVM in the initial request but presume it is using a simple round robin mechanism. You can see this selection process in action in the mod_jserv.log file which logs messages like :-
balance: new session:
balance: continuing session to host.domain Port
Posted by Mike Shaw | August 11, 2006 2:18 AM
Posted on August 11, 2006 02:18
Did Applications Technology Group came with any JVM Sizing whitepaper?
Posted by vijay | February 16, 2007 4:36 PM
Posted on February 16, 2007 16:36
Vijay,You might wish to check out this recent article:Tuning JVMs for Release 11iRegards,Steven
Posted by Steven Chan | February 19, 2007 9:06 AM
Posted on February 19, 2007 09:06
Hi Steven
i want add one mor jvm to our middle tier .we have out of memory problem .so we are planning to add one more jvm can you please give what should be consider to add additional JVM
Thanks
Kumar
Posted by kumar | March 20, 2009 7:24 AM
Posted on March 20, 2009 07:24
Hello Kumar,
It is not always a simple answer to your question. If you have enough free CPU and RAM capacity then it should be harmless to add another JVM and could be of some benefit. Whether this solves an OOM error is another matter, as if there is a memory leak (rather than a capacity issue) then you will still run out of memory eventually
I would therefore recomend raising a SR with Oracle Support, so we can review your specific configuration and symptoms and make an informed judgement as to what the best plan will be for your specific circumstances
regards
Mike Shaw
Posted by Mike Shaw | March 24, 2009 12:21 AM
Posted on March 24, 2009 00:21