Dhiraj Soni's Weblog

My recent presentation on "Building a Complex Web Application Using ADF and Siebel" is now on OTN!
Its linked from: http://www.oracle.com/technology/tech/fmw4apps/siebel/webcasts.html

Oracle releases regular newsletters about current happenings in different areas, this is a very useful way to get up to speed. These can be subscribed at this newsletters link.

The recent news on Fusion Middleware is "Oracle to Hit 100 Cities in the Oracle Fusion Middleware Forum World Tour" which is free for all to attend! The latest newsletter has details.

Also check out Best practice centers

Web Service Manager (WSM) product provides security and management capabilities for SOA-enabled applications.
In our case WSM is used to provide Web Services (WS) security by applying policies to each WS and by location transparency. Location transparency means that our WS consumers need not know where the providers are; they just need to know that the services exist,their purpose & location URL. Consumers look to our UDDI registry to get this kind of information.

Key component of WSM is the Application server which it gets deployed on, which was an OC4J server for us. See my previous post on Java tuning which gives you the process by which you go about tuning the server. Here, i'll describe the problem we faced in WSM performance and its resolution.

Problem: WSM's JVM was doing frequent GCs consistenly even during idle times. The GC counts were very high. During load these become worse making the JVM spending much time just working on GC. This causes WSM to perform sluggishly under high loads.

As the image (WSM Problem.jpg) at end of the blog shows The garbase collections were frequent and were more frequent upon load after 22:03
For the duration of the run, the number of Full aka MarkSweep collections were huge 2487 & for minor it were 4289 collections.

Monitoring: We enabled JConsole, which showed us problem happening in real time.
Enable gc log via these JVM parameters:

-verbose:gc
-Xloggc:/directory/ourlogfile.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime

Resolution:
The fix was to introduce -XX:+DisableExplicitGC parameter. Which basically means that disable any explicit GC calls happening in your code. After this was put in the Major collections were reduced to 0,they were not occuring that frequently ,minor collections were reduced drastically as well. we were able to utilize the full heap of JVM which was allocated.

As part of development System.gc() calls should NOT be placed in your code unless absolutely neccessary. This interferes with Java's GC mechanisms which is not necessarily good for your application as found here. If this call is placed, don't do it in loops which occur regularly, this should be one time event maybe after large number of objects were release etc.

Please let me know if any comments.

Recently I was involved in Performance, Scalablity & Reliablity (PSR) Testing of our metalink3 customer support portal. Since most of the technology components are based on Oracle's Application Server relying on OC4J what it came down to was how can we tune OC4J for best performance.

Metalink3 technical components I was involved in tuning were Secure Enterprise Search (SES) and Web Service Manager (WSM) .Both of products use OC4J which runs on Java.

Tuning is a excercise which has to be attacked patiently. I've been doing performance tuning for variety of Oracle products for a long time and one thing I learned is you have to figure out where the problem lies in order to solve it.

Here was the problem our PSR team was facing when i was brought in: After scaling to a particular number of concurrent users the application server would stop responding. They were trying to get to number of users much more than where they were crashing.

As part of the first exercise we enabled JConsole tool on the servers. This is critical to tuning, you ought to have tools to tell you what's going on in the system, otherwise its like shooting in the dark. Oracle has another tool Application diagnostic for java (AD4J) which we're setting to monitor JVM and java objects.

JConsole provides visual view of what's happening within your JVM and comes out of box with JDK 1.5. Important things we were looking for were GC times, threads, and individual Heap pools. After JConsole was set we then ran mini test runs with varying parameters. This showed us the problem shaping up in real-time.

JVM's GC and its memory parameters were tuned based on this info. Here you have to be careful as you have to understand the Operating system and Hardware you're working against,and you have to size parameters appropriately for your system.

After JVM we attacked OC4Js parameters like thread-pool ,http connection parameters, database pool andstatement caching. These were tuned for our needs as well.

Once we were done with final settings we let our PSR environment stress test the application for many hours. The JVMs were performing beautifully.

Here are some steps on how to do this kind of tuning on your OC4J or for that matter any Java based server like Weblogic, Tomcat etc.

a) Gather information on your environment
-Java parameter settings like -Xmx etc.
-OS level - ulimit, memory, OS, processor type, 32 vs 64 bit, shared memory segments, top ,vmstat etc.

b) Based on your analysis of above, determine the Max heap size of your system. Allocate that much memory and set -Xms == - Xmx.

c) Note that JVM 1.4.2 tuning is a lot different than JVM 5.0, consider the ergonomics aspects of JVM 5. Depending on your JVM choose best options.

d) Monitor for performance benefits. - JConsole, or JVMs printouts provide valuable inputs.

Here are some very useful links go get you started:

• Tuning JVM 5 GC


• JVM 5 Ergonomics


• Using JConsole to Monitor Applications


• Java Performance Portal