Web Service Manager (WSM) product provides security and management capabilities for SOA-enabled applications.
In our case WSM is used to provide Web Services (WS) security by applying policies to each WS and by location transparency. Location transparency means that our WS consumers need not know where the providers are; they just need to know that the services exist,their purpose & location URL. Consumers look to our UDDI registry to get this kind of information.
Key component of WSM is the Application server which it gets deployed on, which was an OC4J server for us. See my previous post on Java tuning which gives you the process by which you go about tuning the server. Here, i'll describe the problem we faced in WSM performance and its resolution.
Problem: WSM's JVM was doing frequent GCs consistenly even during idle times. The GC counts were very high. During load these become worse making the JVM spending much time just working on GC. This causes WSM to perform sluggishly under high loads.
As the image (WSM Problem.jpg) at end of the blog shows The garbase collections were frequent and were more frequent upon load after 22:03
For the duration of the run, the number of Full aka MarkSweep collections were huge 2487 & for minor it were 4289 collections.
Monitoring: We enabled JConsole, which showed us problem happening in real time.
Enable gc log via these JVM parameters:
-verbose:gc
-Xloggc:/directory/ourlogfile.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
Resolution:
The fix was to introduce -XX:+DisableExplicitGC parameter. Which basically means that disable any explicit GC calls happening in your code. After this was put in the Major collections were reduced to 0,they were not occuring that frequently ,minor collections were reduced drastically as well. we were able to utilize the full heap of JVM which was allocated.
As part of development System.gc() calls should NOT be placed in your code unless absolutely neccessary. This interferes with Java's GC mechanisms which is not necessarily good for your application as found here. If this call is placed, don't do it in loops which occur regularly, this should be one time event maybe after large number of objects were release etc.
Please let me know if any comments.
