JVM Tuning for Windchill

JVM Tuning for Windchill

Is Your Windchill Installation Using Memory Efficiently?


Jeff Taylor

Sun Microsystems

Updated: April 13, 2007


Notes

    December 11, 2006: Reduced the recommended Java heap size from 3.6 GB to 3.2 GB to avoid “Out of Memory” errors. This seems counter-intuitive, but the explanation is simple. The Java process needs storage outside of the Java heap. For example, when the JVM attempts create a new thread, thread local storage is allocated from outside of the Java heap. Thus, an excessively large Java heap limits the number of threads that the JVM can allocate, and instead of creating the new thread, the JVM crashes.

    April 13, 2007: (1) Replaced setting Eden in megabytes with -XX:NewRatio=2 (2) added -XX:MaxLiveObjectEvacuationRatio=33 (3) removed explicit setting of MaxTenuringThreshold and (4) added setting for the RMI GC interval.

Introduction

    The World Wide Web is full of articles that cover Java Tuning. With so much information available, it is hard for a Windchill Administrator to know where to start. Which approaches are useful? Which articles and options apply to Windchill? How to get started? What are the right settings for Windchill?

    Why is this so hard? The best Java options depend on the hardware that is used for the Windchill server and the usage patterns of the Windchill users. One of the fundamental questions that needs to be answered by every Windchill administrator is: “Is memory being used efficiently?” Every Windchill installation will need to customize –Xmx, which sets the maximum size of the Java heap. The default value is 64MB1., much too small for Windchill. Unfortunately, there is not one correct answer for every installation. When a well written Java program performs poorly, there are two typical causes. Either the Java heap size is two small causing an excessive amount of garbage collection, or a Java heap size that is so big that portions are paged to virtual memory. Either problem can be severe, so finding a balance is important. In conjunction with setting the right heap size, every administrator will need to set the size of the Eden (young), survivor, and tenured (old) generations2. Also, there are a large number of other Java options, some that help Windchill, some that have minimal impact, and some that only apply to JVM releases that are not supported by Windchill3.

    This article will focus on Java 1.4.2. Windchill 8.0 M020, released in May 2006, was tested with the Sun Microsystems Java Software Developer Kit version 1.4.2_09 for Solaris 9 and 10. The Support Matrix notes that higher versions in the 1.4.2_xx series are also expected to work.


A Reasonable Goal

    Try to keep garbage collection at 5% or less of the JVM’s CPU time. If you can’t accomplish this by adjusting the JVM parameters, consider purchasing additional RAM. The information presented in this article is intended to help you understand how to measure the current status to achieve this goal.


The Three Generations

    The Java heap contains Eden, survivor spaces and the old generation2. There are many other articles available which describe the generations and garbage collection algorithm, so I will only state the following observations:


  • Tenured (old) generation garbage collections use more resources than Eden (young) generation garbage collections. My goal as a tuner is to stop objects from being unnecessarily tenured.

  • The rate at which new Java objects are created can not be controlled by the Windchill administrator, rather it is a function of the Windchill usage pattern and the algorithms implemented by those Java programmers who wrote Windchill.

  • The time interval between young generation collections is simply the size of Eden divided by the rate at which new Java objects are being created. Increasing the size of Eden makes the time interval between new generation collections longer.

  • One path for an object to be tenured is to survive a certain number of young generation garbage collections. Therefore, increasing the time interval between young generation collections implies that an object will be older before being tenured. Therefore, increasing the size of Eden is good.

  • Another path for an object to be tenured is for the survivor space to be too small to contain all of the objects which survive a young generation collection, in which case the survivors spill into the old generation. Therefore, a large survivor space is a good thing.

  • When the old generation is full with acknowledgement of the “young generation guarantee”2, the old generation will be collected. Therefore, a large tenured generation is a good thing.


    So a large Java heap is desirable so long as the data fits into RAM. However, finding a balance in the size of the three generations is a key to successful JVM tuning.


How big can you make the Windchill heap?

    There are at least 3 potential limits to the size of the Java Heap: physical memory, virtual memory and process address space. For a discussion of physical memory and virtual memory, please see the section, “How Big is Too Big?” below. Regarding the process address space, the Windchill program consists of Java byte code that runs correctly on either the 32-bit or 64-bit Java Virtual Machine. The 64-bit JVM has a bigger process address space, but I am not sure that PTC has a clearly stated support policy for the 64-bit JVM.

    With the 32-bit version of Java, the size of the Java heap can almost reach the full 32-bit / 4GB process address space. In addition to the Java heap, other segments need to be mapped into the process address space, such as the executable "text" (machine instructions) and stack. Your can use “pmap –x” to view all of the memory segments of a process. With Windchill, I have found that the largest usable 32-bit heap is close to 3.2 GB.

    You can run Windchill inside the 64-bit JVM on Solaris simply by installing the 64-bit Java supplement and adding the –d64 option to the Java command line. The size of the heap with the 64-bit version of Java can be much larger, however, with Windchill, you may have a better overall user experience by using several 32-bit JVM's (for example, 2 Tomcat JVM's and 4 foreground method servers), rather than two very large JVM's (1 tomcat plus 1 MS). With huge heaps, users will experience long, seemingly random, pauses when the GC finally kicks in.


Enabling GC logging and Example Java Options

    The options shown here would be a good starting point for a new installation. The sizes should be evaluated for your hardware and usage patterns.


Tomcat:

setenv JAVA_OPTS "-server -Xms3200m -Xmx3200m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 -XX:+UseParNewGC -XX:ParallelGCThreads=8 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:PermSize=64m -XX:MaxPermSize=64m -XX:+UseTLAB -XX:+ResizeTLAB -XX:+DisableExplicitGC -XX:+UseMPSS -Dsun.rmi.dgc.client.gcInteraval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/opt/ptc/Windchill_8.0/logs/tomcat_gc.log"


Windchill 8.0:

wt.manager.cmd.MethodServer.java.command=$(wt.java.cmd.quoted) $(wt.manager.cmd.common.java.args) -Djava.protocol.handler.pkgs\\=HTTPClient -DHTTPClient.disableKeepAlives\\=true -DHTTPClient.dontChunkRequests\\=true -Xms3200m -Xmx3200m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 -XX:+UseParNewGC -XX:ParallelGCThreads=8 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+UseTLAB -XX:+ResizeTLAB -XX:+DisableExplicitGC -XX:+UseMPSS -Dsun.rmi.dgc.client.gcInteraval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/opt/ptc/Windchill_8.0/logs/MethodServer_gc.log $(wt.manager.cmd.MethodServer.platform.java.args) wt.method.MethodServerMain


Multiple Method Servers

    If your Windchill Application tier server has more than 8 GB of RAM, you will want to consider using multiple method servers and/or multiple tomcat instances. Using the Windchill parameters above, the multiple JVM’s would write all of the logging data into one file, making analysis very difficult. Here is an example to force each Windchill Method Server to write into a unique log file.

Changes to wt. properties:

wt.manager.cmd.BackgroundMethodServer=/opt/ptc/Windchill_8.0/launch_bms.ksh

wt.manager.cmd.MethodServer=/opt/ptc/Windchill_8.0/launch_ms.ksh

wt.manager.cmd.ServerManager.java.command=$(wt.java.cmd.quoted) $(wt.manager.cmd.common.java.args) -Xms512m -Xmx512m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 -XX:SurvivorRatio=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=8 -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseMPSS -Dsun.rmi.dgc.client.gcInteraval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/opt/ptc/Windchill_8.0/logs/ServerManager_gc.log wt.manager.ServerManagerMain


/opt/ptc/Windchill_8.0/launch_ms.ksh


#!/bin/ksh

WT_HOME=/opt/ptc/Windchill_8.0
WEB_INF=$WT_HOME/codebase/WEB-INF

CLASSPATH=\\
$WT_HOME/codebase:\\
$WEB_INF/lib/activation.jar:\\
$WEB_INF/lib/ie.jar:\\
$WEB_INF/lib/ie3rdpartylibs.jar:\\
$WEB_INF/lib/install.jar:\\
$WEB_INF/lib/mail.jar:\\
$WEB_INF/lib/wc3rdpartylibs.jar:\\
$WEB_INF/lib/wncWeb.jar:\\
$WEB_INF/lib/pdmlWeb.jar:\\
$WEB_INF/lib/pjlWeb.jar:\\
$WT_HOME/lib/servlet.jar:\\
$WT_HOME/lib/CounterPart.jar:\\
$WT_HOME/lib/wnc.jar:\\
$WT_HOME/lib/pdml.jar:\\
$WT_HOME/lib/pjl.jar:\\
$WT_HOME/lib/wnc-test.jar:\\
/usr/j2se/lib/tools.jar


exec /usr/j2se/jre/bin/java \\
-server \\
-classpath $CLASSPATH \\
-noverify \\
-Djava.protocol.handler.pkgs=HTTPClient -DHTTPClient.disableKeepAlives=true -DHTTPClient.dontChunkRequests=true \\
-Xms3200m -Xmx3200m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 \\
-XX:PermSize=96m -XX:MaxPermSize=96m \\
-XX:+UseParNewGC -XX:ParallelGCThreads=8 -XX:SurvivorRatio=4 \\
-XX:TargetSurvivorRatio=90 \\
-XX:+UseTLAB -XX:+ResizeTLAB \\
-XX:+DisableExplicitGC \\
-XX:+UseMPSS \\
-Dsun.rmi.dgc.client.gcInteraval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 \\
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails \\
-Xloggc:$WT_HOME/logs/MethodServer`date +%y%m%d%H%M%S`_gc.log \\
wt.method.MethodServerMain \\
wt.method.log.file=$WT_HOME/logs/MethodServer`date +%y%m%d%H%M%S`.log


/opt/ptc/Windchill_8.0/launch_bms.ksh


#!/bin/ksh

WT_HOME=/opt/ptc/Windchill_8.0
WEB_INF=$WT_HOME/codebase/WEB-INF

CLASSPATH= #Same as /opt/ptc/Windchill_8.0/launch_ms.ksh (removed to save space)

exec /usr/j2se/jre/bin/java \\
-server \\
-classpath $CLASSPATH \\
-noverify \\
-Djava.protocol.handler.pkgs=HTTPClient -DHTTPClient.disableKeepAlives=true -DHTTPClient.dontChunkRequests=true \\
-Xms1000m -Xmx1000m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 \\
-XX:PermSize=96m -XX:MaxPermSize=96m \\
-XX:+UseParNewGC -XX:ParallelGCThreads=8 -XX:SurvivorRatio=4 \\
-XX:TargetSurvivorRatio=90 \\
-XX:+UseTLAB -XX:+ResizeTLAB \\
-XX:+DisableExplicitGC \\
-XX:+UseMPSS \\
-Dsun.rmi.dgc.client.gcInteraval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 \\
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails \\
-Xloggc:$WT_HOME/logs/BackgoundMethodServer`date +%y%m%d%H%M%S`_gc.log \\
wt.method.MethodServerMain \\
wt.method.log.file=$WT_HOME/logs/BackgoundMethodServer`date +%y%m%d%H%M%S`.log \\
wt.method.serviceName=BackgroundMethodServer \\
wt.queue.executeQueues=true wt.queue.queueGroup\\=default \\
wt.adapter.enabled\\=false \\
wt.method.minPort\\=3000


    Using this change to the wt.properties file and the two scripts provided, a unique garbage collection log file will be written for every Windchill Method Server. This is a necessary step in preparing to use more advanced analysis tools.


VisualGC, jstat and PrintGCStats

    Three tools that I like to use to analyze Java performance are VisualGC5, jstat6 and PrintGCStats7. Each tool is useful and provides a unique view of Java’s memory management.

    A good place to start is VisualGC, which provides a graphical display of the Java generations. VisualGC can help a Windchill administrator get an intuitive feel of how memory is being used. Because the graphical display could impact performance, I run jstatd on the Windchill server and VisualGC on a remote Solaris workstation or Windows laptop. The jstatd daemon seems to have minimal impact on Windchill performance. One example of information that be quickly gleaned from VisualGC is by observing the Survivor Age Histogram Window. If survivors are walking the whole way down the histogram and then being promoted to the tenured generation, they may be legitimate long-lived objects. On the other hand, if the objects march very quickly down the histogram, try to increase the size of Eden to slow the speed at which survivors march down the histogram. If survivors are being dumped into the tenured generation before surviving the full MaxTenuringThreshold youngGCs, you should increase the size of the survivor spaces by lowering the SurvivorRatio.

    While VisualGC helps the administrator to understand the behavior of the Windchill method server and the Tomcat JVM’s, it is not a useful basis for comparison. In other words, you can’t put the VisualGC feedback into a spreadsheet. To record and compare behavior, use jstat. Jstat is a tool that is provided with Java 1.5 that can nevertheless be used to monitor the Windchill Java 1.4 JVM’s. Here is an example output (after some clean up with a text editor to align the columns and to annotate the rows):


# pgrep java | xargs -n 1 /usr/jdk/jdk1.5.0_06/bin/jstat -gc

S0C   S1C       S0U     S1U     EC         EU       OC       OU      PC      PU     YGC   YGCT   FGC   FGCT   GCT
273024.0 273024.0    0.0  1751.0 1092352.0  882496.2 2048000.0 805586.6 65536.0 31095.8 115   82.588  2  39.730 122.318 (tomcat)
273024.0 273024.0 1745.9     0.0 1092352.0 1070228.3 2048000.0 923693.3 65536.0 30294.0 138   97.911  4  90.186 188.097 (tomcat)
273024.0 273024.0    0.0 60328.1 1092352.0  892815.4 2048000.0 659197.6 98304.0 62288.3 361  387.152  2  29.291 416.443 (MethodServer)
273024.0 273024.0    0.0 66246.3 1092352.0  622980.6 2048000.0 587980.6 98304.0 61971.7 298  356.208  1  13.550 369.758 (MethodServer)


    The Windchill server is running two Tomcat instances and two Windchill method servers. Jstat displays the capacity and utilization for all of the generations, the number of garbage collections and the time spent in garbage collection. I used PC, the permanent generation capacity, to determine which JVM’s were Tomcat and which were method servers. It has been noted, based on historic PU values (permanent generation utilization) that a 64MB permanent generation is more than sufficient for Tomcat, but sometimes too small for Windchill method servers. If you have more than a handful of Full Garbage Collections (FGC) per hour, it is time to make substantial changes to your configuration. Also, if the total garbage collection time divided by the JVM’s CPU time (ps –ef | grep java) is greater than 5%, you need to modify your configuration.

    Jstat is a tool which can be used by the Windchill administrator to sample the JVM’s. By sample, I mean “a finite part of a statistical population whose properties are studied to gain information about the whole”8. Sampling is an important technique, but some details are smoothed over. Jstat would not be a particularly good tool to answer questions such as “were the user performance complaints that came in after everyone returned from lunch due to excessive garbage collections at this particular time.” Or, “how much space is available after the garbage collection completes?” The final and most detailed level of garbage collection analysis comes from using Java options -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails combined with PrintGCStats to summarize the results. Here is an example of PrintGCStats output:


# /opt/PrintGCStats -v ncpu=8 MethodServer060718120205_gc.log
what      count       total       mean        max  stddev
gen0(s)     206     248.765     1.20760     4.000  0.3632
gen0t(s)    206     248.815     1.20784     4.001  0.3632
GC(s)       206     248.815     1.20784     4.001  0.3632
alloc(MB)   206  219750.464  1066.74982  1066.750  0.0025
promo(MB)   206     645.217     3.13212    53.974  4.2256

alloc/elapsed_time = 219750.464 MB /  4653.224 s = 47.225 MB/s
alloc/tot_cpu_time = 219750.464 MB / 37225.792 s =  5.903 MB/s
alloc/mut_cpu_time = 219750.464 MB / 35235.275 s =  6.237 MB/s
promo/elapsed_time =    645.217 MB /  4653.224 s =  0.139 MB/s
promo/gc0_time     =    645.217 MB /   248.815 s =  2.593 MB/s
gc_seq_load        =   1990.517 s  / 37225.792 s =  5.347%
gc_conc_load       =      0.000 s  / 37225.792 s =  0.000%
gc_tot_load        =   1990.517 s  / 37225.792 s =  5.347%


All three tools are useful and have a unique place in JVM tuning.


How big is too big? (Resident Set Size vs. Virtual Size)

    The summary of the article up to this point is to increase the size of the JVM generations and/or add additional Windchill method servers and Tomcat instances to take advantage of your RAM. This should leave you with questions such as

  • How do you know if the sizes that are currently set are too big?
  • How do you know if the sizes that are currently set are too small?
  • How much RAM is available to allow the JVM sizes to be increased?


    The heap sizes are too big if memory pressure is causing excessive virtual memory activity. A good indicator is the scan rate, displayed by either “sar –g” or vmstat. The scan rate should be at or close to zero. If the scanner kicks in for a short time but returns to zero, virtual memory pressure is not having a significant impact on your performance. If the system is always scanning, you need to kill non-critical processes, reduce the size of your Java heap, or add more RAM to the system.

    The heap sizes are too small and/or the ratios need to be adjusted if you are spending more that 5% of your CPU time in garbage collection. See jstat, above.

    If you want to increase the size of your heaps, you will need to determine how much RAM is available. It is important to differentiate between a process’s working set, resident set size and virtual size. The working set is the set of memory addresses that a program will need to use in the near future9. The resident set size of a process is the size of a process’s memory address space that is currently in RAM. The virtual size is the entire size of the process’s memory including pages that are currently in RAM, pages that the operating system has paged to the disk drive, and addresses that have been allocated but not yet been touched and hence have not yet been mapped. It is common for the resident set size of a Windchill process to be smaller than the virtual size of the process because the process will have pages that have not yet been mapped,

    Conversely, it is a problem if memory pressure has caused the OS to page your Java heap to disk. One tool that can be used to view the resident set size and virtual size is the ps command.

    To view the resident set size and virtual size of your Java processes:

# ps -e -orss,vsz,args | grep java | sort -n

413888 704712 /usr/j2se/jre/bin/java -server -classpath /opt/ptc/Windchill_8.0/codebase:/opt/
605496 1248608 /usr/j2se/jre/bin/java -server -classpath /opt/ptc/Windchill_8.0/codebase:/opt/
2525888 3984944 /usr/j2se/jre/bin/java -server -classpath /opt/ptc/Windchill_8.0/codebase:/opt/
2590776 3993064 /usr/j2se/jre/bin/java -server -classpath /opt/ptc/Windchill_8.0/codebase:/opt/
2636216 3968328 /usr/j2se/bin/java -server -Xms3200m -Xmx3200m -XX:NewSize=1400m -XX:MaxNewSize
2748040 3964296 /usr/j2se/bin/java -server -Xms3200m -Xmx3200m -XX:NewSize=1400m -XX:MaxNewSize


To add the sizes of all of your processes’ resident set sizes (12.3 GB in this example):

# ps -e -orss,vsz,args | awk '{printf( "%d+",$1)}END{print 0}' | bc
12354976


To add the sizes of all of your processes’ virtual sizes (19.3 GB in this example):

# ps -e -orss,vsz,args | awk '{printf( "%d+",$2)}END{print 0}'| bc
19383912

    I should point out that this technique is not 100% accurate. When two or more processes allocate a given shared memory segment or a shared library, the shared memory is added for each process rather than just once. In the case of Java programs with large heaps, this is an insignificant rounding error. On a system that is running Oracle, a different tool needs to be used (maybe pmap –x) because Oracle’s large SGA will be counted many times.

    The example above is taken from a server with 16 GB of RAM. The fact that the virtual size is bigger than RAM should be noted and investigated. Has the working set been paged out? Again, a good indicator is the scan rate, displayed by either “sar –g” or vmstat as discussed above.


    Another view of your system memory is available from mdb –k. Here is an example when there is a lot of free memory:


# mdb -k
Loading modules: [ unix krtld genunix specfs dtrace ufs sd ip sctp usba random fcp fctl nca lofs logindmux ptm md cpc fcip sppp crypto nfs ipc ]

> ::memstat
Page Summary            Pages               MB %Tot
------------ ---------------- ---------------- ----
Kernel                  89123              696  4%
Anon                   589676             4606 28%
Exec and libs            3973               31  0%
Page cache             196119             1532  9%
Free (cachelist)       141024             1101  7%
Free (freelist)       1067444             8339 51%

Total                 2087359            16307
Physical              2054319            16049

    The ::memstat is a heavy weight command that can impact system performance for several minutes. You will want to use it during off-hours.


Conclusion

    A Windchill administrator needs to observe and minimize both Java garbage collection and virtual memory pressure. Java garbage collection should take no more than 5% of the JVM’s CPU cycles. The virtual memory scan rate should remain at zero most of the time. If the administrator can not accomplish both goals on a given server, RAM needs to be added to the server.


References

1) “java - the Java application launcher” (http://java.sun.com/j2se/1.4.2/docs/tooldocs/solaris/java.html)
2) “Tuning Garbage Collection with the 1.4.2 Java Virtual Machine” (http://java.sun.com/docs/hotspot/gc1.4.2/)
3) “A Collection of JVM Options” (http://blogs.sun.com/roller/resources/watt/jvm-options-list.html)
4) “Java Tuning White Paper” (http://java.sun.com/performance/reference/whitepapers/tuning.html)
5) http://java.sun.com/performance/jvmstat/visualgc.html
6) http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
7) “Turbo-charging Java HotSpot Virtual Machine, v1.4.x to Improve the Performance and Scalability of Application Servers” (http://java.sun.com/developer/technicalArticles/Programming/turbo/)
8) http://www.m-w.com/dictionary/sample
9) http://www.memorymanagement.org/glossary/w.html
10) “Java Performance Documentation” (http://java.sun.com/docs/performance/index.html)
11) “jvmstat 3.0” http://java.sun.com/performance/jvmstat/)
12) “GC Portal” (http://java.sun.com/developer/technicalArticles/Programming/GCPortal/)
13) “Java 2 Platform, Standard Edition (J2SE Platform), version 1.4.2” (http://java.sun.com/j2se/1.4.2/1.4.2_whitepaper.html)


Comments:

Extremely helpful JVM tuning guide that otherwise doesn't exist from PTC. I did have a question regarding a presentation you recently put on at the 2008 PTC-User World Conference in Long Beach, CA.

Is it possible to either post that presentation here or forward that presentation to me via e-mail?

I thought your presentation provided a little bit more helpful information on this subject. Specifically surrounding the use of visualgc and jvmstat. I regret not requesting a copy at the conference and I have not been able to find it via the PTC-User Portal.

Posted by Gilbert Espino on July 09, 2008 at 10:55 AM EDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

user12620111

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today