Jeff Taylor's Weblog
-
PTC-Windchill
May 25, 2007
JVM Tuning for Windchill
JVM
Tuning for Windchill
Is
Your Windchill Installation Using Memory Efficiently?
Jeff
Taylor
Sun
Microsystems
Updated: April 13, 2007
Notes
December
11, 2006: Reduced the recommended Java heap size from 3.6 GB to 3.2
GB to avoid “Out of Memory” errors. This seems
counter-intuitive, but the explanation is simple. The Java process
needs storage outside of the Java heap. For example, when the JVM
attempts create a new thread, thread local storage is allocated from
outside of the Java heap. Thus, an excessively large Java heap
limits the number of threads that the JVM can allocate, and instead
of creating the new thread, the JVM crashes.
April
13, 2007: (1) Replaced setting Eden in megabytes with -XX:NewRatio=2
(2) added -XX:MaxLiveObjectEvacuationRatio=33 (3) removed explicit
setting of MaxTenuringThreshold and (4) added setting for the RMI GC
interval.
Introduction
The World Wide Web is full of articles
that cover Java Tuning. With so much information available, it is
hard for a Windchill Administrator to know where to start. Which
approaches are useful? Which articles and options apply to
Windchill? How to get started? What are the right settings for
Windchill?
Why is this so hard? The best Java
options depend on the hardware that is used for the Windchill server
and the usage patterns of the Windchill users. One of the
fundamental questions that needs to be answered by every Windchill
administrator is: “Is memory being used efficiently?” Every
Windchill installation will need to customize –Xmx, which sets the
maximum size of the Java heap. The default value is 64MB1.,
much too small for Windchill. Unfortunately, there is not one
correct answer for every installation. When a well written Java
program performs poorly, there are two typical causes. Either the
Java heap size is two small causing an excessive amount of garbage
collection, or a Java heap size that is so big that portions are
paged to virtual memory. Either problem can be severe, so finding a
balance is important. In conjunction with setting the right heap
size, every administrator will need to set the size of the Eden
(young), survivor, and tenured (old) generations2. Also,
there are a large number of other Java options, some that help
Windchill, some that have minimal impact, and some that only apply to
JVM releases that are not supported by Windchill3.
This article will focus on Java 1.4.2.
Windchill 8.0 M020, released in May 2006, was tested with the Sun
Microsystems Java Software Developer Kit version 1.4.2_09 for Solaris
9 and 10. The Support Matrix notes that higher versions in the
1.4.2_xx series are also expected to work.
A Reasonable Goal
Try to keep garbage collection at 5% or
less of the JVM’s CPU time. If you can’t accomplish this by
adjusting the JVM parameters, consider purchasing additional RAM.
The information presented in this article is intended to help you
understand how to measure the current status to achieve this goal.
The Three
Generations
The Java heap contains Eden, survivor
spaces and the old generation2. There are many other
articles available which describe the generations and garbage
collection algorithm, so I will only state the following
observations:
Tenured
(old) generation garbage collections use more resources than Eden
(young) generation garbage collections. My goal as a tuner is to
stop objects from being unnecessarily tenured.The
rate at which new Java objects are created can not be controlled by
the Windchill administrator, rather it is a function of the
Windchill usage pattern and the algorithms implemented by those Java
programmers who wrote Windchill.The
time interval between young generation collections is simply the
size of Eden divided by the rate at which new Java objects are being
created. Increasing the size of Eden makes the time interval
between new generation collections longer.One
path for an object to be tenured is to survive a certain number of
young generation garbage collections. Therefore, increasing the
time interval between young generation collections implies that an
object will be older before being tenured. Therefore, increasing
the size of Eden is good.Another
path for an object to be tenured is for the survivor space to be too
small to contain all of the objects which survive a young generation
collection, in which case the survivors spill into the old
generation. Therefore, a large survivor space is a good thing.When
the old generation is full with acknowledgement of the “young
generation guarantee”2, the old generation will be
collected. Therefore, a large tenured generation is a good thing.
So a large Java heap is desirable so
long as the data fits into RAM. However, finding a balance in the
size of the three generations is a key to successful JVM tuning.
How big can you
make the Windchill heap?
There are at least 3 potential limits
to the size of the Java Heap: physical memory, virtual memory and
process address space. For a discussion of physical memory and
virtual memory, please see the section, “How Big is Too Big?”
below. Regarding the process address space, the Windchill program
consists of Java byte code that runs correctly on either the 32-bit
or 64-bit Java Virtual Machine. The 64-bit JVM has a bigger process
address space, but I am not sure that PTC has a clearly stated
support policy for the 64-bit JVM.
With the 32-bit version of Java, the
size of the Java heap can almost reach the full 32-bit / 4GB process
address space. In addition to the Java heap, other segments need to
be mapped into the process address space, such as the executable
"text" (machine instructions) and stack. Your can use
“pmap –x” to view all of the memory segments of a process.
With Windchill, I have found that the largest usable 32-bit heap is
close to 3.2 GB.
You can run Windchill inside the 64-bit
JVM on Solaris simply by installing the 64-bit Java supplement and
adding the –d64 option to the Java command line. The size of the
heap with the 64-bit version of Java can be much larger, however,
with Windchill, you may have a better overall user experience by
using several 32-bit JVM's (for example, 2 Tomcat JVM's and 4
foreground method servers), rather than two very large JVM's (1
tomcat plus 1 MS). With huge heaps, users will experience long,
seemingly random, pauses when the GC finally kicks in.
Enabling GC
logging and Example Java Options
The options shown here would be a good
starting point for a new installation. The sizes should be evaluated
for your hardware and usage patterns.
Tomcat:
setenv
JAVA_OPTS "-server -Xms3200m -Xmx3200m -XX:NewRatio=2
-XX:MaxLiveObjectEvacuationRatio=33 -XX:+UseParNewGC
-XX:ParallelGCThreads=8 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:PermSize=64m -XX:MaxPermSize=64m
-XX:+UseTLAB -XX:+ResizeTLAB -XX:+DisableExplicitGC -XX:+UseMPSS
-Dsun.rmi.dgc.client.gcInteraval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 -verbose:gc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-Xloggc:/opt/ptc/Windchill_8.0/logs/tomcat_gc.log"
Windchill 8.0:
wt.manager.cmd.MethodServer.java.command=$(wt.java.cmd.quoted)
$(wt.manager.cmd.common.java.args)
-Djava.protocol.handler.pkgs\\=HTTPClient
-DHTTPClient.disableKeepAlives\\=true
-DHTTPClient.dontChunkRequests\\=true -Xms3200m -Xmx3200m
-XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 -XX:+UseParNewGC
-XX:ParallelGCThreads=8 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:+UseTLAB -XX:+ResizeTLAB
-XX:+DisableExplicitGC -XX:+UseMPSS
-Dsun.rmi.dgc.client.gcInteraval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 -verbose:gc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-Xloggc:/opt/ptc/Windchill_8.0/logs/MethodServer_gc.log
$(wt.manager.cmd.MethodServer.platform.java.args)
wt.method.MethodServerMain
Multiple Method
Servers
If your Windchill Application tier
server has more than 8 GB of RAM, you will want to consider using
multiple method servers and/or multiple tomcat instances. Using the
Windchill parameters above, the multiple JVM’s would write all of
the logging data into one file, making analysis very difficult.
Here is an example to force each Windchill Method Server to write
into a unique log file.
Changes to wt. properties:
wt.manager.cmd.BackgroundMethodServer=/opt/ptc/Windchill_8.0/launch_bms.ksh
wt.manager.cmd.MethodServer=/opt/ptc/Windchill_8.0/launch_ms.ksh
wt.manager.cmd.ServerManager.java.command=$(wt.java.cmd.quoted)
$(wt.manager.cmd.common.java.args) -Xms512m -Xmx512m -XX:NewRatio=2
-XX:MaxLiveObjectEvacuationRatio=33 -XX:SurvivorRatio=8
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=8
-XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseMPSS
-Dsun.rmi.dgc.client.gcInteraval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 -verbose:gc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-Xloggc:/opt/ptc/Windchill_8.0/logs/ServerManager_gc.log
wt.manager.ServerManagerMain
/opt/ptc/Windchill_8.0/launch_ms.ksh
#!/bin/ksh
WT_HOME=/opt/ptc/Windchill_8.0
WEB_INF=$WT_HOME/codebase/WEB-INF
CLASSPATH=\\
$WT_HOME/codebase:\\
$WEB_INF/lib/activation.jar:\\
$WEB_INF/lib/ie.jar:\\
$WEB_INF/lib/ie3rdpartylibs.jar:\\
$WEB_INF/lib/install.jar:\\
$WEB_INF/lib/mail.jar:\\
$WEB_INF/lib/wc3rdpartylibs.jar:\\
$WEB_INF/lib/wncWeb.jar:\\
$WEB_INF/lib/pdmlWeb.jar:\\
$WEB_INF/lib/pjlWeb.jar:\\
$WT_HOME/lib/servlet.jar:\\
$WT_HOME/lib/CounterPart.jar:\\
$WT_HOME/lib/wnc.jar:\\
$WT_HOME/lib/pdml.jar:\\
$WT_HOME/lib/pjl.jar:\\
$WT_HOME/lib/wnc-test.jar:\\
/usr/j2se/lib/tools.jar
exec
/usr/j2se/jre/bin/java \\
-server
\\
-classpath
$CLASSPATH \\
-noverify
\\
-Djava.protocol.handler.pkgs=HTTPClient
-DHTTPClient.disableKeepAlives=true
-DHTTPClient.dontChunkRequests=true \\
-Xms3200m
-Xmx3200m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 \\
-XX:PermSize=96m
-XX:MaxPermSize=96m \\
-XX:+UseParNewGC
-XX:ParallelGCThreads=8 -XX:SurvivorRatio=4 \\
-XX:TargetSurvivorRatio=90
\\
-XX:+UseTLAB
-XX:+ResizeTLAB \\
-XX:+DisableExplicitGC
\\
-XX:+UseMPSS
\\
-Dsun.rmi.dgc.client.gcInteraval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 \\
-verbose:gc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
\\
-Xloggc:$WT_HOME/logs/MethodServer`date +%y%m%d%H%M%S`_gc.log
\\
wt.method.MethodServerMain
\\
wt.method.log.file=$WT_HOME/logs/MethodServer`date
+%y%m%d%H%M%S`.log
/opt/ptc/Windchill_8.0/launch_bms.ksh
#!/bin/ksh
WT_HOME=/opt/ptc/Windchill_8.0
WEB_INF=$WT_HOME/codebase/WEB-INF
CLASSPATH=
#Same as /opt/ptc/Windchill_8.0/launch_ms.ksh (removed to save space)
exec
/usr/j2se/jre/bin/java \\
-server
\\
-classpath
$CLASSPATH \\
-noverify
\\
-Djava.protocol.handler.pkgs=HTTPClient
-DHTTPClient.disableKeepAlives=true
-DHTTPClient.dontChunkRequests=true \\
-Xms1000m
-Xmx1000m -XX:NewRatio=2 -XX:MaxLiveObjectEvacuationRatio=33 \\
-XX:PermSize=96m
-XX:MaxPermSize=96m \\
-XX:+UseParNewGC
-XX:ParallelGCThreads=8 -XX:SurvivorRatio=4 \\
-XX:TargetSurvivorRatio=90
\\
-XX:+UseTLAB
-XX:+ResizeTLAB \\
-XX:+DisableExplicitGC
\\
-XX:+UseMPSS
\\
-Dsun.rmi.dgc.client.gcInteraval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 \\
-verbose:gc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
\\
-Xloggc:$WT_HOME/logs/BackgoundMethodServer`date
+%y%m%d%H%M%S`_gc.log \\
wt.method.MethodServerMain
\\
wt.method.log.file=$WT_HOME/logs/BackgoundMethodServer`date
+%y%m%d%H%M%S`.log \\
wt.method.serviceName=BackgroundMethodServer
\\
wt.queue.executeQueues=true
wt.queue.queueGroup\\=default \\
wt.adapter.enabled\\=false
\\
wt.method.minPort\\=3000
Using this change to the wt.properties
file and the two scripts provided, a unique garbage collection log
file will be written for every Windchill Method Server. This is a
necessary step in preparing to use more advanced analysis tools.
VisualGC, jstat
and PrintGCStats
Three tools that I like to use to
analyze Java performance are VisualGC5, jstat6
and PrintGCStats7. Each tool is useful and provides a
unique view of Java’s memory management.
A good place to start is VisualGC,
which provides a graphical display of the Java generations. VisualGC
can help a Windchill administrator get an intuitive feel of how
memory is being used. Because the graphical display could impact
performance, I run jstatd on the Windchill server and VisualGC on a
remote Solaris workstation or Windows laptop. The jstatd daemon
seems to have minimal impact on Windchill performance. One example
of information that be quickly gleaned from VisualGC is by observing
the Survivor Age Histogram Window. If survivors are walking the
whole way down the histogram and then being promoted to the tenured
generation, they may be legitimate long-lived objects. On the other
hand, if the objects march very quickly down the histogram, try to
increase the size of Eden to slow the speed at which survivors march
down the histogram. If survivors are being dumped into the tenured
generation before surviving the full MaxTenuringThreshold youngGCs,
you should increase the size of the survivor spaces by lowering the
SurvivorRatio.
While VisualGC helps the administrator
to understand the behavior of the Windchill method server and the
Tomcat JVM’s, it is not a useful basis for comparison. In other
words, you can’t put the VisualGC feedback into a spreadsheet. To
record and compare behavior, use jstat. Jstat is a tool that is
provided with Java 1.5 that can nevertheless be used to monitor the
Windchill Java 1.4 JVM’s. Here is an example output (after some
clean up with a text editor to align the columns and to annotate the
rows):
#
pgrep java | xargs -n 1 /usr/jdk/jdk1.5.0_06/bin/jstat -gc
S0C
S1C S0U S1U EC EU OC OU
PC PU YGC YGCT FGC FGCT GCT
273024.0
273024.0 0.0 1751.0 1092352.0 882496.2 2048000.0 805586.6
65536.0 31095.8 115 82.588 2 39.730 122.318 (tomcat)
273024.0
273024.0 1745.9 0.0 1092352.0 1070228.3 2048000.0 923693.3
65536.0 30294.0 138 97.911 4 90.186 188.097 (tomcat)
273024.0
273024.0 0.0 60328.1 1092352.0 892815.4 2048000.0 659197.6
98304.0 62288.3 361 387.152 2 29.291 416.443 (MethodServer)
273024.0
273024.0 0.0 66246.3 1092352.0 622980.6 2048000.0 587980.6
98304.0 61971.7 298 356.208 1 13.550 369.758 (MethodServer)
The Windchill server is running two
Tomcat instances and two Windchill method servers. Jstat displays
the capacity and utilization for all of the generations, the number
of garbage collections and the time spent in garbage collection. I
used PC, the permanent generation capacity, to determine which JVM’s
were Tomcat and which were method servers. It has been noted, based
on historic PU values (permanent generation utilization) that a 64MB
permanent generation is more than sufficient for Tomcat, but
sometimes too small for Windchill method servers. If you have more
than a handful of Full Garbage Collections (FGC) per hour, it is time
to make substantial changes to your configuration. Also, if the
total garbage collection time divided by the JVM’s CPU time (ps –ef
| grep java) is greater than 5%, you need to modify your
configuration.
Jstat is a tool which can be used by
the Windchill administrator to sample the JVM’s. By sample, I mean
“a finite part of a statistical population whose properties are
studied to gain information about the whole”8.
Sampling is an important technique, but some details are smoothed
over. Jstat would not be a particularly good tool to answer
questions such as “were the user performance complaints that came
in after everyone returned from lunch due to excessive garbage
collections at this particular time.” Or, “how much space is
available after the garbage collection completes?” The final and
most detailed level of garbage collection analysis comes from using
Java options -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
combined with PrintGCStats to summarize the results. Here is an
example of PrintGCStats output:
#
/opt/PrintGCStats -v ncpu=8 MethodServer060718120205_gc.log
what
count total mean max stddev
gen0(s)
206 248.765 1.20760 4.000 0.3632
gen0t(s) 206 248.815 1.20784 4.001 0.3632
GC(s)
206 248.815 1.20784 4.001 0.3632
alloc(MB)
206 219750.464 1066.74982 1066.750 0.0025
promo(MB)
206 645.217 3.13212 53.974 4.2256
alloc/elapsed_time
= 219750.464 MB / 4653.224 s = 47.225 MB/s
alloc/tot_cpu_time
= 219750.464 MB / 37225.792 s = 5.903 MB/s
alloc/mut_cpu_time
= 219750.464 MB / 35235.275 s = 6.237 MB/s
promo/elapsed_time
= 645.217 MB / 4653.224 s = 0.139 MB/s
promo/gc0_time
= 645.217 MB / 248.815 s = 2.593 MB/s
gc_seq_load
= 1990.517 s / 37225.792 s = 5.347%
gc_conc_load
= 0.000 s / 37225.792 s = 0.000%
gc_tot_load
= 1990.517 s / 37225.792 s = 5.347%
All three tools are useful and have a
unique place in JVM tuning.
How big is too
big? (Resident Set Size vs. Virtual Size)
The summary of the article up to this
point is to increase the size of the JVM generations and/or add
additional Windchill method servers and Tomcat instances to take
advantage of your RAM. This should leave you with questions such as
- How
do you know if the sizes that are currently set are too big? - How
do you know if the sizes that are currently set are too small? - How
much RAM is available to allow the JVM sizes to be increased?
The heap sizes are too big if memory
pressure is causing excessive virtual memory activity. A good
indicator is the scan rate, displayed by either “sar –g” or
vmstat. The scan rate should be at or close to zero. If the scanner
kicks in for a short time but returns to zero, virtual memory
pressure is not having a significant impact on your performance. If
the system is always scanning, you need to kill non-critical
processes, reduce the size of your Java heap, or add more RAM to the
system.
The heap sizes are too small and/or the
ratios need to be adjusted if you are spending more that 5% of your
CPU time in garbage collection. See jstat, above.
If you want to increase the size of
your heaps, you will need to determine how much RAM is available. It
is important to differentiate between a process’s working set,
resident set size and virtual size. The working set is the set of
memory addresses that a program will need to use in the near future9.
The resident set size of a process is the size of a process’s
memory address space that is currently in RAM. The virtual size is
the entire size of the process’s memory including pages that are
currently in RAM, pages that the operating system has paged to the
disk drive, and addresses that have been allocated but not yet been
touched and hence have not yet been mapped. It is common for the
resident set size of a Windchill process to be smaller than the
virtual size of the process because the process will have pages that
have not yet been mapped,
Conversely, it is a problem if memory
pressure has caused the OS to page your Java heap to disk. One tool
that can be used to view the resident set size and virtual size is
the ps command.
To view the resident set size and
virtual size of your Java processes:
#
ps -e -orss,vsz,args | grep java | sort -n
413888
704712 /usr/j2se/jre/bin/java -server -classpath
/opt/ptc/Windchill_8.0/codebase:/opt/
605496
1248608 /usr/j2se/jre/bin/java -server -classpath
/opt/ptc/Windchill_8.0/codebase:/opt/
2525888
3984944 /usr/j2se/jre/bin/java -server -classpath
/opt/ptc/Windchill_8.0/codebase:/opt/
2590776
3993064 /usr/j2se/jre/bin/java -server -classpath
/opt/ptc/Windchill_8.0/codebase:/opt/
2636216
3968328 /usr/j2se/bin/java -server -Xms3200m -Xmx3200m
-XX:NewSize=1400m -XX:MaxNewSize
2748040
3964296 /usr/j2se/bin/java -server -Xms3200m -Xmx3200m
-XX:NewSize=1400m -XX:MaxNewSize
To add the sizes of all of your
processes’ resident set sizes (12.3 GB in this example):
#
ps -e -orss,vsz,args | awk '{printf( "%d+",$1)}END{print
0}' | bc
12354976
To add the sizes of all of your
processes’ virtual sizes (19.3 GB in this example):
#
ps -e -orss,vsz,args | awk '{printf( "%d+",$2)}END{print
0}'| bc
19383912
I should point out that this technique
is not 100% accurate. When two or more processes allocate a given
shared memory segment or a shared library, the shared memory is added
for each process rather than just once. In the case of Java programs
with large heaps, this is an insignificant rounding error. On a
system that is running Oracle, a different tool needs to be used
(maybe pmap –x) because Oracle’s large SGA will be counted many
times.
The example above is taken from a
server with 16 GB of RAM. The fact that the virtual size is bigger
than RAM should be noted and investigated. Has the working set been
paged out? Again, a good indicator is the scan rate, displayed by
either “sar –g” or vmstat as discussed above.
Another view of your system memory is
available from mdb –k. Here is an example when there is a lot of
free memory:
#
mdb -k
Loading
modules: [ unix krtld genunix specfs dtrace ufs sd ip sctp usba
random fcp fctl nca lofs logindmux ptm md cpc fcip sppp crypto nfs
ipc ]
>
::memstat
Page Summary Pages MB %Tot
------------
---------------- ---------------- ----
Kernel
89123 696 4%
Anon
589676 4606 28%
Exec
and libs 3973 31 0%
Page
cache 196119 1532 9%
Free
(cachelist) 141024 1101 7%
Free
(freelist) 1067444 8339 51%
Total 2087359 16307
Physical
2054319 16049
The ::memstat is a heavy weight command
that can impact system performance for several minutes. You will
want to use it during off-hours.
Conclusion
A Windchill administrator needs to
observe and minimize both Java garbage collection and virtual memory
pressure. Java garbage collection should take no more than 5% of the
JVM’s CPU cycles. The virtual memory scan rate should remain at
zero most of the time. If the administrator can not accomplish both
goals on a given server, RAM needs to be added to the server.
References
1) “java - the Java application
launcher”
(http://java.sun.com/j2se/1.4.2/docs/tooldocs/solaris/java.html)
2) “Tuning
Garbage Collection with the 1.4.2 Java Virtual Machine”
(http://java.sun.com/docs/hotspot/gc1.4.2/)
3) “A Collection of JVM Options”
(http://blogs.sun.com/roller/resources/watt/jvm-options-list.html)
4) “Java Tuning White Paper”
(http://java.sun.com/performance/reference/whitepapers/tuning.html)
5)
http://java.sun.com/performance/jvmstat/visualgc.html
6)
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
7) “Turbo-charging Java HotSpot
Virtual Machine, v1.4.x to Improve the Performance and Scalability of
Application Servers”
(http://java.sun.com/developer/technicalArticles/Programming/turbo/)
8) http://www.m-w.com/dictionary/sample
9)
http://www.memorymanagement.org/glossary/w.html
10) “Java Performance Documentation”
(http://java.sun.com/docs/performance/index.html)
11) “jvmstat 3.0”
http://java.sun.com/performance/jvmstat/)
12) “GC Portal”
(http://java.sun.com/developer/technicalArticles/Programming/GCPortal/)
13) “Java 2 Platform, Standard
Edition (J2SE Platform), version 1.4.2”
(http://java.sun.com/j2se/1.4.2/1.4.2_whitepaper.html)
Extremely helpful JVM tuning guide that otherwise doesn't exist from PTC. I did have a question regarding a presentation you recently put on at the 2008 PTC-User World Conference in Long Beach, CA.
Is it possible to either post that presentation here or forward that presentation to me via e-mail?
I thought your presentation provided a little bit more helpful information on this subject. Specifically surrounding the use of visualgc and jvmstat. I regret not requesting a copy at the conference and I have not been able to find it via the PTC-User Portal.