The JRockit Blog

Welcome to my new home at Oracle! It replaces my old BEA blog, and will - as before - focus mainly on JRockit-related topics. I will start by reposting my most frequently read blog posts from dev2dev and then go where fancy takes me. If you have a request, post it here or in one of the JRockit forums on forums.oracle.com. Well met!

Originally posted 8/14 2006 on http://dev2dev.bea.com/blogs/hstahl, reposted on request.

We sometimes get bug reports from users who are having issues with JRockit on a Linux distribution that we do not support officially, such as Debian, Ubuntu or Fedora Core. Sometimes - in particular when JRockit hangs intermittently - the cause is a broken OS. Here are two things you can do to check your Linux installation.

1. Verify that you are using the NPTL threading library

Before the 2.6 kernel, most Linux distributions used the old pthreads library. We have seen a number of issues with this over the years. To check your kernel version:

hstahl@sthx6421:~> cat /proc/version
Linux version 2.6.5-7.244-smp (geeko@buildhost) (gcc version 3.3.3 (SuSE Linux)) #1 SMP Mon Dec 12 18:32:25 UTC 2005

In this case, the kernel is 2.6.5. Anything based on 2.6 or later uses NPTL (plus the 2.4-based kernel in Red Hat EL 3.0, but that's a special case)

Also, make sure that you are not setting the LD_ASSUME_KERNEL environment variable!

2. Verify that your distribution handles signals correctly

JRockit uses OS signals to suspend and resume Java threads. Unfortunately, this sometimes exposes bugs in the underlying OS, causing the JRockit process to hang (or crash).

To check your Linux installation; download this small C program, compile and run it on your computer following the instructions in the source code. If the program hangs or crashes, then you most likely have a broken kernel and/or glibc. Try updating your OS to a later build/patch level/service pack and rerun. If you still have an issue, please report it to the appropriate vendor and/or community.

If the test program fails, then you will see intermittent JRockit hangs and/or crashes on your platform. It may take days or weeks, but it will happen eventually.

Note that we have for the past two years or so run this program or similar tests on all OSes we support, and we always make sure such OS bugs are fixed before we add it to our list of supported configurations, so if you are using one of those then you should be safe.

Originally posted in December 2005 on http://dev2dev.bea.com/blogs/hstahl, reposted on request with a few minor updates. I find it amazing that the very useful "non-contiguous heap" feature described here is still unique to JRockit after 2.5 years...

As you may be aware, the maximum heap size on Window using JRockit - or for that matter any JVM we are aware of - has been limited to slightly below 2 GB. There have been two reasons for this. One is that the maximum process size on Windows has been limited to 2 GB, though this can be worked around by using the /3GB kernel switch. The second is that JVMs have required a contiguous memory space for the Java heap for efficiency reasons, which causes the maximum Java heap size to be limited by DLLs loaded into the process address space.

For more information and previous discussion on this topic see:
Microsoft PAE & /3GB docs
JRockit docs
Sun blog

With the soon to be released JRockit 5.0 R26 (update: released in 2006) this barrier is broken! By introducing support for split heaps, we have been able to significantly increase the Java heap size on Windows.

Windows 2003/XP or later using the /3GB switch (32-bit OS)
1.85 GB - JRockit versions prior to 2006
2.85 GB - JRockit versions post 2006

Windows 2003/XP x64 Edition with a 32-bit JVM (64-bit OS)
2.05 GB - JRockit versions prior to 2006
3.85 GB - JRockit versions post 2006

How do I enable this feature?
32-bit Windows
1. Add the /3GB switch to your boot.ini file as described here
2. Install and run a recent version of JRockit specifying a large -Xmx

64-bit Windows
1. Install and run the 32-bit version of JRockit specifying a large -Xmx

How large is the performance overhead?
Zero!

What if I need an even larger heap?
Use one of the many 64-bit version of JRockit.

Are there any drawbacks?
You still need to make sure there is enough memory for JVM internals, compiled code and any native libraries you will be using. If JRockit exits with an out of memory error in native code, try decreasing the heap size slightly.

Originally posted 3/28 2007 on http://dev2dev.bea.com/blogs/hstahl. We haven't run on a bigger machine since, and I haven't seen anyone else claim the crown so I guess this "record" still stands.

I recently came across this old thread on TSS and just had to dig up an old log file I had lying around. I originally intended to post this last fall, but got sidetracked and forgot. Anyway - better late than never.

/opt/jrockit/bin/java -Xms3500g -Xmx3500g -XXgcthreads=1024 -XXlargepages ...
[memdbg ] Large pages enabled.
[memdbg ] Large pages for code enabled.
...
[memory ] GC strategy: parallel
[memory ] using large pages - maximal oldspace size commited at startup
[memory ] heap size: 3670016000K, maximal heap size: 3670016000K
...
[memdbg ] old collection 4 started
...
[gcpause] total mark time: 28786.228 ms
[memdbg ] ending marking phase
[memdbg ] starting parallel sweeping phase
...
[gcpause] total sweep time: 2380.615 ms
[memdbg ] ending sweeping phase
[gcpause] old collection phase 0 pause time: 31220.365000 ms, (start time: 3934.166 s)
[gcpause] (pause includes compaction: 1864.060 ms (internal), update ref: 657.861 ms)
[memdbg ] Page faults before GC: 4, page faults after GC: 4, pages in heap: 229376000

That ~30s pause time doesn't look so impressive until you realize that the heap size is not 3.5 gigabytes, but 3.5 TERABYTES!!!

Yes, we actually ran JRockit on an SGI Altix server with 512 dual-core Itanium CPUs using 1024 threads for parallel GC and 3.5 TB of heap - all in physical RAM. And this with a standard JRockit version, not some special prototype. So if you ever wondered how far you could scale your JRockit, now you know :-)

I have to admit that JRockit did not scale perfectly up to this size, but then this is hardly a typical configuration and you should be able to get reasonable performance and lower pause times by using our generational mostly-concurrent GC and doing some other command line tweaking.

Originally posted in December 2007, reposted on request. The problem described here is generic and applies to all JVMs using mark-n-sweep or similar GC algorithms.

JRockit Mission Control is now free for development and evaluation, you can download it here.

One major cause for long GC pause times is heap fragmentation. How problematic this for an application depends on its allocation pattern. The worst possible case is an application that allocates a mix of objects with very different sizes and lifetimes. After the application has been running for a while, the Java heap will be fragmented by lots of long lived Java objects spread out across the heap. There may be plenty of free space available, but no large block of contiguous free memory. When the application then attempts to allocate a large object such as an array, it is unable to find room to store it. The result will be a long GC pause while the heap is compacted.

If you suspect you have this issue with your application, the first step is to try to find out exactly how fragmented the heap is. One simple way of doing this is by using the JRockit Runtime Analyzer. Here's an example:

1. Start a Java application on your workstation
2. Start up JRockit Mission Control using the Start menu icon in Windows or the $JR_HOME/bin/jrmc executable
3. Locate the process you want to analyze, right-click and select to start a JRA recording.

4. Select recording time (here 2 minutes) and start the recording

5. After the recording has finished, it will be opened in the JRMC GUI. Select the Heap tab. Heap fragmentation is displayed in black in the Heap Contents pie chart.

The application in this example is well behaved and shows only 11% fragmentation - well within acceptable limits. I would start getting concerned if it went above 30%, and more if it continued to increase. Another warning sign is if the free memory distribution (pie chart on the right) contains a very large proportion of smaller free blocks.

Originally posted in December 2007, reposted on request. The problem described here is generic and applies to all JVMs using mark-n-sweep or similar GC algorithms.

Heap fragmentation occurs when a Java application allocates a mix of small and large objects that have different life times. The main negative effect of fragmentation is that long GC pause times caused by the JVM being forced to compact the Java heap. These long pause times are typically triggered when your Java program attempts to allocate a large object, such as an array.

As described in my previous blog entry on this topic, you can use JRockit Mission Control to find out how fragmented the heap is. But a fragmented heap is only a problem if it leads to long pause times (or an OutOfMemoryError). To find out the impact on pause times, you can run with the -Xverbose:gcpause command line flag, which will give you something like:

[INFO ][gcpause] old collection phase 1-0 pause time: 73.214054 ms, (start time: 15.807 s)
[INFO ][gcpause] (pause includes compaction: 1.029 ms (external), update ref: 1.532 ms)
[INFO ][gcpause] Threads waited for memory 66.612 ms starting at 17.568 s
[INFO ][gcpause] old collection phase 1-0 pause time: 66.449507 ms, (start time: 17.569 s)
[INFO ][gcpause] (pause includes compaction: 1.236 ms (internal), update ref: 1.488 ms)

The pauses in this example are clearly not a problem, but it can sometimes be much longer than this.

If you don't want to restart your JVM, you can enable this during runtime by running jrmcd <pid> verbosity set=gcpause=info. After you have the data you need, disable informational logs with jrcmd <pid> verbosity set=gcpause=error.

Or you can do a JRA recording (see the previous blog entry) and look in the GC details tab, where the time spent in compaction is clearly visible:

Before we look into the possible strategies for dealing with fragmentation, it is crucial to understand what causes it. The first key observation is that fragmentation is caused by GC. When the JVM performs GC it will clear out dead objects. It's the act of removing these dead objects that creates the holes in the heap. Memory allocation only has an indirect impact, in that it can create a pattern in the heap that later leads to the GC fragmenting the heap.

A second key observation is that fragmentation is only a problem if you can't use the holes in the heap. As long as you only allocate small objects, it doesn't matter how fragmented the heap is.

With these two observations in mind, here are some tips:

1. Increase the heap size

Increasing the heap size will decrease the frequency of GCs. One benefit of this is that objects are more likely to be dead than if GCs are very frequent, and if more objects are dead then there will be fewer live objects around that can contribute to create holes. In other words, the heap holes will on average be larger, which implies less fragmentation. Also, if GCs are less frequent, you can possible afford longer pause times since the impact of GC on throughput will be lower. Be aware that increasing the heap size can cause a slight increase in pause times.

A special case is to run with an infinitely large heap and never GC, which will of course avoid fragmentation completely.

2. Use a generational GC

Running JRockit with -Xgc:gencon or -Xgc:genpar will enable the use of a nursery or young space. The nursery will store recently allocated objects and when it is full a nursery GC will be performed, in which objects that are still alive will be moved to the old space. Since all objects that survive are (eventually) moved to the old space, the nursery will never be fragmented. And fragmentation of the old space will happen much more slowly since objects moved there will on average survive for a long time. Also, since the old space will fill up less rapidly, fragmentation-causing old space GCs will be less frequent.

A common strategy for avoiding the cost of old space GCs (often called "full GCs") is to configure your nursery size so that almost all objects die before they reach the old space. If you do this carefully, you can postpone old space GCs for a very long time. I've seen some installations where the app has been configured to avoid old space GCs for a full day, after which it is restarted to force creation of a "clean" heap. This may be more frequent among non-JRockit users, since our compaction is fairly efficent and the pause times tend to be acceptable. One word of warning here: This strategy is not guaranteed to avoid full GCs, since that depends on the load on your application, exact heap layout etc, so don't rely on it too much and configure it with a large safety margin.

3. Tune compaction parameters

The default behavior of JRockit is to analyze the fragmentation of the heap and do a little bit of compaction every old space GC cycle. The proportion of the heap that it decides to defragment is called the compaction ratio which is typically stated as a percentage of the heap size, where a common figure would be perhaps 5%. If your application causes a lot of fragmentation, you can configure this ratio manually which gives you the ability to create a balance of power between the GC and your memory-hungry Java program. You can try with -XXcompactratio=10 or so to start with. A high number will lead to longer pause times, but also means that the JVM will be able to cope with higher fragmentation.

If you want to do advanced tuning, look in the JRockit reference guide for parameters that impact compaction. Two examples are -XXinternalcompactratio and -XXexternalcompaction.

4. Don't allocate memory

Ok, so it's time to look at what you can do in your Java code. The most obvious tip is avoid memory allocation. This will have direct impact on both the frequency of GCs and indirectly on the GC pause time, since less objects will be alive at the time of a GC. You can use the Memory Leak Detector to analyze your application's allocation pattern and trace down excessive allocation to where in your source code it occurs. See Marcus Hirt's blog entry on this subject for tips.

5. Avoid allocating large objects

Arrays and other large objects are always the biggest culprit when it comes to fragmentation. They cause the heap to fill up quickly, leading to frequent GCs, they create irregular patterns in the heap and they request big contiguous chunks of memory on the heap at allocation time, which can be impossible for the JVM to fulfill without first doing compaction. To avoid excessive allocation of large objects, think twice before copying arrays etc. All code involving string processing (char arrays), XML, I/O (byte arrays) etc is a target for optimization in this area. Again, the Memory Leak Detector is a very powerful tool for analyzing this.

6. Allocate objects with similar lifespan in chunks

This is my last tip, and it is a bit esoteric so please bear with me... The idea is that any larger operation can decrease its impact on fragmentation by allocating the memory it needs in a chunk and then keep it alive until the operation is complete. Consider a J2EE transaction such as a servlet request. When you start this request, you can have one metaobject created that allcates most or all of the objects that you need to process that particular request. Since these allocations will all be performed by a single thread and very closely spaced in time, they will typically end up stored as a contiguous block in the Java heap. If you keep all these objects alive until the transaction are done, they will all become subject for GC at the same time, and cleared as dead objects during the same GC cycle. Ergo, less fragmentation. So nulling out objects prematurely to decrease live data may not be the best in the long run.

Final words

That's it for this time... I hope you found this useful! Don't hestitate to ask if you found something unclear. Keep up the good coding!