Marcus Hirt's Blog

As most of you probably know by now, JRockit Mission Control, which was introduced with JRockit 5.0 (R26), includes a set of powerful tools to monitor, manage, profile and hunt down memory leaks in your Java applications. JRockit Mission Control is now available for both the JDK 5.0 and 1.4.2 versions of JRockit (since JRockit R26.4 and later).

Two of the things that makes JRockit Mission Control special, are its dynamic nature and low overhead, which makes it suitable even for production type environments. A nice example is a recent entry in Mike Shierberl's Cold Fusion blog. The blog entry says some nice things about JRockit Mission Control and shows how JRockit Mission Control can be put to good use in a Cold Fusion context.

Mission Control 2.0, internally dubbed Energy (since it's mc^2), is going to be released as part of the JRockit R27.1 release, which is currently targeted at mid December. Since December isn't that far off, I thought I'd give you a little sneak peak of what it will look like.

Mission Control 2.0 is:

A set of powerful, low overhead, production and development time tools that can be used to monitor, manage and profile your application. It also includes a powerful tool for hunting down memory leaks.

The new JRockit JVM tools platform based on RCP technology.

Some of the improvements compared to Mission Control 1.0 are:

All the tools are running within the JRockit Mission Control application.

More metrics (including metrics for the Deterministic GC (available with WLRT)).

Auto-discovery of local JVMs + auto launching of the management agent when connecting to them (>=5.0).

Support for in-memory communication with the management agent of locally discovered JVMs (>=5.0).

Support for the JRockit Discovery Protocol for discovering running JRockits on the network.

Support for connecting from Mission Control 2.0 to R27.1 and R26.4 (both 1.4 and 5.0)JRockit JVM's.

Native platform L&F.

Drag and Drop support.

Context sensitive, built in, help.

Possible to manipulate views freely, which for instance makes it easy to compare profiling recordings head to head.

Pluggable architecture.

Here's a screen shot with Mission Control running JRA and Memleak (click to get the full image):

[Dazzling image of JRockit Mission Control lost in migration]

And here's a screen shot with Mission Control running mostly console (click to get the full image):

[Another amazing image of JRockit Mission Control lost in migration]

Our Japanese customers will hopefully be delighted to find that the Mission Control GUI is now localized. Here's a screen shot (click to get the full image):

[Yet another fine image lost in migration]

In the not very distant future (no target date yet) Mission Control will also be offered as a set of eclipse/workshop plug-ins from an update site, so that you always can have the latest Mission Control running from within your development environment.

The upgrade path from Mission Control 1.0 to Mission Control 2.0 is fairly easy; Mission Control 1.0 enterprise licenses are compatible with 2.0 - the upgrade is free!

The next addition to the Mission Control family of tools is already being worked on, and even though I can't reveal any details just yet, I can tell you that the little Java Developer within me is jumping up and down with joy and excitement. ;)

JRockit Mission Control, the low overhead profiling tools suite for JRockit, has gotten a total overhaul and is now built upon Eclipse RCP technology. Mission Control 2.0 contains both a lot of new GUI improvements, as well as more detailed profiling information. It is included with the JRockit R27.1 JDK, which was recently made generally available.

For more information about what's new in Mission Control, please see my previous blog entry - Mission Control 2.0 Sneak Peak

For information on how to obtain a free evaluation/developer license, please see the Mission Control homepage.

The JRockit Memory Leak Detection tool (henceforth abbreviated Memleak) is part of JRockit Mission Control 2.0, the JRockit tools suite. It has served many people quite well in hunting for memory leaks in Java, i.e. hunting for the cause to why objects that shouldn't be in use still have references to them.

How to hunt for these memory leaks using the Memleak tool has been discussed to death, so this blog entry is going to show you other fun filled uses of the tool. ;)

First off we'll explore how Memleak can help us look at how any given type is being used within the JVM.

Start Mission Control and hook it up to a JRockit doing pretty much nothing but sleeping. In my case I'm connecting it to a JRockit started on my local machine.

You may want to pause the updating of the trend table by pressing the pause button in the toolbar above the table, especially if JRockit is very busy.

After pausing the updating of the table, locate a class for which you want to explore the dependencies, say Hashtable entries (java.util.Hashtable$Entry), and select Show Referring Types from the context menues to see what is pointing to that particular type.

You'll find that, not surprisingly, the entries are pointed to by hashtable entry arrays. Click the plus sign to work your way backwards, and you'll find out that, again not very surprisingly, various hashtable clones are pointing to these arrays. Click Hashtable to expand who is using Hashtables. Woooah!? We get a funny arrow directly to Hashtable from our Entry, thus completing a cycle in our graph.

Hmmm. Someone is obviously putting Hashtables into Hashtables. Now, who does that? Let's look at the instances that are taking part in that particular relationship. Bring up the context menu on the java.util.Hashtable$Entry, and select show instances. A dialog will pop up, prompting you to select for what relationship you want to list the instances. We select Hashtable, since we want to find out what entries that are pointing to hashtables. We get a list of entries, in my case 5 candidates.

As we can see, there are four of them keeping alive more than 100 000 bytes of data. Let's pick the top one - bring up the context menu and select to show who's pointing to that particular instance (Show Referring Instances).

Working our way backwards we see that it's the com.sun.jmx.mbeanserver.RepositorySupport holding on to a Hashtable that contains Hastables. We can now select the RepostiorySupport instance and verify that it indeed holds Hastables by bringing up the context menu and selecting "Inspect Instance".

Summary:

In this example we used the Memory Leak Detector in Mission Control for looking at relations between various types and instances in our running JVM. In this particular case we followed up on where people are putting hashtables into hashtables.

The JRockit JVM incorporates technology (the JRockit Deterministic Garbage Collector) which can guarantee an upper bound for worst case pause times for garbage collections. This greatly helps in reducing unexpected latencies. When writing Java applications, however, it is still possible to introduce latencies through non-optimal use of Java Synchronization, I/O and so on. These latencies can cause poor performance and/or response times in your application and can be quite hard to find. The latency analyzer provides a simple way to analyze where such latencies are caused.

The JRockit Latency Analyzer is built into JRA (JRockit Runtime Analyzer) in Mission Control 3.0. When starting a JRA recording it is now possible to select a Latency Recording Profile. 

When such a profile is selected, or Enable Latency Recording is selected from the advanced options, the recorded profile will contain latency information. When a recording contains such information, three additional tabs (Latency Log, Latency Graph, Latency Traces), two views (Event Types, Properties) and a new Latency Perspective will be at your disposal.

The Latency Log tab includes a log which simply lists all the events in a table and lets you filter them. 

The Latency Graph tab includes a graph, showing for each thread which latency events that have happened during the duration of the recording.

The Latency Traces tab includes a tree table view showing you stack traces that have been involved in causing these latencies, and how much latency has been caused by the traces leading to a certain node.

In the example below we have found the response times of our latency sensitive application somewhat disappointing. The CPU isn't saturated, so we've decided to make a JRA recording with latency information in it. We're now viewing the recording in the latency graph (below). At the top of each view is the range selection tool, where you can zoom and pan to filter the events over time. I've zoomed in on an arbitrary part of the recording to show how the range selector looks when zoomed in. This particular recording shows the same problem all over the recording, so panning to another part would yield a very similar image.

From the graph we can see that we seem to do quite a lot Java Synchronization. In the log view (screen shot below) we can look at each of the individual events. In the screen shot I've filtered on the Java Synchronization Events and added the first bunch of them to the Operative Set. They are the ones highlighted in the Range Selector. Seems a good portion of them are indeed the Java Synchronization Events. Another way to select only the Java Synchronization Events would be to deselect the other Event Types in the Event Type view.

If we switch to the trace view, we can see that almost all the latency is caused from calls to a java.util.logging Logger.

Now, if we remove those logging calls, almost no Java Synchronization events remain, and thus the latency caused by those events have been eliminated. See below for a picture of a recording with the logging calls eliminated.

JRockit Mission Control 3.0 was recently made generally available. For more information about the features in Mission Control 3.0, please see the R27.3 release notes and the Mission Control Documentation.

For information about what was introduced with Mission Control 2.0, please see my previous blog entry - Mission Control 2.0 Sneak Peak

One of the parts of Mission Control is the JRA (the JRockit Runtime Analyzer). The JRA is a powerful profiler that can tell you a lot about the performance bottlenecks in your application whilst keeping the overhead barely noticeable.

There are new options added to the JRA recordings almost every release. In recent releases of Mission Control the JRA wizard will check what capabilities the JRockit you are attempting to start a recording on has, and present you with configurable options accordingly. The problem is that the legacy protocol used when connecting to a 1.4 version of JRockit unfortunately does not support querying for the available parameters. Do not despair; there are still means of starting JRA recordings that contain all the new goodies. This is how:

There is a neat little command line tool called jrcmd that can be found in JROCKIT_HOME/bin. The jrcmd tool can be used to invoke commands on a locally running JRockit process. It works like this:

1. Do JROCKIT_HOME/bin/jrcmd to list the JRockit processes on your machine.
2. Pick your favourite one and do JROCKIT_HOME/bin/jrcmd <pid> help
3. Pick the command you wish to know more about, for example JROCKIT_HOME/bin/jrcmd <pid> help startjrarecording
4. Invoke the command with the settings of your choice.

In this case, the command we want to use is startjrarecording, and what we want to know is what parameters we can use on a particular JRockit when issuing the startjrarecording command.

On the windows box I'm currently on, it looks like this:

D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe
2640 com.jrockit.mc.rcp.start.MCMain

5012 jrockit.tools.jrcmd.JrCmd

Notice how it lists itself. In this case I'm interested in knowing the available commands on the JRockit currently running my Mission Control (com.jrockit.mc.rcp.start.MCMain), so I do:

D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 help
2640:
The following commands are available:
        kill_rmp_server
        start_rmp_server
        kill_management_server
        start_management_server
        checkjrarecording
        stopjrarecording
        startjrarecording
        print_object_summary
        memleakserver
        print_class_summary
        print_codegenlist
        run_optfile
        dump_codelayout
        dump_codelist
        dump_codemap
        print_utf8pool
        print_properties
        print_threads
        datadump_request
        runsystemgc
        runfinalization
        heap_diagnostics
        oom_diagnostics
        print_exceptions
        version
        timestamp
        command_line
        memprof
        sanity
        verbosity
        set_filename
        help
        print_memusage
For more information about a specific command use 'help '.
Parameters to commands are optional unless otherwise stated.

To find out what parameters are available to me when starting a jrarecording I do:

D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 help startjrarecording
2640:
Starts a JRA recording.
        filename           - name of the file to store JRA recording to
                             (string, jrarecording.xml)
        recordingtime      - length of the recording in seconds (int, 60)
        delay              - delay before starting recording in seconds (int,
                             0)
        methodsampling     - enable method sampling (bool, true)
        gcsampling         - enable gc information (bool, true)
        heapstats          - include heap statistics (bool, true)
        nativesamples      - include native code in sampling (bool, false)
        methodtraces       - include stack traces (bool, true)
        tracedepth         - depth of stack traces (int, 16)
        sampletime         - time between samples in milliseconds (int, 30)
        zip                - zip the recording (bool, true)
        hwsampling         - use hardware sampling if possible (bool, false)
        threaddump         - do full threaddumps at start and end of recordning
                             (bool, true)
        threaddumpinterval - also do threaddumps every 'n' interval (can be
                             specified as x[ns|ms|s]) (time, 0s)
        latency            - include latency analysis (bool, false)
        latencythreshold   - do not record events shorter than this number (can
                             be specified as x[ns|ms|s]) (time, 20ms)
        cpusamples         - sample cpu usage during the recording (bool,
                             true)
        cpusampleinterval  - cpu sample interval (can be specified as
                             x[ns|ms|s]) (time, 1s)

In this case I found out that I can indeed start up a recordings with new features such as latency analysis data on this particular JVM. To actually start such a recording I could for example write:

D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 startjrarecording laten
cy=true latencythreshold=20ms recordingtime=120s
2640:
JRA recording started.

JRockit Mission Control just got even better! Some improvements will be quite obvious and very visible. Other improvements are subtle usability fixes that simply makes Mission Control more pleasant to use. In this blog I'll briefly list some of the new features, in no particular order, as well as some new freely available services.

Mission Control available as Eclipse Plug-ins

For the first time we're making Mission Control available as a set of Eclipse Plug-ins! This means you will have the power of Mission Control readily available from within your IDE. It also means you will be able to update Mission Control from within the IDE.

The Eclipse integration comes with basic JDT integration, such as the ability to jump to source from stacktraces, classes and methods being displayed in Mission Control.

Running Mission Control within Eclipse requires a 3.3 version of Eclipse or above. Also note that, to get the most out of Mission Control running in Eclipse, you should run Eclipse on JRockit. More on interesting ways of doing that in an upcoming blog. :)

Mission Control Public Bugzilla

There is now a public bugzilla for Mission Control available at bugzilla.bea.com. It is monitored by the Mission Control developers and provides an efficient means for posting change requests and suggestions for enhancements.

Improved GUI

A lot of GUI changes have been made to make Mission Control play nice in an Eclipse environment. The menu layouts have changed. The JRA wizard has been improved greatly. JRA recordings can now be stored in an Eclipse project as a resource and launched by just double clicking (starting the default editor) on the resource. The memory consumption has been lowered and performance improved (yes, we doggy fed Mission Control to itself ;) ). Even more performance improvements will be available in the next major version of Mission Control.

You can now export charts from the Console as images. You can synchronize the charts X-ranges. You can add and remove charts from the Overview in the Console. You can, after freezing the updates of a chart, hover over data points to get exact readings. There are filters available to more easily find specific MBeans. And much, much more.

The Dial concept is back! Some of you may remember them from the old days of the Management Console. They are configurable and can show any attribute you want. And, if you do not like them, you can simply remove them altogether from the overview.

More Data

There is now more data available in Mission Control. In JRA we show the nursery size before and after a young space collection. We can record lazy unlocking profiling data. In the latency recorder we're recording more information in the events (for instance thread transition information for park/unpark).

Fixes

And we have of course fixed problems found, for instance Memleak not properly updating the trend table under certain conditions. Please see the release notes for further information.

Summary

There is a new Mission Control out with new features and better community support. Try it out and let us know what you think!

For more information:

• JRockit
• JRockit Mission Control

The next version of JRockit Mission Control just went live! Feature-wise it is only a minor upgrade, I will go into detail about the most exciting one in a bit, but there are some other changes to this release that are quite profound.

The, in my humble opinion, greatest change is that all JRockit Mission Control features are available by default. There is no need to dabble with licenses. Everything is free to use for evaluation, which among other things means that you can now try out the latency analyzer properly.

JRockit Mission Control has also been relocated to a new homepage at OTN, located at http://www.oracle.com/technology/products/jrockit/missioncontrol/index.html. The next few days a lot of new JRockit Mission Control related material will be posted there.

The update site has moved as well, and can now be found at http://www.oracle.com/technology/software/products/jrockit/missioncontrol/updates/eclipse-3.3/jrmc/.

As you will undoubtedly notice, everything has also been re-branded.

Now, the cool new feature is actually a JRockit JVM side upgrade, that will be automatically picked up by the JRockit Mission Control client. It is now possible to do very cheap memory allocation profiling using the JRockit Latency Analyzer.

There is a new type of latency events available in the latency recordings. They are triggered whenever the JRockit JVM needs to acquire a new Thread Local Area or whenever the JRockit JVM needs to allocate a new large object. These events provides a good approximation to what is causing the most pressure on the memory system. This means that it is quite easy to find out where to start optimizing allocation behaviour using JRMC LAT.

Here is an example:

1. First select the JVM you wish to start a JRA recording on. 
   
   
2. In the JRA recording wizard, ensure that you start a JRA recording with latency information in it.
   You can for example use the Latency Recording Template. 
   
   
3. Then open the Latency Log tab and open a histogram over the events.
   Make the histogram over the Object Allocation - Class Name property by selecting it in the second combo box. 
     
   Note: since the allocation events aren't recognized yet as being from the standard producer, you may need to enable them in the event palette to the left.
   
   As you can see, most of the pressure on the memory system seems to be from the allocations of Strings. This is not uncommon. Together with arrays of primitive types, Strings are quite commonly high ranking here.
   
4. Now right click on the String row in the table, and set the operative set to the events corresponding to that row.
   The selected event property will unfortunately be reset, which will cause the histogram to be reset as well, but never mind that.
   
5. Move over to the Latency Traces view and check that you only want to see the events in the operative set. 
      You can now clearly see exactly where you should focus your efforts to reduce the amount of String allocation to reduce the pressure on the memory system.

Reducing the pressure on the memory system leads to less frequent garbage collections and thus a more responsive system.

For more information on the JRockit JVM R27.6 release, please have a look at the release notes.

This year JRockit Mission Control will be represented at Oracle Open World! I'll have one session and three hands-on labs.

The preliminary schedule is as follows:

Date: 2008-09-21

Hands-On Lab: Diagnostics with Oracle JRockit Mission Control

Session ID: S299540
Track: Oracle Develop: Java
Room: Golden Gate B2
Start Time: 13:15

Date: 2008-09-22

Hands-On Lab: Diagnostics with Oracle JRockit Mission Control

Session ID: S299540
Track: Oracle Develop: Java
Room: Golden Gate B2
Start Time: 16:00

Date: 2008-09-23

Nonintrusive Profiling and Diagnostics with Oracle JRockit Mission Control

Session ID: S299518
Track: Oracle Develop: Java
Room: Nob Hill AB
Start Time: 13:00

Hands-On Lab: Diagnostics with Oracle JRockit Mission Control

Session ID: S299540
Track: Oracle Develop: Java
Room: Golden Gate B2
Start Time: 16:00

I'm looking forward to seeing you there! :)