Thursday Nov 28, 2013

Java Mission Control - Production Time Profiling Out of the Box

By Guest blogger: Marcus Hirt

With the release of 7u40, there is a new kid on the JDK tooling block – Java Mission Control. This article will try to explain why it is worthwhile to take a closer look at this technology, as well as provide pointers on how to get started. 


Mission Control is a production time profiling and diagnostics tools suite that originated with the tooling available for the JRockit JVM. 

As part of the development effort of the JRockit JVM we started building tooling for analyzing the runtime performance of the JVM. We needed information about how real production systems were utilizing JRockit. Requests to get customers to lend us their latest top secret trading applications for evaluation in-house were usually, quite understandably, met with telling blank stares calling our sanity into question. It probably wouldn’t have done us much good anyways, since we wanted production data from systems under real loads. Thus we built a tool (the JRockit Runtime Analyzer, which later evolved into the JRockit Flight Recorder) with low enough overhead that we could convince customers to actually use it to collect production time data for us.

Eventually we accidentally solved some customer problems using the tools and customers started asking us if they could license them. The idea was born to spend some more resources on the tooling and make it a commercial tool to pay for its development. JRockit Mission Control was born.

Java Mission Control

After Oracle acquired Sun Microsystems, Oracle suddenly had two of the top three most commonly used general purpose JVMs available on the market. One was the open sourced reference JVM, with a lot of people knowing the source and licensees basing their own ports and versions of JVM off of it. The other JVM, whilst being a quick and pretty little thing, was proprietary with a rather small number of people knowing the code base. Instead of having to support two JVMs, Oracle wanted to pool the available resources into building a best of breed JVM. It was decided that the base would be HotSpot and that the most useful features in JRockit would be ported over – one of them being Mission Control.

In JDK 7u40 the functionality available in HotSpot had reached critical mass, and the very first version of Java Mission Control (unfortunately versioned 5.2.0) was released. It mainly contains equivalents of two of the JRockit Mission Control tools – the JMX Console and the Flight Recorder. There is no on-line heap analyzer yet. There is however a set of quite useful (experimental) plug-ins for JMC, extending Mission Control to do heap dump analysis, targeted analysis for various oracle products or simply extending existing functionality in more (or sometimes less – yes, I’m looking at you Twitter Trigger Action) useful ways .

Getting Started

Starting Java Mission Control is quite easy. Download a recent enough Java 7 JDK (7u40 or later), then simply launch %JDK_HOME%/bin/jmc. The alien thing that now starts is not, as I am sometimes asked, a native application. It’s Java, but it’s built upon Eclipse RCP technology. If you would rather run JMC inside of Eclipse, you can install JMC into your Eclipse from the JMC update site. A word of caution here – because of a bug in Eclipse/SWT, Mission Control performance is horrible in Eclipse 4.x. This bug is slated to be be fixed in Eclipse 4.4, but until then I strongly recommend either using the stand-alone version of JMC, or installing it into an Eclipse 3.8.2. 

The JMX Console

The console in Mission Control can be thought of as a JConsole on steroids. It allows you to monitor JMX data in various ways, to take action when attributes attain certain values and to persist the data and later look at what has been recorded. There are various experimental plug-ins for the console, such as a Coherence plug-in, a plug-in for running JConsole plug-ins and a plug-in for tweeting messages when a action triggers. 

To connect the console to a JVM, simply choose the JVM you want to connect to in the browser tree and select Start JMX Console. If the JVM was started locally or with JDP, then it will automatically appear in the JVM browser. If you have a remote JVM without JDP running, just enable the built in JMXRMI agent as you normally would to be able to connect with JMX clients such as JConsole.

The JMX console is typically used to monitor a small set of critical attributes, such as the CPU load and Java Heap usage, sampled at a relatively low frequency. The console can be configured to take action when undesirable values are reached for an attribute, and one of those actions can be to dump Flight Recorder data. The JMX console also contains special tabs for looking at thread information, such as deadlocked threads, per thread allocation information and per thread profiling information. That said, the JMX console is used for monitoring the runtime. When profiling capabilities or better diagnostic information is needed, the one stop shop is really the Flight Recorder.

The Java Flight Recorder

The Java Flight Recorder can be thought of as of the equivalent to the Data Flight Recorder for an airplane, but for the Java runtime. While it is running It records information about the JVM and its environment. When something “interesting” happens, the data in the Flight Recorder can be dumped, and the information analyzed off-line to gain an understanding of why everything suddenly went from good to “interesting”. Having the Flight Recorder running as an almost unnoticeable impact on the performance of the Java Application running in the JVM. The overhead is usually well below a per cent. This is achieved by having a high performance recording engine built directly into the runtime and collecting data already being tracked by the runtime or as part of an activity where the data is reachable. (As opposed to actively having to do additional work to get the data.) There are a lot of interesting things that can be said about the recording engine implementation, but this being an overview article, I’ll just move on to how to use it, not how it does what it does.

Creating Flight Recordings

The most important difference to how the Flight Recorder worked in JRockit is that in HotSpot two JVM start-up flags must be enabled on the JVM for which you want to do flight recordings:

-XX:+UnlockCommercialFeatures -XX:+FlightRecorder

And that was probably the most important line in this article. 

There are two different types of recordings, and you can have multiple recordings (of different types) running simultaneously:

1. Timed recordings 

These recording run for a pre-configured duration. They are automatically stopped when the time is up. If initiated from JMC, they will be automatically downloaded and opened in the user interface when done. 

2. Continuous recordings 

These recordings have no explicit end time and must be dumped by the end user. 

Now there are three different ways you can do actual recordings, once the parameters are in place:

1. From Mission Control. This is probably the easiest way. Just point and click. 

2. From jcmd. This is a very useful way to control the Flight Recorder from the command line. Quite useful when you can’t access the machine that is running the JVM of interest from Mission Control and you only have access to a shell. 

3. Using command line flags. This is handy when you want to always run with a continuous recording, or when you want to record the start up behaviour of the JVM right from the very start. 

If you want to know more about how to create Flight Recordings, this blog entry is probably a good place to start. 

Analyzing Flight Recordings

There is a lot of useful information in the flight recordings, and there are a lot of different things the information can be used for. For example:

Method profiling. The Flight Recorder will quite happily do method profiling in production systems with a very low overhead. As a matter of fact, it’s even enabled in the continuous template, so go ahead and use it. It will tell you where the hotspots are in your application. In other words, if you have a CPU bound problem, the method profiling information will tell you where to optimize to get things to go faster. 

GC profiling. The GC implementations emit useful events at GC related activity. Information that can be used to check on the live set, semi-refs, GC pauses (and their individual sub-phases) etc. Quite useful for GC tuning, finding out if you’re overusing finalizers and more. 

Allocation profiling. If you do notice a lot of garbage collections, but don’t notice anything strange about the individual GC phases, you may want to kick back a bit on the allocation. Allocation profiling will help you see where all that allocation activity is putting it’s toll on the memory system. 

WebLogic Server analysis. WebLogic is producing its own set of events for the Flight Recorder. They are quite useful in their own right, but can also be good for putting all the other information recorded in a context - “What was really happening during this transaction?”. This article on the Operative Set shows some of the capabilities. 

Latency Profiling. The Flight Recorder has a lot of different events for various thread stalling activities that can occur, such as blocking on monitor enter, parking, waiting etc. I can’t believe I haven’t written a blog post on this yet. Shame on me. This is usually the first place to look if you haven’t got a CPU bound problem, but still performance issues. 

OS information. CPU load, JVM CPU load, environment variables, running processes – there is a lot of operating system information. If you still can’t find what you’re looking for, Mission Control has a D-Trace plug-in for retrieving everything you ever wanted to know, but were too afraid to ask. Note that the overhead from using D-Trace, even with very few probes, is usually more than an order of magnitude worse than just using the Flight Recorder – use with caution. 

There is much more information available from the event providers built into the JVM, such as class loading and compiler events. One way to learn more details about what is available is to take a closer look at the metadata from a recording.


Since JDK 7u40 there is a new tools suite bundled with the JDK – Java Mission Control. The main focus of the suite is on production time profiling and diagnostics. This has the benefit that the data gathered is quite true to the dynamics of the application being profiled, as the observer effect is kept quite low. In other words, instead of profiling the profiler itself, most of the time is actually spent profiling the application and the runtime. Whilst the main focus of Mission Control is production systems, it can be quite useful in development too. It is also free for use in development, as per the standard Oracle Binary Code License (BCL).

This article provided a brief introduction to Java Mission Control.

Further Information

Java Mission Control Homepage

My Blog

@javamissionctrl;  @hirt


Insider News from the Java Team at Oracle!

javeone logo


« November 2013 »