Wednesday Jul 30, 2014

Deep monitoring with JMX

The Java Platform is designed as a modular system, where each item in the conceptual diagram provides specific functionality. One commonly requested feature of software platforms is the ability to monitor an application for CPU, memory and resource usage, and other statistics. The Java Platform Standard Edition (Java SE) has provided the Java Management eXtension (JMX) since Java SE 5.0 (2004). There are several benefits to having this type of monitoring as part of the platform:

  1. Unlike operating system tools (e.g. htop or Task Manager), the details are specific to the application. Instead of just overall CPU and memory usage, it is possible to see what part of the application is consuming those resources.
  2. Applications can surface their own business-related Management Bean to the JMX rather than building a new monitoring user interface.
  3. The monitoring agent is already available inside the runtime. There is no need to embed additional items in your application.
  4. JMX connections take place at the JVM level rather than inside the application. Monitoring information is available either locally or remotely. By using different firewall rules for JMX than your application, you can scope monitoring to a smaller audience.
  5. System administrators can institute direct connections rather than periodic polling, for faster alerts or reaction time.

This entry will cover the role of JMX and the ways in which it is designed to help developers and system administrators.

JMX Basics: an agent and a viewer

JMX capabilities are split into two parts:

  1. The JMX agent that serves information is available within the runtime. This runs alongside your application.
  2. A JMX viewer, like jConsole or Java Mission Control. These viewers use a local or remote connection to read information from the running agent. Other third party viewers will be covered later.

By default, jConsole shows information about memory usage, CPU usage, thread usage, and class information. Unlike standard operating system tools like htop or Task Manager, the information is at a deeper level inside the JVM and can also seen on authenticated remote systems. By default JMX allows local connections to the process owner -- instead of username/password, you are authenticated by NTFS permissions (on Windows).

screenshot of jconsole

In this screenshot of one small Java application I was running, you can clearly see the periodic small garbage collections. The minor incremental collection has significantly improved since the late 1990s.

The separate MBeans tab provides ways to show additional diagnostics as well as additional information. When using third party Java applications, I often launch them through small scripts and do not know the complete invocation information. The com.sun.managementDiagnosticCommand lets me click a "vmCommandLine" button and see what the launch script actually did.

screenshot of jconsole

In many situations, such as application servers, the application may be running on an entirely different host.

Authenticated remote monitoring for System Administrators

Any remote monitoring system must come with proper access controls built in. The remote monitoring capabilities in JMX allow for clients to monitor information happening on servers or other devices without having any user interface on those systems. Applications typically turn on remote JMX capabilities either through launch scripts setting:

-Dcom.sun.management.jmxremote.port=portNum
-Djava.rmi.server.hostname=OptionalHostnameConnectionRestriction

When specifying the port number above, it is an important distinction that JMX information is served separately from the application. For example a web application might serve requests on ports 80 and 443 yet allow JMX connections on a separate port with more restrictive firewall rules. Normal users would be unable to even attempt JMX access.

After configuring a firewall rule, it is critical to authenticate users against one of the three main methods:

  1. File-based passwords are probably the simplest to set up but require a shared secret and are sometimes harder to change. Your JRE already has an example configured file at JRE/lib/management/ jmxremote.password.template.
  2. LDAP for enterprises. The benefit of LDAP integration is group authentication, as well as automatic integration with joiner/leavers processes.
  3. SSL certificates for password-less connections. This allows client authentication based on public keys in a way that no passwords are ever exchanged.

The JMX Agent user guide contains details on all other JMX configuration parameters. These capabilities can also integrate JMX with SNMP tools. For other protocols, consider using JMXMP through OpenDMK.

Do not turn security off

Within the JMX documentation, there are options listed for turning security off so that no authentication and authorization is performed on connections. Do not use that configuration unless you are temporarily testing a different tool and just wondering if authentication is causing a problem. Once you ascertain yes or no, you should stop turning authentication off.

Third party monitoring example: Nagios

JMX provides a level of application introspection that provides system administrators with a way to look inside the application and provide some level of diagnostics that are beneficial to the development teams. It can also be integrated into systems like Nagios’ JMX monitor and extends alert and monitoring capabilities to act on the extra details.

Exposing custom metrics to JMX for Developers

Items within the Java platform are designed for extensibility. While all applications look for details about resource usage, other information can be beneficial. The Java tutorial on JMX provides details on how to create new MBeans that would expose custom information to jConsole.

There are a variety of cases where a developer may want to create a new MBean. For example:

  •  A batch file processing task may count the number of files it has processed. Instead of printing to a lengthy console, the developer could monitor progress and leave the console for important information.
  • An application may count the number of successful logins per day, failed logins, or a ratio between the two. They could then monitor the system for odd behavior such as unexpected high or low usage.
  • For various use cases, the application could track how many times that use case has happened. This is similar to the login tracking above, but for different features. This would be a simple way of tracking popularity of use cases in relation to each-other.
  • Any other information that is beneficial for tracking, charting, or alerting.

API documentation is available within the java.lang.management package.

Sample code for creating MBeans and interacting with JMX within the Java SE Downloads. Download the "Demos and Samples" project and open the sample/jmx folder.

Open-Source Example: Apache QPid

Apache QPid is a message broker for reliable inter-application communication. During operation, QPid uses JMX to expose a helpful information and statistics about the runtime. A system administrator can analyze throughput, message queue statistics, or other things that would be unavailable to other monitoring tools.

The JMX Management section of the online QPid Message Broker book explains how their system can be monitored. For developers looking to create MBeans, that chapter also links directly to their open-source code for more examples.

JMX on Embedded Devices

Java SE Embedded provides JMX within its Compact 3 profile. The functionality is the same as what you would encounter on non-embedded devices.

For additional details on Compact Profiles in the embedded space, see "Compact Profiles: Space and Security" or the jrecreate command.

Advanced Monitoring

The standard jConsole system is beneficial for monitoring, but many organizations have different needs. There are different application servers, clusters, different types of systems, embedded devices, and many other cases. The Java Platform’s open standards have provided a stable foundation for a number of advanced monitoring systems. Some monitoring systems use JMX, others use the JVM TI.

Oracle provides a commercial monitoring system named Mission Control and Flight Recorder that is able to surface an application’s statistics and take automatic action. Flight Recorder has been available in all JREs since Java 7 update 2 (April 2012).

Unlike many monitoring systems that begin recording on a trigger, flight recorder is a minimal-overhead black-box that continually records. When your trigger event occurs, it can provide details about what went on before that event happened. As a result, developers can track down bugs that cannot be replicated outside of a production environment.

 

Marcus Hirt has done a video introduction to Java Mission Control that explains many details and features, such as creating recordings, setting triggers, and controlling remote operations. The video covers the current version of Mission Control (5.3) that took place after HotSpot and JRockit were merged into a single VM.

JMX Usage

The built-in monitoring capabilities of the Java platform provide developers and system administrators with ways to understand their applications without building a new interface. Developers are able to create custom MBeans that expose application-level information to any JMX browser such as jConsole or Mission Control. Many monitoring systems use JMX as a way of understanding internal application health.

For a more complete description on the role of JMX and examples of how to configure it, react to events, or build new MBeans, please see the Java Management Extensions (JMX) Technology Tutorial or Monitoring and Management for the Java Platform.

About

Science Duke
This blog contains topics related to Java SE, Java Security and Usability. The target audience is developers, sysadmins and architects that build, deploy and manage Java applications. Contributions come from the Java SE Product Management team.

Search

Categories
Archives
« August 2015
SunMonTueWedThuFriSat
      
1
2
3
4
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
     
Today