Wednesday Jul 30, 2014

Deep monitoring with JMX

The Java Platform is designed as a modular system, where each item in the conceptual diagram provides specific functionality. One commonly requested feature of software platforms is the ability to monitor an application for CPU, memory and resource usage, and other statistics. The Java Platform Standard Edition (Java SE) has provided the Java Management eXtension (JMX) since Java SE 5.0 (2004). There are several benefits to having this type of monitoring as part of the platform:

  1. Unlike operating system tools (e.g. htop or Task Manager), the details are specific to the application. Instead of just overall CPU and memory usage, it is possible to see what part of the application is consuming those resources.
  2. Applications can surface their own business-related Management Bean to the JMX rather than building a new monitoring user interface.
  3. The monitoring agent is already available inside the runtime. There is no need to embed additional items in your application.
  4. JMX connections take place at the JVM level rather than inside the application. Monitoring information is available either locally or remotely. By using different firewall rules for JMX than your application, you can scope monitoring to a smaller audience.
  5. System administrators can institute direct connections rather than periodic polling, for faster alerts or reaction time.

This entry will cover the role of JMX and the ways in which it is designed to help developers and system administrators.

JMX Basics: an agent and a viewer

JMX capabilities are split into two parts:

  1. The JMX agent that serves information is available within the runtime. This runs alongside your application.
  2. A JMX viewer, like jConsole or Java Mission Control. These viewers use a local or remote connection to read information from the running agent. Other third party viewers will be covered later.

By default, jConsole shows information about memory usage, CPU usage, thread usage, and class information. Unlike standard operating system tools like htop or Task Manager, the information is at a deeper level inside the JVM and can also seen on authenticated remote systems. By default JMX allows local connections to the process owner -- instead of username/password, you are authenticated by NTFS permissions (on Windows).

screenshot of jconsole

In this screenshot of one small Java application I was running, you can clearly see the periodic small garbage collections. The minor incremental collection has significantly improved since the late 1990s.

The separate MBeans tab provides ways to show additional diagnostics as well as additional information. When using third party Java applications, I often launch them through small scripts and do not know the complete invocation information. The com.sun.managementDiagnosticCommand lets me click a "vmCommandLine" button and see what the launch script actually did.

screenshot of jconsole

In many situations, such as application servers, the application may be running on an entirely different host.

Authenticated remote monitoring for System Administrators

Any remote monitoring system must come with proper access controls built in. The remote monitoring capabilities in JMX allow for clients to monitor information happening on servers or other devices without having any user interface on those systems. Applications typically turn on remote JMX capabilities either through launch scripts setting:

-Dcom.sun.management.jmxremote.port=portNum
-Djava.rmi.server.hostname=OptionalHostnameConnectionRestriction

When specifying the port number above, it is an important distinction that JMX information is served separately from the application. For example a web application might serve requests on ports 80 and 443 yet allow JMX connections on a separate port with more restrictive firewall rules. Normal users would be unable to even attempt JMX access.

After configuring a firewall rule, it is critical to authenticate users against one of the three main methods:

  1. File-based passwords are probably the simplest to set up but require a shared secret and are sometimes harder to change. Your JRE already has an example configured file at JRE/lib/management/ jmxremote.password.template.
  2. LDAP for enterprises. The benefit of LDAP integration is group authentication, as well as automatic integration with joiner/leavers processes.
  3. SSL certificates for password-less connections. This allows client authentication based on public keys in a way that no passwords are ever exchanged.

The JMX Agent user guide contains details on all other JMX configuration parameters. These capabilities can also integrate JMX with SNMP tools. For other protocols, consider using JMXMP through OpenDMK.

Do not turn security off

Within the JMX documentation, there are options listed for turning security off so that no authentication and authorization is performed on connections. Do not use that configuration unless you are temporarily testing a different tool and just wondering if authentication is causing a problem. Once you ascertain yes or no, you should stop turning authentication off.

Third party monitoring example: Nagios

JMX provides a level of application introspection that provides system administrators with a way to look inside the application and provide some level of diagnostics that are beneficial to the development teams. It can also be integrated into systems like Nagios’ JMX monitor and extends alert and monitoring capabilities to act on the extra details.

Exposing custom metrics to JMX for Developers

Items within the Java platform are designed for extensibility. While all applications look for details about resource usage, other information can be beneficial. The Java tutorial on JMX provides details on how to create new MBeans that would expose custom information to jConsole.

There are a variety of cases where a developer may want to create a new MBean. For example:

  •  A batch file processing task may count the number of files it has processed. Instead of printing to a lengthy console, the developer could monitor progress and leave the console for important information.
  • An application may count the number of successful logins per day, failed logins, or a ratio between the two. They could then monitor the system for odd behavior such as unexpected high or low usage.
  • For various use cases, the application could track how many times that use case has happened. This is similar to the login tracking above, but for different features. This would be a simple way of tracking popularity of use cases in relation to each-other.
  • Any other information that is beneficial for tracking, charting, or alerting.

API documentation is available within the java.lang.management package.

Sample code for creating MBeans and interacting with JMX within the Java SE Downloads. Download the "Demos and Samples" project and open the sample/jmx folder.

Open-Source Example: Apache QPid

Apache QPid is a message broker for reliable inter-application communication. During operation, QPid uses JMX to expose a helpful information and statistics about the runtime. A system administrator can analyze throughput, message queue statistics, or other things that would be unavailable to other monitoring tools.

The JMX Management section of the online QPid Message Broker book explains how their system can be monitored. For developers looking to create MBeans, that chapter also links directly to their open-source code for more examples.

JMX on Embedded Devices

Java SE Embedded provides JMX within its Compact 3 profile. The functionality is the same as what you would encounter on non-embedded devices.

For additional details on Compact Profiles in the embedded space, see "Compact Profiles: Space and Security" or the jrecreate command.

Advanced Monitoring

The standard jConsole system is beneficial for monitoring, but many organizations have different needs. There are different application servers, clusters, different types of systems, embedded devices, and many other cases. The Java Platform’s open standards have provided a stable foundation for a number of advanced monitoring systems. Some monitoring systems use JMX, others use the JVM TI.

Oracle provides a commercial monitoring system named Mission Control and Flight Recorder that is able to surface an application’s statistics and take automatic action. Flight Recorder has been available in all JREs since Java 7 update 2 (April 2012).

Unlike many monitoring systems that begin recording on a trigger, flight recorder is a minimal-overhead black-box that continually records. When your trigger event occurs, it can provide details about what went on before that event happened. As a result, developers can track down bugs that cannot be replicated outside of a production environment.

 

Marcus Hirt has done a video introduction to Java Mission Control that explains many details and features, such as creating recordings, setting triggers, and controlling remote operations. The video covers the current version of Mission Control (5.3) that took place after HotSpot and JRockit were merged into a single VM.

JMX Usage

The built-in monitoring capabilities of the Java platform provide developers and system administrators with ways to understand their applications without building a new interface. Developers are able to create custom MBeans that expose application-level information to any JMX browser such as jConsole or Mission Control. Many monitoring systems use JMX as a way of understanding internal application health.

For a more complete description on the role of JMX and examples of how to configure it, react to events, or build new MBeans, please see the Java Management Extensions (JMX) Technology Tutorial or Monitoring and Management for the Java Platform.

Wednesday Jul 02, 2014

Diagnosing TLS, SSL, and HTTPS

When building inter-connected applications, developers frequently interact with TLS-enabled protocols like HTTPS. With recent emphasis on encrypted communications, I will cover the way in which the JDK evolves regarding protocols, algorithms, and changes, as well as some advanced diagnostics to better understand TLS connections like HTTPS.

Most developers will not have to do this level of diagnosis in the process of writing or running applications. In the event that you do, the following information should provide enough information to understand what's happening within secure connections.

Stability: The evolution of protocols and algorithms

For the last 15 years (since 1998), the Java platform has evolved through the Java Community Process where companies, organizations, and dedicated individuals develop and vote on specifications to determine what makes up the Java Platform. Much of the efforts are centered on compatibility, like the TCK, ensuring that different implementations are compatible with each-other and that developers can predict how their applications will run. We are not changing critical default options (like TLS protocol) within minor versions.

The following chart depicts the protocols and algorithms supported in each JDK version:


JDK 8
(March 2014 to present)
JDK 7
(July 2011 to present)
JDK 6
(2006 to end of public updates 2013)
TLS Protocols
TLSv1.2 (default)
TLSv1.1
TLSv1
SSLv3
TLSv1.2
TLSv1.1
TLSv1 (default)
SSLv3


TLSv1 (default)
SSLv3
JSSE Ciphers:
Ciphers in JDK 8
Ciphers in JDK 7
Ciphers in JDK 6
Reference:
JDK 8 JSSE JDK 7 JSSE JDK 6 JSSE
Java Cryptography Extension, Unlimited Strength (explained later)
JCE for JDK 8 JCE for JDK 7 JCE for JDK 6

Sample Java code for making an HTTPS connection

Making an HTTPS connection in Java is relatively straight-forward. I will post the code here with the intent focused on tuning and understanding the underlying capabilities.

Sample back-end code for making an SSL connection:

final URL url = new URL("https://example.com");
try(final InputStream in = url.openStream()){
  //…
}

Or the connection can be tuned through a cast:

final HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
//operate on conn
conn.connect();
try(final InputStream in = conn.getInputStream()){
  //…
}

Example: Qualys SSL Labs' "View My Client" Page

Qualys SSL Labs maintains a collection of tools that are helpful in understanding SSL/TLS connections. One in particular is a View My Client page, which will display information about the client connection. By integrating with that page, I was able to control the implementation as I used different Java tuning parameters.

To test parameter tuning, I implemented a small JavaFX application in JavaScript. It displays that page in a WebView, showing information about the underlying Java SSL/TLS client connection. You can find the code in the appendix.

JSSE Tuning Parameters

When diagnosing TLS-related issues, there are a number of helpful system properties. They are generally covered in their relevant sections of JSSE but this single collection may help anyone looking to understand the flexibility of Java’s implementation or diagnose connection details.

javax.net.debug
 Prints debugging details for connections made.
Example: -Djavax.net.debug=all or -Djavax.net.debug=ssl:handshake:verbose
https.protocols
Controls the protocol version used by Java clients. For older versions, this can update the default in case your Java 7 client wants to use TLS 1.2 as its default.
Example: -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2
http.agent
When initiating connections, Java will apply this as its user-agent string. Modifying this will handle cases where the receiving party responds differently based on the user-agent.
Example: -Dhttp.agent="known agent"
java.net.useSystemProxies
java.net.useSystemProxiesUse proxy details from the operating system itself.
Example: -Djava.net.useSystemProxies=true
http.proxyHost
http.proxyPort
The proxy connection to use for HTTP connections.
Example: -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=8080
https.proxyHost
https.proxyPort
The same as above, except that configuration is separate between HTTP and HTTPS.
http.proxyUser
http.proxyPassword
https.proxyUser
https.proxyPassword
Password-based credentials for the above proxies.

Many other protocols and properties can be found within the following areas:

Example of diagnosing a problem

When making an HTTPS connection, let’s assume that the client threw the following exception due to a failed handshake with the server:

 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure 

SSLHandshakeException is a subclass of the IOException, so you do not need to catch is explicitly. Most developers will not need an explicit catch, but it may help you more easily diagnose the cause of any IOException.

When applying the -Djavax.net.debug=all property from above, the failure associated with this SSLHandshakeException would appear immediately after algorithm negotiation in the logs.

 JDK 7 (fails on unsupported algorithm) JDK 8 (works fine)
Cipher Suites: […Long list of ciphers…]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {…}
Extension ec_point_formats, formats: [uncompressed]
Extension server_name, server_name: [host_name: HOST]
***
main, WRITE: TLSv1 Handshake, length = 168
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
Cipher Suites: […Long list of ciphers…]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {…}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: …
Extension server_name, server_name: [type=host_name (0), value=HOST]
***
main, WRITE: TLSv1.2 Handshake, length = 226
main, READ: TLSv1.2 Handshake, length = 89
*** ServerHello, TLSv1.2
RandomCookie:  GMT: -1809079139 bytes = { …}
Session ID:  {…}
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
Extension ec_point_formats, formats: [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
***
%% Initialized:  [Session-1, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
** TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
main, READ: TLSv1.2 Handshake, length = 2308

In the case above, the failure occurred during the handshake. The most likely cause for that is algorithm support. The JDK provides a separate package called JCE Unlimited Strength, designed to add stronger algorithm support than what’s available by default. Qualys SSL Labs provides a different server SSL test that will enumerate which algorithms a server supports.

Many TLS error messages are covered in a few pieces of documentation:

Adding stronger algorithms: JCE Unlimited Strength

In a high security environment, one way of strengthening algorithms in the JDK is through the JCE Unlimited Strength policy files. In this particular case, replacing those policy files within JDK 7 allows it to use the stronger variants of existing algorithms and connect successfully.

JCE Unlimited Strength downloads: JDK 8, JDK 7, or JDK 6.

Appendix

The following code will open Qualys SSL Labs’ View My Client page within a Java client. To test configurations, run this like:

jjs -fx viewmyclient.js
jjs -fx -Dhttps.protocols=TLSv1 viewmyclient.js

var Scene = javafx.scene.Scene;
var WebView = javafx.scene.web.WebView;
var browser = new WebView();
browser.getEngine().load("https://ssllabs.com/ssltest/viewMyClient.html");
$STAGE.scene = new Scene(browser);
$STAGE.show();
About

Science Duke
This blog contains topics related to Java SE, Java Security and Usability. The target audience is developers, sysadmins and architects that build, deploy and manage Java applications. Contributions come from the Java SE Product Management team.

Search

Categories
Archives
« July 2014 »
SunMonTueWedThuFriSat
  
1
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
31
  
       
Today