Tracing JMX - What's going on in my MBeanServer?

... ever wondered how to debug your JMX application? Here are a few tricks...

In his blog 'Another piece of the tool puzzle', Alan Bateman has shown how to use Java SE 6 Attach API to dynamically activate the JMX RMI connector in a running Java SE 6 VM. In this blog, I will build on his example and show how to dynamically upload an MBean that will activate JMX traces in the attached application [Note: you will need Java SE 6 for this, since the getAgentProperties method we use was added to Java SE 6 b65].

For those of you who are running Tiger (J2SE 5.0)- this example still work with Tiger provided that:

  • your application already has a JMX Connector running
  • you know its JMX Service URL
  • you remove the Java SE 6 specific 'attach' code from the Activate class before compiling it with Tiger.
  • you provide the service:jmx:... URL as command line argument to the Activate class, not the PID.

Traces in JMX

You may not be aware of this, but Sun's implementation of JMX uses java.util.logging to log debug traces. Many of these traces concern internal unexposed classes, but they may help you understand what is going on with your application.

The Sun's JMX implementation has two sets of loggers:

  • javax.management.\*: all loggers related to the JMX API.
  • javax.management.remote.\*: loggers specifically related to the JMX Remote API.

You will find a more complete description of JMX Loggers there.

In this blog, I will show how to activate the JMX traces in two different ways:

  • statically, using a logging.properties file
  • dynamically, using a JMXTracing MBean. We will see that in Java SE 6, we can do this for a running application, even if the JMX connector was not enabled on the command line!

Using a logging.properties file

Nothing is simpler: start your application with the following flags:

java -Djava.util.logging.config.file=<logging.properties> ....

where logging.properties activates traces for JMX loggers:

handlers= java.util.logging.ConsoleHandler
.level=INFO

java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter

java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
//
javax.management.level=FINEST
javax.management.remote.level=FINER

Using a JMX Tracing MBean

The JMX Tracing MBean - shown below, is a small MBean that registers a ConsoleHandler at the root of the Logger tree, and uses the java.util.logging:type=Logging MBean to enable JMX loggers.
[see JMXTracing.java and JMXTracingMBean.java for the full extract].

/\*
 \* JMXTracing.java
 \*
 \* Created on March 2, 2006, 2:05 PM
 \*/

package com.sun.jmx.examples.util.tracing;
...

/\*\*
 \* Class JMXTracing
 \* The JMXTracing MBean is a user friendly interface for setting up JMX 
 \* traces.
 \* 
 \* @author dfuchs
 \*/
public class JMXTracing extends StandardMBean 
        implements JMXTracingMBean, MBeanRegistration {

    private final static Logger LOG = 
            Logger.getLogger(JMXTracing.class.getName());
    
    private final static ObjectName loggingMBean;
    static {
        try {
            loggingMBean = 
                    ObjectName.getInstance(LogManager.LOGGING_MXBEAN_NAME);
        } catch (MalformedObjectNameException x) {
            throw new UndeclaredThrowableException(x);
        }
    }
    
    public JMXTracing() 
        throws NotCompliantMBeanException {
        this(true,true);
    }
    
    // autostart is used in postRegister   method
    // autostop  is used in postDeregister method 
    //
    public JMXTracing(boolean autostart, boolean autostop) 
        throws NotCompliantMBeanException {
        super(JMXTracingMBean.class);
        this.autostart = autostart;
        this.autostop  = autostop;
    }
   
    ...

    // See JMXTracingMBean.java
    public String[] getLoggerNames() {
        final String[] res;
        try {
            res = (String[])
                mbeanServer.getAttribute(loggingMBean,"LoggerNames");
            Arrays.sort(res);
        } catch (RuntimeException x) {
            throw x;
        } catch (Exception x) {
            throw new UndeclaredThrowableException(x);
        }
        return res;
    }


    // See JMXTracingMBean.java
    public String[] list(String pattern) {
        
        final String[] names = getLoggerNames();
        final Set list = new TreeSet();
        for (String name : names ) {
            if (name.matches(pattern)) list.add(name);
        }
        return list.toArray(new String[list.size()]);
    }

    // See JMXTracingMBean.java
    public String[] switchTo(String pattern, String level) {
        
        final Level l = Level.parse(level);
        final String[] list = list(pattern);
        final String[] signature = { 
            "java.lang.String","java.lang.String"};
        final Set result = new TreeSet();
        
        for (String name:list) {
            try {
                mbeanServer.invoke(loggingMBean,"setLoggerLevel",new Object[] {
                    name, level
                    },signature);
                result.add(name);
            } catch (Exception x) {
                LOG.fine("Failed to switch " + name + " to " + level +": "+x);
                continue;
            }
        }
        return result.toArray(new String[result.size()]);
    }
    
    // See JMXTracingMBean.java
    public String getLoggerLevel(String loggerName)
    throws JMException {
        final String[] signature = {
            "java.lang.String"};
        
        try {
            return (String) mbeanServer.invoke(loggingMBean,"getLoggerLevel",
                new Object[] { loggerName },signature);
        } catch (RuntimeException x) {
            throw x;
        } catch (JMException x) {
            throw x;
        }
        
   }
    
    // See JMXTracingMBean.java
   public Map switchJMX(String jmxLevel, String jmxRemoteLevel) {
        final Level l1 = Level.parse(jmxLevel);
        final Level l2 = Level.parse(jmxRemoteLevel);
        final TreeMap tr = new TreeMap();
        
        for (String log : switchTo("javax.management.\*",jmxLevel)) {
            try {
                tr.put(log,getLoggerLevel(log));
            } catch (Exception x) {
                LOG.fine("Failed to get level for " + log);
            }
        }
        for (String log : switchTo("javax.management.remote.\*",
                jmxRemoteLevel)) {
            try {
                 tr.put(log,getLoggerLevel(log));            
            } catch (Exception x) {
                LOG.fine("Failed to get level for " + log);
            }
        }
        for (String log : switchTo("com.sun.jmx.\*",jmxLevel)) {
            try {
                 tr.put(log,getLoggerLevel(log));            
            } catch (Exception x) {
                LOG.fine("Failed to get level for " + log);
            }
        }
        for (String log : switchTo("com.sun.jmx.remote.\*",
                jmxRemoteLevel)) {
            try {
                 tr.put(log,getLoggerLevel(log));            
            } catch (Exception x) {
                LOG.fine("Failed to get level for " + log);
            }
        }
        return tr;
    }
    

    // See JMXTracingMBean.java
    public String[] switchAllJMX(String level) {
        final Level l1 = Level.parse(level);
        final Set res = switchJMX(level,level).keySet();
        final String[] result = res.toArray(new String[res.size()]);
        Arrays.sort(result);
        return result;
    }
    

    // See JMXTracingMBean.java
    public synchronized void setLogOnConsole(boolean on) {
        if (on && handler == null) {
            handler = new ConsoleHandler();
            handler.setLevel(Level.FINEST);
            Logger.getLogger("").addHandler(handler);
        } else if ((!on) && (handler != null)) {
            try {
                Logger.getLogger("").removeHandler(handler);
            } catch (Exception x) {
                LOG.fine("Failed to remove handler");
            }
            handler = null;
        }
    }
   
    // See JMXTracingMBean.java
    public synchronized boolean isLogOnConsole() {
        return handler!=null;
    }
    
    // See JMXTracingMBean.java
    public void enableConsoleLogging() {
        setLogOnConsole(true);
    }
    
    // See JMXTracingMBean.java
    public void disableConsoleLogging() {
        setLogOnConsole(false);
    }
    
    // See JMXTracingMBean.java
    public boolean isDebugOn() {
        return Level.FINEST.equals(LOG.getLevel());
    }
    
    // See JMXTracingMBean.java
    public void setDebugOn(boolean on) {
        if (on) LOG.setLevel(Level.FINEST);
        else LOG.setLevel(Level.INFO);
    }
        
    public ObjectName preRegister(MBeanServer server, ObjectName name) 
        throws Exception {
        if (name == null) 
            name=new ObjectName(this.getClass().getPackage().getName()+
                    ":type="+
                    this.getClass().getSimpleName());
        objectName = name;
        mbeanServer = server;
        
        if (!server.isRegistered(loggingMBean))
            throw new InstanceNotFoundException(loggingMBean.toString());
        return name;
    }

    public void postRegister(Boolean registrationDone) {
        //TODO postRegister implementation;
        if (!registrationDone.booleanValue()) return;
        if (autostart) {
            switchJMX(Level.FINEST.toString(),Level.FINER.toString());
            setDebugOn(false);
            enableConsoleLogging();
        }
    }

    public void postDeregister() {
        //TODO postDeregister implementation;
        if (autostop) {
            switchAllJMX(Level.INFO.toString());
            disableConsoleLogging();
        }
    }

    ...

    private transient MBeanServer mbeanServer;
    private transient ObjectName objectName;
    private transient ConsoleHandler handler = null;
    
    private final boolean autostart;
    private final boolean autostop;
    
    /\*\*
     \* Registers a JMXTracing MBean in the platform MBeanServer
     \*\*/
    public static void register() throws JMException {
        ManagementFactory.getPlatformMBeanServer().
                createMBean(JMXTracing.class.getName(),null);
    }

}

To enable the JMX traces in your application, you simply need to register a JMXTracingMBean in the Platform MBeanServer. You could do this in many different ways - the simplest being to statically call JMXTracing.register() in your main().

Dynamically activate JMX Traces in a running JVM

Today however, I want to emphasize something much cooler:

The Activate class below shows how to dynamically activate these traces in a running JVM!
The small piece of code below feature this in five steps:

  • Step 1: obtain a JMXServiceURL that will let you connect to the running VM. You can either pass the VM PID as returned by jps, or the VM JMXServiceURL - if you already know it.
    If you pass the PID and the JMX Remote connection is not yet enabled, the code will dynamically load and start up the VM JMX Agent. This is the part that was discussed in length by Alan Bateman in his blog.

    // extract from Activate.java //
    

    public static String getConnectorAddress(String jvmRef) 
        throws Exception {
        
        // check whether we already have the address
        if (jvmRef.startsWith("service:jmx:")) return jvmRef;
            
        // attach to the target application
        VirtualMachine vm = VirtualMachine.attach(jvmRef);
        
        // get the connector address
        String connectorAddress =
                vm.getAgentProperties().getProperty(CONNECTOR_ADDRESS);
        
        // no connector address, so we start the JMX agent
        if (connectorAddress == null) {
            String agent = vm.getSystemProperties().getProperty("java.home") +
                    File.separator + "lib" + File.separator + "management-agent.jar";
            vm.loadAgent(agent);
            
            // agent is started, get the connector address
            connectorAddress =
                    vm.getAgentProperties().getProperty(CONNECTOR_ADDRESS);
            assert connectorAddress != null;
        }
        return connectorAddress;
    }

  • Step 2: Create and connect a JMXConnector from the JMXServiceURL, and get an MBeanServerConnection from the connected connector.

    // extract from Activate.java //
    
    /\*\*
     \* Activate JMX traces in a running VM. 
     \* args[0] should be either the VM PID (as returned by {@code jps}) or
     \* the JMX Service URL of the VM JMX Connector.
     \*\*/
    public static void main(String args[]) throws Exception {	
 
        if (args.length == 0) {
            System.err.println("Usage: Activate <pid|JMXServiceURL>");
            System.exit(1);
        }
        
        final String connectorAddress = getConnectorAddress(args[0]);
        try {
            // establish connection to connector server
            JMXServiceURL url = new JMXServiceURL(connectorAddress);
            JMXConnector c = JMXConnectorFactory.connect(url);	
            MBeanServerConnection server = c.getMBeanServerConnection();
            

  • Step 3: Obtain the location of the JMXTracing MBean codebase.

    // extract from Activate.java //
    

            // Get JMXTracing class location
            final ProtectionDomain pd = JMXTracing.class.
                getProtectionDomain();
            final CodeSource cs = pd.getCodeSource();
            final URL clurl = cs.getLocation();
            

  • Step 4: Create an MLet to upload the JMXTracing code in the running VM.

    // extract from Activate.java //
    
            // Create an MLet to upload JMXTracing class in target VM.
            final Object[] params = {new URL[] { clurl }, Boolean.TRUE};
            final String[] signature = {params[0].getClass().getName(), 
                    "boolean"};
            
            final ObjectInstance mlet = 
                server.createMBean(MLet.class.getName(),
                    new ObjectName(JMXTracing.class.getPackage().getName()+
                        ":type=MLet"),params,signature);
            System.out.println(String.valueOf(mlet.getObjectName()) + 
                " registered."); 
            

  • Step 5: Simply use the simplest form of createMBean to create a JMXTracing MBean in the running VM. As soon as the MBean is registered, you should be seeing the JMX traces on your application standard error stream (System.err).

    // extract from Activate.java //

            // Register the JMXTracing MBean in the target VM
            final ObjectInstance tracing =
                server.createMBean(JMXTracing.class.getName(),null);
            System.out.println(String.valueOf(tracing.getObjectName()) + 
                " registered."); 
   

That's it!

Simply compile JMXTracing.java, JMXTracingMBean.java, and Activate.java - and put the generated clases in a jar. Then get your application PID with jps -l as shown by Alan in his blog.

Then upload and deploy the JMXTracing MBean in your application by calling:

java -classpath your-jmxtracing-jar com.sun.jmx.examples.util.tracing.Activate your-application-pid

Your application will start spitting out JMX traces on its System.err

You can now also connect with a regular jconsole - and update the logger's level by calling operations on the JMXTracing MBean from the JConsole MBean tab...

Cool, isn't it?
Cheers to all,
-- daniel

    PS#1: the code shown in this blog assumes that your application is not secured. To achieve the same thing with secure applications you would need to configure the access control (java.policy) of your application appropriately.

    PS#2: the Activate class shown here only works when launched on the same machine as your application process. To achieve the same thing with remote JVMs you would need to set up an HTTP server for the JMXTracing jar, or to use NFS, and provide a JMXServiceURL as input - since the attach API only works locally.

Comments:

Where can I get com.sun.tools.attach.VirtualMachine ? I can see the doc in Sun's site, but I can't find the class in the JDK nor a download link.

Posted by Loïc on May 20, 2009 at 11:08 AM CEST #

Oooops ! Sorry, it's in tools.jar in the JDK.

Posted by Loïc on May 20, 2009 at 11:14 AM CEST #

It's in tools.jar in your JDK install.
-- daniel

Posted by daniel on May 20, 2009 at 11:14 AM CEST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Daniel Fuchs blogs on Scene Builder, JMX, SNMP, Java, etc...

The views expressed on this blog are those of the author and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today