Building a Remotely Stoppable Connector Server

JMX is a wonderful tool to monitor and troubleshoot running applications. The new JDK 6 Attach API makes it very easy to attach to a running Java process, and start a JMX agent that will expose monitoring and configuration data to JMX consoles - like JConsole. However, there are some situations where you wish to start a JMX agent on demand, explore the monitoring data or diagnose the probable cause of an observed problem, and then close your JMX agent, leaving the application just how you found it.

In this post, I will discuss a means by which you can upload and start such a remotely stoppable JMX agent. Here is how.

Starting an agent on demand

In some of my previous posts I have outlined how to use a premain agent in order to create a custom JMX RMI Connector Server. In this post, we're going to reuse much of the concepts explained before. However, we will add the ability to dynamically upload the JMX agent in the running application by creating an Agent-Main class. In fact, an Agent-Main class is nothing more than a class which has a method:

        public static void agentmain(String agentArgs);

Our agentmain method will create and start a Remotely Stoppable JMX RMI Connector Server. Later on, when we're done with monitoring our application, we will be able to stop the JMX Connector Server that was created by our agentmain, thus leaving the application in its original state (or close to it).

Custom agents started on demand cannot be stopped

The problem with custom agents created on demand is that they can't be stopped: you can't stop them at the end of the agentmain() or premain() method, because it's precisely at this point that you will start using them. And the problem with running custom agents is that they usually hold some non daemon threads alive (e.g. RMI DGC...) and thus prevent the application in which they run from exiting when all its non-daemon threads have finished working.

This behaviour is a much desirable feature in a standalone JMX application, where you can control when to start and stop your JMX Connector Server - but it leads to awkward situations in the case of JMX Agents started from premain() or agentmain().

In a previous post I have suggested a work around which consisted on creating a CleanThread daemon, which would randomly select a non-daemon thread (excluding those kept alive by the presence of a started RMI JMXConnectorServer), join it, and proceed again with the same logic until all non-daemon threads that remain are the "RMI Reaper" and the "DestroyJavaVM" threads. At that point, the CleanThread daemon would simply stop the JMXConnectorServer, which would make it possible for the RMI Reaper thread to terminate and for the application to exit normally.
This trick however presents several limitations:

  • It only works with the JMX RMI/JRMP connector. There's no way to do the same thing for the JMX RMI/IIOP connector, for instance.
  • It doesn't allow you to stop the connector when you no longer need it. The connector will stay started until the application exits.

In this post, we will show how to:

  • Dynamically load and start a Remotely Stoppable JMX RMI Connector Server - using an Agent-Main class
  • Perform some remote monitoring action through this connector - e.g. using JConsole from a remote machine.
  • Then remotely stop the JMX RMI Connector Server, leaving the application running in a clean state.

If needed, the JMX Connector Server can also be re-created and re-started later on. To make things less easy, we will do all of this with a secure firewall-friendly (one single port) RMI Connector Server.

Making a Remotely Stoppable JMX Connector Server

A Stoppable JMX Agent
click to enlarge picture

The trick I will be suggesting here is simply to use an MBeanServerForwarder which will hold a reference to the JMX Connector Server on which it is configured.

Our MBeanServerForwarder is a simple pass-through which passes on all requests it receives to an underlying MBeanServer.
The MBeanServerForwarder can be configured with a series of InvokeOperationInterceptors. The InvokeOperationInterceptors are used to intercept a given invoke operation directed at an MBean, or at a set of MBeans defined by an ObjectName pattern.

In our case, we will register a single InvokeOperationInterceptor (red blob on the picture) which will intercept a stop operation directed at a "fake" MBean.
When an invoke operation directed at that "fake" MBean reaches the MBeanServerForwarder, the MBeanServerForwarder diverts it to our InvokeOperationInterceptor. If the operation is stop(), the InvokeOperationInterceptor will stop the JMX Connector Server.

The "fake" MBean is not returned by queryNames() or queryMBeans() since it is not registered in the underlying MBeanServer. The name of the "fake" MBean must thus be a secret shared between the program that will create and start the connector server, and the program that will remotely stop the connector server.
Using such a shared secret is acceptable in our case because we're using a secure JMX Connector Server. You will also see that my implementation is a bit lax - because it stores that secret in a System property (the secret is not too heavily guarded).

Project Sources

The code source that does all of this comprises the following classes:

  • StoppableAgent.java: our Agent-Main class. Can also be used as a premain agent.
  • ForwardingInterceptor.java: our configurable MBeanServerForwarder, implemented as a java.lang.reflect.Proxy.
  • InvokeOperationInterceptor.java a default InvokeOperationInterceptor which acts as a pass-through.
  • Stopper.java: creates and configure a ForwardingInterceptor that can stop its connector server. Can also send a stop request through a given JMXConnector
  • Attach.java: a command line tool. The RMI port and secret name are passed through system properties:
    • start <pid>: uses the Attach API to dynamically start a StoppableAgent in the target process.
    • status: creates a JMXConnector and checks whether the remote server that was started in a previous call is still alive (useful for tests).
    • list: list running java processes
    • stop: creates a JMXConnector to connect to the StoppableAgent and invoke stop on the "fake" MBean.
    • help: prints a crude help message

You can see the example sources here in the form of a NetBeans 6 project.

Building the StoppableAgent jar

Once you have compiled all the classes, you can create an agent jar by invoking an ant target like this one in your build.xml (already done in the provided NetBeans 6 project):

    <!-- Builds dist.agent.jar -->    
    <target name="-build-agent-jar"
        description="build an agent jar that can be used either with -javaagent or the Attach API">
        <jar basedir="${build.classes.dir}" 
            jarfile="${dist.agent.jar}">
                <manifest>
                    <attribute name="Premain-Class" value="example.rmi.agent.StoppableAgent"/>
                    <attribute name="Agent-Class" value="example.rmi.agent.StoppableAgent"/>
                </manifest>
        </jar>
        <echo>To use this application with agent try:</echo>
        <echo>java &lt;SSL Properties&gt; -Dexample.rmi.agent.port=3000 -Dexample.rmi.agent.stopper=FooBar -cp:${dist.agent.jar} example.rmi.agent.Attach start &lt;pid&gt;</echo>
    </target>
                            

If you're using the provided NetBeans 6 project simply clean & build the StoppableAgent project: the agent jar (dist/jagent.jar) will be automatically generated.

Loading and Starting the agent in a Running application

The StoppableAgent class creates a secure JMX RMI Connector Server. This means that the target application in which the agent will be loaded must have an appropriate truststore and keystore. The Attach class provided in this example will forward its own default SSL configuration to the target application if the example.rmi.agent.ssl.config.send System property is set to true. In the general case, this is a very bad idea as it is profundly unsecure - and this is why the default is example.rmi.agent.ssl.config.send=false. A much better idea would be to configure SSL properly when starting the target application.

If you don't have a target application to play with, you could use this one:

    public class Test { 
        public static void main(String[] args) throws Exception {
            System.out.println("Strike Enter to exit:");
            System.in.read();
        }
    }
        

Compile it and start it as follow:

   
   java -classpath . \\
        -Djavax.net.ssl.keyStore=<keystore> \\
        -Djavax.net.ssl.keyStorePassword=<password> \\
        -Djavax.net.ssl.trustStore=<truststore> \\
        -Djavax.net.ssl.trustStorePassword=<trustword> \\
        Test
        
Note: If you haven't already a keystore and trustore to play with, the Monitoring and Management guide has a section that explains how to create them.

To start the stoppable agent in a target application use the following command:

   
   java -Dexample.rmi.agent.port=<port> \\
        -Dexample.rmi.agent.stopper=<secret-name> \\
        -cp <agent.jar>:<jdk.home>/tools.jar \\
        -Djavax.net.ssl.keyStore=<keystore> \\
        -Djavax.net.ssl.keyStorePassword=<password> \\
        -Djavax.net.ssl.trustStore=<truststore> \\
        -Djavax.net.ssl.trustStorePassword=<trustword> \\
        example.rmi.agent.Attach start <pid>
        

To check that the StoppableAgent was correctly started run the following command:

  
   java -Dexample.rmi.agent.port=<port> \\
        -Dexample.rmi.agent.stopper=<secret-name> \\
        -cp <agent.jar>:<jdk.home>/tools.jar \\
        -Djavax.net.ssl.keyStore=<keystore> \\
        -Djavax.net.ssl.keyStorePassword=<password> \\
        -Djavax.net.ssl.trustStore=<truststore> \\
        -Djavax.net.ssl.trustStorePassword=<trustword> \\
        example.rmi.agent.Attach status
        

Connecting from remote

To connect with JConsole, start it with the following command line:

  
   jconsole -J-Djavax.net.ssl.keyStore=<keystore> \\
        -J-Djavax.net.ssl.keyStorePassword=<password> \\
        -J-Djavax.net.ssl.trustStore=<truststore> \\
        -J-Djavax.net.ssl.trustStorePassword=<trustword>
        

Then in the JConsole connection window select Remote Connection and simply type <host>:<port> in the entry field - and that's it! JConsole will connect to your application through your JMX RMI SSL Connector.

Stopping the agent

To stop the agent, you now only need to invoke the following command:

  
   java -Dexample.rmi.agent.port=<port> \\
        -Dexample.rmi.agent.stopper=<secret-name> \\
        -cp <agent.jar>:<jdk.home>/tools.jar \\
        -Djavax.net.ssl.keyStore=<keystore> \\
        -Djavax.net.ssl.keyStorePassword=<password> \\
        -Djavax.net.ssl.trustStore=<truststore> \\
        -Djavax.net.ssl.trustStorePassword=<trustword> \\
        example.rmi.agent.Attach stop
        

Some additional considerations

The 'start' command can only be invoked from the local machine where the application is running. This is because we use the Attach API to upload the stoppable agent.

However, once done, the agent can be accessed from remote in a secure way. Since 'status' and 'stop' use a regular JMX connection with the stoppable agent, they could also be invoked from a remote machine.

Cheers,
-- daniel

Comments:

Your solutions, while innovative, are basically just tricks to get around the horrible application architecture of what is supposedly an enterprise management/monitoring framework. By your own admission the cons are pretty serious:
\* It only works with the JMX RMI/JRMP connector. There's no way to do the same thing for the JMX RMI/IIOP connector, for instance.
\* It doesn't allow you to stop the connector when you no longer need it. The connector will stay started until the application exits.

Is there any move on Sun's part to create a REAL solution to these challenges? I can't imagine my company is the only one in the world that has a default deny policy on their firewall. My security guys will only give me a port or two, not access to 65,535 ports on the server.

Frustrated by JMX,
ErikC

Posted by ErikC on April 17, 2008 at 07:00 PM CEST #

Hi Erick,

In retrospect, it was certainly a mistake to have the default agent use two port numbers, instead of a single one.
Bear in mind however that having the possibility to start a connector on demand through the attach API is quite an advance feature of the Sun JVM.
All these problems that I cite arise when you try to use this attach on demand feature:

The regular use case for managing an application is to start and stop the JMXConnector server in the main() of your application, where none of these problems arise since you have full control over the connector configuration, and over when to start it and stop it.

Note also that the remotely stoppable connector trick shown above precisely works with all kind of connectors, JRMP, IIOP, or whatever.
What didn't work with the IIOP connector was the CleanThread daemon - which indeed was a horrible trick, but if you use a remotely stoppable connector like above you won't need that trick.

For some time now we've been working on a WebService connector for JMX (this is JSR 262) - and this will certainly help to get through firewall.

I do believe that these issues around the JMX RMI connector and firewalls is something we do need to address in the next version of the API for JDK 7.

I am sorry to hear your frustration. I wrote these articles in the hope it could help quickly getting around firewall issues.
Did you get to try my remotely stoppable JMXConnectorServer above?

Best regards, and let me know if I can help.

-- daniel

Posted by daniel on April 18, 2008 at 02:23 AM CEST #

Thanks for the comment, and I apologize for sounding so harsh. My fellow coworkers pointed out that I sounded overly frustrated...but it was with JMX not you! :-)

See the problem is we're trying to monitor Tomcat via JMX and we'd rather not branch the code base modify the main() for ourselves as to make upgrades easier.

Oh well, we'll keep looking for a compromise that works between SEC & DEV.

Cheers,

ErikC

Posted by ErikC on April 21, 2008 at 05:50 PM CEST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Daniel Fuchs blogs on Scene Builder, JMX, SNMP, Java, etc...

The views expressed on this blog are those of the author and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today