More on premain and JMX Connectors

In my previous blog entry, I have talked about how to take advantage of java agents in order to start a custom JMX Connector in a Java application, without modifying the application. This is particularly useful when you need to monitor Java applications which are located behind a firewall. In that case, it makes it possible to write a java agent that will start a RMI connector configured in a firewall-friendly manner. However, there's a catch. This is what this entry is about.

The JMX Remote API was originally designed to make it easy for stand alone applications to export their management and monitoring data exposed by means of MBeans to remote management applications. In this context, an application which wanted to be managed through JMX would:

  1. create a JMXConnectorServer through the JMXConnectorServerFactory,
  2. start it,
  3. and eventually stop it before exiting.

However Java agents are now introducing a new constraint: a java agent may start a JMX Connector server at any arbitrary time, but has no clue as to when the application is actually finished working and when the connector should be stopped.
JMXConnectorServers were designed to keep an application alive until they were stopped, but what was an advantage for standalone JMX applications is now becoming an hindrance in the scope of dynamically loaded agents.
Indeed, once a JMX Connector server has been started by a java agent, it will keep the application alive, preventing the application VM to exit until the connector server is explicitely stopped.

But how can a poor java agent detect that the application in which it was loaded is about to exit, and that it should now close the connector server it has previously started?

The first idea that comes to mind is quite naturally to use shutdown hooks, but this is practically unworkable since the JMX Connector Server that was started will precisely prevent the application to exit, and thus shutdown hooks to be called, before it is explicitely stopped. Dead end.

A second idea, suggested by a resourceful JMX user on the SDN JMX forum, is to start a new daemon thread that will monitor the "main" thread, detect when it terminates, and stop the connector server at that point.
This is indeed a smart idea that will probably work in most of the common cases. However, in the general case, the termination of the main thread doesn't necessarily mean that the application has finished working and is about to exit. Indeed, an application terminates only when all its non daemon threads have finished working. The main thread could have been used simply to create some new non daemon thread and could have terminated right away. The application wouldn't exit until the other non daemon threads have finished working, which could be quite later, if those threads are used to carry out the bulk of the application work.

Another possibility could be to register the created JMXConnectorServer in the platform MBeanServer. This way you could stop it from the JConsole MBeans tab when it is no longer needed. However, it still requires a user interaction, and this solution might therefore not be always practicable.

So what other possibilities remain? Unfortunately I don't have any miracle solution. I was only able to build on the idea of monitoring the "main" thread, extending it to detect the moment when the only non-daemon threads that remains are those which are kept alive by the presence of a started JMXConnectorServer. In our case of a JMX RMI Connector, that would be the "RMI Reaper" thread and the "DestroyJavaVM" thread.

So I have experimented writing a "CleanThread", that will randomly select a non-daemon thread (excluding those kept alive by the presence of a started RMI JMXConnectorServer), join it, and proceed again with the same logic until all non-daemon threads that remain are the "RMI Reaper" and the "DestroyJavaVM" threads. At that point, the CleanThread simply stops the JMXConnectorServer, which makes it possible for the RMI Reaper thread to terminate and for the application to exit normally.

Here is the code, and how to plug it in the java agent premain I was discussing in my previous blog.

Thanks a lot to "HugoT" who asked this question on the forum!

Cheers,
-- daniel

Update: see also my post on Building a Remotely Stoppable Connector which suggests another possible alternative.
/\*
 \*
 \* Copyright 2007 Sun Microsystems, Inc.  All Rights Reserved.
 \*
 \* Redistribution and use in source and binary forms, with or without
 \* modification, are permitted provided that the following conditions
 \* are met:
 \*
 \*   - Redistributions of source code must retain the above copyright
 \*     notice, this list of conditions and the following disclaimer.
 \*
 \*   - Redistributions in binary form must reproduce the above copyright
 \*     notice, this list of conditions and the following disclaimer in the
 \*     documentation and/or other materials provided with the distribution.
 \*
 \*   - Neither the name of Sun Microsystems nor the names of its
 \*     contributors may be used to endorse or promote products derived
 \*     from this software without specific prior written permission.
 \*
 \* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 \* IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 \* THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 \* PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 \* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 \* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 \* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 \* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 \* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 \* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 \* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 \*
 \*/
public class CustomAgent {
     ...
     public static class CleanThread extends Thread {
        private final JMXConnectorServer cs;
        public CleanThread(JMXConnectorServer cs) {
            super("JMX Agent Cleaner");
            this.cs = cs;
            setDaemon(true);
        }
        public void run() {
            boolean loop = true;
            try {
                while (loop) {
                    final Thread[] all = new Thread[Thread.activeCount()+100];
                    final int count = Thread.enumerate(all);
                    loop = false;
                    for (int i=0;i<count;i++) {
                        final Thread t = all[i];
                        // daemon: skip it.
                        if (t.isDaemon()) continue;
                        // RMI Reaper: skip it.
                        if (t.getName().startsWith("RMI Reaper")) continue;
                        if (t.getName().startsWith("DestroyJavaVM")) continue;
                        // Non daemon, non RMI Reaper: join it, break the for
                        // loop, continue in the while loop (loop=true)
                        loop = true;
                        try {
                            System.out.println("Waiting on "+t.getName()+
                                    " [id="+t.getId()+"]");
                            t.join();
                        } catch (Exception ex) {
                            ex.printStackTrace();
                        }
                        break;
                    }
                }
                // We went through a whole for-loop without finding any thread
                // to join. We can close cs.
            } catch (Exception ex) {
                ex.printStackTrace();
            } finally {
                try {
                    // if we reach here it means the only non-daemon threads
                    // that remain are reaper threads - or that we got an
                    // unexpected exception/error.
                    //
                    cs.stop();
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
            }
        }
    }

    public static void premain(String agentArgs)
        throws IOException {
        // see my previous blog entry.
        ...
        final JMXConnectorServer cs = ... ;
        // Start the RMI connector server.
        //
        System.out.println("Start the RMI connector server on port "+port);
        cs.start();

        // Start the CleanThread.
        //
        final Thread clean = new CleanThread(cs);
        clean.start();
    }
}
Comments:

First off, thank you Daniel for provided such useful infomation in your blog entries.

I have your CustomAgent working on a development server and a couple of production servers. These are ColdFusion v8 web servers running under JRun, JRE 1.6. I am able to connect to them using the single port through the firewall, but getting mixed results. I am a CF developer with zero Java experience who has been tasked with "tuning the JVM".

On the dev server I can use both jConsole and jVisualVM 1.0.1 and see all of the graphs and information. On the prod servers, I can use jConsole and see the graphs and info. When it comes to jVisualVM, I can see the Thread tab's graphs, but not the graphs on the Monitor tab. Shouldn't it be all or nothing?

The documentation for jVisualVM states that jStatd MUST be running with the same security credentials as the monitored JVM before jVisualVM can connect and monitor. If this is the case, then how can it be that I can see everything on the dev server?

I am now starting to investigate jStatd, but also wondering what it does that JMX is not (yet) able to do? When should we used jStatd? Can you explain the difference between RMI/jStatd and JMX? The more that I read and absorb, the less I seem to comprehend. :-{

TIA.

Kenneth

Posted by Kenneth Mackenzie on November 20, 2008 at 03:07 PM CET #

Hi Kenneth,

VisualVM can use several channels to get information from a target VM. Among those are jstat an JMX. Some of the information can be obrtained by only one channel, some can be obtained by both.

VisualVM has a set of built-in priorities which will make it use one channel or the other dependending on what kind of information it monitors.

In your case, since you have taken the time to open-up a firewall friendly RMI connector on the server, you could probably do without the jstat channel - especially for monitoring your production servers. In that case you don't need to start the jstat daemon on the remote host.

To monitor your target server through its RMI connector, simply go to VisualVM File menu and chose 'Add JMX Connection'. Paste the full jmx service URL of your server in the connection window (should be something like service:jmx:rmi:///jndi/rmi://<host>:port/<somestring>) - this depends on what you coded in the CustomAgent.

If you have deployed a secure JMX connector (RMI over SSL as shown here http://blogs.sun.com/jmxetc/entry/jmx_connecting_through_firewalls_using), then you will also need to pass the appropriate SSL properties on VisualVM command line - that would be the same options you would pass to JConsole as shown here
http://blogs.sun.com/jmxetc/entry/jmx_connecting_through_firewalls_using - that is all the -J-Djavax.net.ssl... properties.

Hope this helps,
-- daniel

Posted by daniel on November 24, 2008 at 09:32 AM CET #

Hi Daniel,

Thanks for the reply. The prioritized channels is probably what I am experiencing. I am reluctant to install or run jStatd, unless absolutely necessary.

My JMX connections use the simple "hostname:port" format; retained in jVisualVM, but manually entered each time for jConsole. The long format for the JMX connection also work to connect, but I still do not see any of the traces on the "Monitor" tab. Although, I can see the real-time traces on the "Threads" tab.

For jVisualVM, the connection to the dev server shows the traces on the "Monitor" tab, while the connections to the prod servers do not.

For jConsole, connections to both dev and prod show all info.

Is jVisualVM "choosing" to use the available channel for the "Threads" tab, and an unavailable channel (jStatd) for the "Monitor" tab? Is there a way to "force" jVisualVM to use the JMX channel?

I'd like to eventually use jVisualVM exclusively, and hope to see visualGC fully integrated and using a JMX channel.

Thanks again for all the great assistance that you provide.

Kenneth

Posted by guest on November 24, 2008 at 12:29 PM CET #

Hi Kenneth,

What you describe is strange. For instance, I have a GlassFish server running on machine 'adelin'. If I run jvisualvm from the machine 'pitre', and add a JMX connection to 'adelin:8686' (my glassfish server has a JMX connection opened on port 8686), then I can see both the Monitoring tab and thread tab working correctly - both must get their information through JMX since I haven't started jstatd on 'adelin'

Can you check that you don't have any jstatd daemon running on the machine where your server is running?

Cheers,

-- daniel

Posted by daniel on November 26, 2008 at 04:22 AM CET #

Hi Daniel,

No jstatd installed. Only differences between the two scenarios are:

The dev server is in front of the firewall running "Java HotSpot(TM) Server VM (1.6.0_01-b06) for windows-x86"; while the production servers are behind the firewall running "Java HotSpot(TM) 64-Bit Server VM (10.0-b19) for windows-amd64 JRE (1.6.0_04-b12)".

Not a great deal to go on ...

Thanks again for your time and responses,

Kenneth

Posted by Kenneth Mackenzie on December 02, 2008 at 01:01 PM CET #

Hi Daniel,

Could it possibly be the type of garbage collector used? I just added the agent to another couple of production servers.

The first whose GC is: -XX:+UseParallelGC
The second's GC is: -XX:+UseParNewGC (with -XX:+UseConcMarkSweepGC)

I can now see the traces on the Monitor tab of VisualVM for the first server, but not the second.

Thanks again for such a cool agent and a great blog,

Kenneth

Posted by Kenneth Mackenzie on December 20, 2008 at 07:36 PM CET #

I Kenneth,

The visualvm experts told me this is a known bug:
https://visualvm.dev.java.net/issues/show_bug.cgi?id=128

It should be fixed in visualvm 1.1.

Posted by daniel on January 06, 2009 at 02:03 AM CET #

Daniel - Thanks much on this excellent post. The above works for me except for two cases.
I'm trying this on Tomcat and I see the following:
1) Shutting down a tomcat server produces a stack trace with root cause
Caused by: java.net.BindException: Address already in use: JVM_Bind
at java.net.PlainSocketImpl.socketBind(Native Method)
at
... snipped ...
sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RMIDirectSocketFactory.java:27)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RMIMasterSocketFactory.java:333)

The "address" in use is the agent port provided as a JVM argument. Funnily enough, this is thrown from the LocateRegistry.createRegistry(port) (right in the middle on premain) statement and is thrown only when you try to shutdown a the running server (in my case Tomcat).
The server finally dies with the following error "
FATAL ERROR in native method: processing of -javaagent failed"

2) The "shutdown" script doesn't cleanly shut down ... and it eventually has to do a kill -9

I'm using your "CleanThread" implementation and not the StoppableAgent one, to be specific.

Am I doing anything obviously wrong?
Thanks,
--Das

Posted by Das on March 29, 2009 at 03:25 PM CEST #

Just thought of posting an update. For Tomcat, a cleaner approach seems to be to use the org.apache.catalina.mbeans.JmxRemoteLifecycleListener (http://www.mail-archive.com/dev@tomcat.apache.org/msg30561.html) that will fix the JMX Connector port as expected.

Thanks,
--Das

Posted by Das on April 30, 2009 at 04:56 PM CEST #

Actually the LifecycleListener has a nice catch to it. It actually exposes only the mbeanserver of Tomcat, which could be something else than the platformMbeanServer. They basically just try to get the first available mbeanserver if there is one, otherwise one will be created.

The only way to consistently get the platformMbeanServer is by setting all the required commandline options for jmx on the java command line. This forces the creations of a mbeanserver and this is practically always the first one.

Cheers,

HugoT

Posted by HugoT on May 29, 2009 at 08:50 AM CEST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Daniel Fuchs blogs on Scene Builder, JMX, SNMP, Java, etc...

The views expressed on this blog are those of the author and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today