Java 5, premain, RMI Connectors, Single Port, SSL, and Firewall.

... So many words I couldn't even put them all in the title...

I've been asked several times how to make my example of javaagent which starts a firewall friendly JMX RMI Connector work on JDK 5. Well, here is how. However, beware of the catch: if you use SSL and want to connect with JConsole then you need to use Java 6 JConsole on the client side.

To summarize it, here is what you need to do:

  1. Add a public static void premain(String agentArgs, Instrumentation inst) method that calls premain(String agentArgs)
  2. Manually bind the RMIServerImpl stub to the RMI Registry
  3. Manually create and starts an RMIConnectorServer instead of going through the JMXConnectorServerFactory.

The overhead of the two last steps is because JDK 5 doesn't have a fix for 5107423. This was fixed in JDK 6 - and makes it possible to specify the socket factory to use when binding to the RMI registry through JNDI. Since we don't have the fix, we can't use JNDI, and we get to do everything by hand.

Here is the new code you need: (see the rest of the code in my previous post)


    // This the new method you need
    public static void premain(String agentArgs, Instrumentation inst) 
        throws IOException {
        premain(agentArgs);
    }
    
    public static void premain(String agentArgs) 
        throws IOException {
        // Ensure cryptographically strong random number generator used
        // to choose the object number - see java.rmi.server.ObjID
        //
        System.setProperty("java.rmi.server.randomIDs", "true");

        // Start an RMI registry on port specified by example.rmi.agent.port
        // (default 3000).
        //
        final int port= Integer.parseInt(
                System.getProperty("example.rmi.agent.port","3412"));
        System.out.println("Create RMI registry on port "+port);
        
        // We create a couple of SslRMIClientSocketFactory and 
        // SslRMIServerSocketFactory. We will use the same factories to export
        // the RMI Registry and the JMX RMI Connector Server objects. This 
        // will allow us to use the same port for all the exported objects.
        // If we didn't use the same factories everywhere, we would have to
        // use at least two ports, because two different RMI Socket Factories
        // cannot share the same port.
        //
        final SslRMIClientSocketFactory csf = new SslRMIClientSocketFactory();
        final SslRMIServerSocketFactory ssf = new SslRMIServerSocketFactory();
        
        // Create the RMI Registry using the SSL socket factories above.
        // In order to use a single port, we must use these factories 
        // everywhere, or nowhere. Since we want to use them in the JMX
        // RMI Connector server, we must also use them in the RMI Registry.
        // Otherwise, we wouldn't be able to use a single port.
        //
        final Registry registry = 
              LocateRegistry.createRegistry(port, csf, ssf);
        
        
        // Retrieve the PlatformMBeanServer.
        //
        System.out.println("Get the platform's MBean server");
        MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();

        // Environment map.
        //
        System.out.println("Initialize the environment map");
        HashMap<String,Object> env = new HashMap<String,Object>();
        
        // Manually creates and binds a JMX RMI Connector Server stub with the 
        // registry created above: the port we pass here is the port that can  
        // be specified in "service:jmx:rmi://"+hostname+":"+port - where the
        // RMI server stub and connection objects will be exported.
        // Here we choose to use the same port as was specified for the   
        // RMI Registry. We can do so because we're using \*the same\* client
        // and server socket factories, for the registry itself \*and\* for this
        // object.
        //
++      final RMIServerImpl stub = 
++            new RMIJRMPServerImpl(port, csf, ssf, env);
        
        // Create an RMI connector server.
        //
        // As specified in the JMXServiceURL the RMIServer stub will be
        // registered in the RMI registry running in the local host on
        // port <port> with the name "jmxrmi". This is the same name the
        // out-of-the-box management agent uses to register the RMIServer
        // stub too.
        //
        // The port specified in "service:jmx:rmi://"+hostname+":"+port
        // is the second port, where RMI connection objects will be exported.
        // Here we use the same port as that we choose for the RMI registry. 
        // The port for the RMI registry is specified in the second part
        // of the URL, in "rmi://"+hostname+":"+port
        //
        // We construct a JMXServiceURL corresponding to what we have done
        // for our stub...
        System.out.println("Create an RMI connector server");
        final String hostname = InetAddress.getLocalHost().getHostName();
        final JMXServiceURL url =
            new JMXServiceURL("service:jmx:rmi://"+hostname+
            ":"+port+"/jndi/rmi://"+hostname+":"+port+"/jmxrmi");
        
        
        System.out.println("creating server with URL: "+url);
        
        // Now create the server manually.... 
        // We can't use the JMXConnectorServerFactory because of 
        // http://bugs.sun.com/view_bug.do?bug_id=5107423
        //
++      final JMXConnectorServer cs = 
++              new RMIConnectorServer(new JMXServiceURL("rmi",hostname,port),
++                                     env,stub,mbs) {
++          @Override
++          public JMXServiceURL getAddress() { return url;}
++
++          @Override
++          public synchronized void start() throws IOException {
++              try {
++                 registry.bind("jmxrmi", stub);
++              } catch (AlreadyBoundException x) {
++                  final IOException io = new IOException(x.getMessage());
++                  io.initCause(x);
++                  throw io;
++              }
++              super.start();
++          }
++      };
           
        
        // Start the RMI connector server.
        //
        System.out.println("Start the RMI connector server on port "+port);
        cs.start();
        System.out.println("Server started at: "+cs.getAddress());

        // Start the CleanThread daemon...
        //
        final Thread clean = new CleanThread(cs);
        clean.start();
    }

Hope you find this useful!

Cheers,
--daniel

PS: you can get rid of the clean thread uglyness by mixing this with using a remotely stoppable connector...

Comments:

Thanks for posting this. I followed these instructions, and now I do not get the "non-JRMP" exception I was experiencing when attempting to setup the JMX Connector with your other post geared toward Java 6.

However, now I get a "non-JRMP" server exception when I connect to the server.

Any ideas?

Posted by Dana P'Simer on July 30, 2008 at 10:38 AM CEST #

Hi,

This is most probably because the client is not using SSL when querying the RMI registry.

If the client is JConsole then you must:

o Use JConsole from JDK 6,
o Call jconsole with all the -J-Djavax.net.ssl.\* flags as shown on the other post,
o Connect using the remote connection window, and supply only <hostname>:<port> as input - do not enter the full JMXServiceURL.

Hope this helps,

-- daniel

Posted by daniel on July 30, 2008 at 11:24 AM CEST #

Hi Daniel,

Thanks for this series of articles. I think it has me on the precipice of solving our problem. We're running Tomcat for our app server on jdk 1.5. And we want to monitor our appserver with JConsole through our firewall. I am starting off by testing this on my workstation. I created a jagent.jar (manifest is set up correctly) and I have started Tomcat with the following:

-server \\
-Xms128m \\
-Xmx512m \\
-Dcom.sun.management.jmxremote \\
-Dcom.sun.management.jmxremote.port=9101 \\
-Dcom.sun.management.jmxremote.password.file=mypasswordfile \\
-Dcom.sun.management.jmxremote.access.file=myaccessfile \\
-Djavax.net.ssl.keyStore=serverkeystore.jks \\
-Djavax.net.ssl.keyStorePassword=changeit \\
-Djavax.net.ssl.trustStore=servertruststore.jks \\
-Djavax.net.ssl.trustStorePassword=changeit \\
-Dmy.agent.agent.rmi.port=3000 \\
-javaagent:/path/to/my/jagent.jar \\

# Irrelevant stuff omitted

And then I start jconsole (JDK 1.6 version) like this:
jconsole \\
-J-Djavax.ssl.keyStore=/Users/agherna/mykeystore.jks \\
-J-Djavax.ssl.keyStorePassword=changeit \\
-J-Djavax.ssl.trustStore=mytruststore \\
-J-Djavax.ssl.trustStorePassword=changeit

I try to connect to my local tomcat with jconsole (1.6) using localhost:3000, but I get Connection failed: non-JRMP server at remote endpoint. Am I not setting something correctly here?

Posted by Andy on July 31, 2008 at 07:05 PM CEST #

Hi Andy,

You are aware that your command line on the server side will start three RMI connectors, right?

The -Dcom.sun.management.jmxremote flag will start a local connector that can be used by JConsole/VisualVM - when running on the same machine under the same user - by attaching to the server's PID.

The other -Dcom.sun.management.jmxremote.\* properties configure a second RMI connector, that will be opened for remote connection on port 9101.

Finally the -javaagent will open a third connector, which is the one that we've been creating above in this post. Also take care that this connector doesn't use the password file and access file - it doesn't parse any of the -Dcom.sun.management.jmxremote.\* property. If you want this connector to also use a password file and access file you will have to add the following properties to its environment map:

// Provide the password file used by the connector server to
// perform user authentication. The password file is a properties
// based text file specifying username/password pairs.
//
env.put("jmx.remote.x.password.file", "password.properties");

// Provide the access level file used by the connector server to
// perform user authorization. The access level file is a properties
// based text file specifying username/access level pairs where
// access level is either "readonly" or "readwrite" access to the
// MBeanServer operations.
//
env.put("jmx.remote.x.access.file", "access.properties");

were "password.properties" is the path to your password file and "access.properties" is the pass to your access file.
http://blogs.sun.com/lmalventosa/entry/secure_management_agent

Now concerning your question here are a few things that you could do:

1) Check the server log/output to verify that the connector server started by the javaagent as properly started, at the port you think it has...

2) check that all the paths provided on the command line are correct

3) If you're running on Linux, make sure that there isn't any hostname resolution issue - see my other post about that:
http://blogs.sun.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole

Finally you might want to enable the various traces on the server side and see if smething happens when your client try to connect.

Hope this helps,
-- daniel

Posted by daniel on August 03, 2008 at 04:35 AM CEST #

No, I did not know that. Thank you for pointing that out.

I started my server like this:

-server \\
-Xms128m \\
-Xmx512m \\
-Dcom.sun.management.jmxremote.password.file=mypasswordfile \\
-Dcom.sun.management.jmxremote.access.file=myaccessfile \\
-Djavax.net.ssl.keyStore=serverkeystore.jks \\
-Djavax.net.ssl.keyStorePassword=changeit \\
-Djavax.net.ssl.trustStore=servertruststore.jks \\
-Djavax.net.ssl.trustStorePassword=changeit \\
-Dmy.agent.agent.rmi.port=3000 \\
-javaagent:/path/to/my/jagent.jar \\

and I changed my custom agent to pick up the com.sun.management.jmxremote.password.file and com.sun.management.jmxremote.access.file properties and put them in to the properties you mention above, but I am still getting the same error.

I noticed in the code above, you are not populating the env map as you were in the previous examples (is this listing just a delta of what was in your previous post). My code populates it as follows:

// Environment map.
//
System.out.println("Initialize the environment map");

HashMap<String,Object> env = new HashMap<String,Object>();
env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE,csf);

env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE,ssf);
env.put("com.sun.jndi.rmi.factory.socket", csf);

env.put("jmx.remote.x.password.file", "/path/to/my/copy/of/jmxremote.password");
env.put("jmx.remote.x.access.file", "/path/to/my/copy/of/jmxremote.access");

Assuming my paths are correct, is there something that should (not) be set when developing this with JDK 1.5?

Posted by Andy on August 04, 2008 at 11:09 AM CEST #

Hi Andy,

env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE,csf);
env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE,ssf);
env.put("com.sun.jndi.rmi.factory.socket", csf);

shouldn't be needed - since we're constructing & binding the RMIJRMPServerImpl by hand - but passing them in the env map shouldn't have any effect - they should be ignored since they are not needed.

An important part in the code above is the JMXServiceURL which is passed to the RMIConnectorServer: it must not be an URL that contains a /jndi/ path - otherwise the server will try to use JNDI to bind the RMIJRMPServerImpl and that will fail on Java 5 because the fix for 5107423 is not there (BTW that's the fix which brings the "com.sun.jndi.rmi.factory.socket" property).

Do you see the line that says "Server started at: "... when you start your server with the java agent?

Hope this helps,
-- daniel

Posted by daniel on August 04, 2008 at 11:25 AM CEST #

Yes, I do. In fact, here is a relevant snippet from the output during tomcat startup:

Create RMI registry on port 3412
Get the platform's MBean server
Initialize the environment map
Create an RMI connector server
creating server with URL: service:jmx:rmi://localhost:3412/jndi/rmi://localhost:3412/jmxrmi
Start the RMI connector server on port 3412
Starting JMXConnectorServer at: service:jmx:rmi://localhost:3412/jndi/rmi://localhost:3412/jmxrmi
Server started at: service:jmx:rmi://localhost:3412/jndi/rmi://localhost:3412/jmxrmi
Waiting on main [id=1]

Posted by Andy on August 04, 2008 at 11:43 AM CEST #

One more thing, I am setting up the JMXServiceURL like this:

final String hostname = InetAddress.getLocalHost().getHostName();
final JMXServiceURL url =
new JMXServiceURL("service:jmx:rmi://"+hostname+
":"+port+"/jndi/rmi://"+hostname+":"+port+"/jmxrmi");

Posted by Andy on August 04, 2008 at 11:45 AM CEST #

And you cannot connect with JConsole from JDK 6 using "localhost:3412"?

Are you sure you're not trying to connect to port 3000?

-- daniel

Posted by daniel on August 04, 2008 at 11:50 AM CEST #

Right. I turned off the custom port as above and I am trying to connect via localhost:3412.

I have the logging turned on for JConsole, but it does not log anything until a successful connection is made (I noticed this when I was connecting via the PID and using the -Dcom.sun.management.jmxremote option). The tomcat log doesn't show anything that resembles an attempt to connect either.

Posted by Andy on August 04, 2008 at 11:54 AM CEST #

humm... Try to call jconsole -debug - hopefully it can give more info if the connection fails...

All I can say is that it should work.
Can you try outside tomcat, using this test application instead:

public class Test {
/\*\*
\* @param args the command line arguments
\*/
public static void main(String[] args) throws IOException {
System.out.println("Strike <Enter> to exit:");
System.in.read();
}
}

Posted by daniel on August 04, 2008 at 12:10 PM CEST #

I tried the test application you give above. I started it with the following arguments:

-javaagent:/Users/agherna/tomcathosts/mywsillinois/bin/jagent.jar \\
-Djavax.net.ssl.keyStore=mykeystore.jks \\
-Djavax.net.ssl.keyStorePassword=changeit \\
-Djavax.net.ssl.trustStore=mytruststore.jks \\
-Djavax.net.ssl.trustStorePassword=changeit

I started JConsole (1.6 version) like this:
jconsole \\
-J-Djava.util.logging.config.file=logging.properties \\
-J-Djavax.ssl.keyStore=mypersonalkeystore.jks \\
-J-Djavax.ssl.keyStorePassword=changeit \\
-J-Djavax.ssl.trustStore=myjssecacerts.jks \\
-J-Djavax.ssl.trustStorePassword=changeit \\
-debug

The output the application gave at startup was as expected (from Netbeans 6.1):

init:
deps-jar:
compile:
run:
Create RMI registry on port 3412
Get the platform's MBean server
Initialize the environment map
Create an RMI connector server
creating server with URL: service:jmx:rmi://localhost:3412/jndi/rmi://localhost:3412/jmxrmi
Start the RMI connector server on port 3412
Server started at: service:jmx:rmi://localhost:3412/jndi/rmi://localost:3412/jmxrmi
Strike <Enter> to exit:
Waiting on main [id=1]
Aug 4, 2008 1:44:41 PM edu.illinois.my.portal.agent.jmx.JmxAgent$CleanThread run
INFO: Waiting on main [id= 1]

But when I attempted to connect to it using localhost:3412, I get the following stacktrace from my Terminal:

java.rmi.ConnectIOException: non-JRMP server at remote endpoint
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:230)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:218)
at sun.tools.jconsole.ProxyClient.<init>(ProxyClient.java:111)
at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:474)
at sun.tools.jconsole.JConsole$3.run(JConsole.java:510)

Posted by Andy on August 04, 2008 at 01:51 PM CEST #

One more question, you say "An important part in the code above is the JMXServiceURL which is passed to the RMIConnectorServer: it must not be an URL that contains a /jndi/ path - otherwise the server will try to use JNDI to bind the RMIJRMPServerImpl and that will fail on Java 5 because the fix for 5107423 is not there (BTW that's the fix which brings the "com.sun.jndi.rmi.factory.socket" property)."

Should the "/jndi" part of the URL not be in the JMXServiceURL when com.sun.jndi.rmi.factory.socket is not part of the environment map? In other words should this:

final JMXServiceURL url = new JMXServiceURL("service:jmx:rmi://"+hostname+
":"+port+"/jndi/rmi://"+hostname+":"+port+"/jmxrmi");

really be:

final JMXServiceURL url = new JMXServiceURL("service:jmx:rmi://"+hostname+
":"+port+"/rmi://"+hostname+":"+port+"/jmxrmi");

?

Posted by Andy on August 04, 2008 at 03:03 PM CEST #

Hi Andy,

Several things might be going wrong - but the more likely is that your SSL config on the client side is not correct. If your config is wrong - you will get this "non-JRMP server at endpoint" exception.

For instance, this is exactly what I get if put a wrong password in the javax.net.ssl.\* properties.
It also happens if I give a path to a keystore or truststore that doesn't exists.

So my diagnostic at this point is that something is wrong with your SSL configuration - either at the client side or the server side or both. Could be that the certificates in your truststore/keystore don't match for instance... Or maybe they are outdated... or...

For the other question concerning the JMXServiceURL, you will have noticed that I use two JMXServiceURLs: one is printed on stdout and is returned by the overriden getAddress() method, the other is passed to the constructor of the RMIConnectorServer. What I meant is that the address passed to the constructor of the connector server must be exactly what I am passing in the code: it must not be the address printed on System.out.

Hope this helps,

-- daniel

Posted by daniel on August 05, 2008 at 04:25 AM CEST #

Daniel,

Thank you. This appeared to be part of the problem. The real problem was that I was starting JConsole like this:

jconsole \\
-J-Djava.util.logging.config.file=logging.properties \\
-J-Djavax.ssl.keyStore=mypersonalkeystore.jks \\
-J-Djavax.ssl.keyStorePassword=changeit \\
-J-Djavax.ssl.trustStore=myjssecacerts.jks \\
-J-Djavax.ssl.trustStorePassword=changeit \\
-debug

when I should have started it like this:

jconsole \\
-J-Djava.util.logging.config.file=logging.properties \\
-J-Djavax.net.ssl.keyStore=mypersonalkeystore.jks \\
-J-Djavax.net.ssl.keyStorePassword=changeit \\
-J-Djavax.net.ssl.trustStore=myjssecacerts.jks \\
-J-Djavax.net.ssl.trustStorePassword=changeit \\
-debug

It came down to the keystore and trust store being ignored since I wasn't specifying the right names for the properties when starting jconsole (the server had the right properties set). Thank you for your help.

Andy

Posted by Andy on August 05, 2008 at 10:42 AM CEST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Daniel Fuchs blogs on Scene Builder, JMX, SNMP, Java, etc...

The views expressed on this blog are those of the author and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today