Troubleshooting connection problems in JConsole

... I've seen a few posts in the Java and JMX forums from developers who were wondering how to find out why JConsole wouldn't connect to their application. So I have decided to write this short blog entry in order to outline a few diagnosing tips...

    Note: if you are using JConsole - you might also want to try the Java VisualVM. Java VisualVM comes with the JDK since JDK 6 update 7. It can be used to troubleshoot, monitor, and improve applications performance. It makes it possible to generate and analyse heap dumps, track down memory leaks, browse the platform's MBeans and perform operations on those MBeans, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. It is also an open source project and can be downloaded as a stand alone tool from visualvm.dev.java.net.

If you have read the JConsole FAQ but are still experiencing difficulties, here are a few additional tips:

Processes not displayed in JDK 6 JConsole connection window

This may be due to weird permissions in the TMP dir. See this post for a more detailed description.

See also this post where David explains how your TMP dir settings can prevent the tutorial examples from working under windows systems - and how to solve it. On the JMX Forum, Thomas provides another hint:

    My problem was a little bit different : I could see processes PID but could not 
    connect to them (nor see the Main class)
    
    The source of the problem ? : hsperfdata was named "hsperfdata_Username" where my 
    login is "username".
    After closing all VM I deleted the dir and re-launched java new dir was created 
    with the good name "hsperfdata_username" and everything went OK.
    

Security

The most common troubles which prevent JConsole to connect to a remote application are linked to SSL/Security configurations. This does not apply if you connect to a local process using its PID, but will most certainly occur if you try to connect to a remote process where remote management was enabled using system properties.

When remote management is enabled using system properties, it is secure by default. This means that there are a minimum configuration steps to perform when you connect to a secure connector using JConsole. These steps are described in details here and there.

If possible, I'd advise to first start the application and JConsole without security enabled. If this works, then try again with security on.
The properties you need to pass in order to disable security are these:

-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false 

security issues are further explained here.

Firewall and RMI

If your application is behind a NAT or a Firewall, the firewall could be blocking JConsole connection. Some answers to common problems can be obtained here and there.

Update: A common problem with RMI and firewall is that the JMX default agent will not let you specify which port to use to export the server's RMI stub. To control the port on which the RMI stub is exported you will need to create your own JMXConnectorServer instead of using the connector used by the default agent.
Luis-Miguel Alventosa has just contributed two excellent blog entries which will teach you how to mimick the out-of-the-box JMX management agent and work with SSL/TLS-based RMI Socket Factories.
The first entry will teach you how to get full control over the JMX RMI Connector.
The second blog entry is not directly linked to JMX but may unveil some of the mysteries that surround the fine points of using SSL/TLS with RMI.
Update 2: The JDK 6 Monitoring and Management Guide has been updated with a section about Monitoring Applications through a Firewall. In particular it explains how to configure the two port numbers used by a JMX RMI Connector Server.
Update 3: I have also written a new entry explaining How To Connect Through Firewall Using JMX - Without modifying your server application.
Update 4: If you are trying to access a server which is behind a NAT - you will most probably have to start your server with the option
      -Djava.rmi.server.hostname=<public/NAT address>
  
so that the RMI stubs sent to the client contain the server's public address allowing it to be reached by the clients from the outside.
This was nicely summarized by DongWoo Lee in a very nice picture.

Switching on JConsole debug traces

JConsole has a -debug option that can provide useful information when trying to understand why it fails to connect. If these traces are not sufficient to make a diagnosis, you can also try to switch on the JMX and/or security traces as explained in the next section.

Switching on JMX and Security traces

Finally - here is probably the simplest way to understand what is going on: switch on the traces.

You can switch on JMX traces on the client side by calling jconsole with the following flags:

jconsole -J-Djava.util.logging.config.file=logging.properties

where logging.properties is your logging.properties file. This will activate both JMX and JMX remote traces on the client side (jconsole)

A few times ago I was blogging about how to switch on the JMX traces and gave a description of the JMX loggers and logging properties. Both JMX and JMX Remote traces can be activated in this way.

You can use the logging.properties file I have shown there. If you're trying to debug connection problems I'd recommend you switch the javax.management.remote logger to FINEST - and not to FINER as shown in the file.

If it's a security related problem (using SSL) you may also want to use:

-J-Djava.security.debug=all

or if you want to reduce the amount of security traces you can get a list of debug options by typing:

$ java -Djava.security.debug=help foo

all           turn on all debugging
access        print all checkPermission results
combiner      SubjectDomainCombiner debugging
jar           jar verification
logincontext  login context results
policy        loading and granting
provider      security provider debugging
scl           permissions SecureClassLoader assigns

The following can be used with access:

stack     include stack trace
domain    dumps all domains in context
failure   before throwing exception, dump stack
          and domain that didn't have permission

Note: Separate multiple options with a comma

See the Security Tutorial or Andrew's blog on Fine granularity diagnosis on security for more info on java.security.debug.

JConsole and BEA Weblogic Server 9.0

This article describes how to connect JConsole to Weblogic MBeanServer. Thanks Prashant for pointing this out!
The article however has one typo: the correct form of the JMXServiceURL to use on the client side is in fact:

   service:jmx:iiop:///jndi/iiop://host:port/weblogic.management.mbeanservers.runtime

Linux

If you're running on Linux, or if you're experimenting with SSH tunneling you might encounter issues with the way hostnames are resolved. See this comment and watch out for indications that RMI might use the address "127.0.1.1". This problem has also been described by Philipp Reichart (if you can read German). To solve it you might have to set the -Djava.rmi.server.hostname=<hostname or localhost or ip> property. If you're in local simply try with -Djava.rmi.server.hostname=localhost or -Djava.rmi.server.hostname=127.0.0.1. If you are running jconsole on a different host then have a look at the FAQ about JConsole on Linux.

Diagnosing what's wrong

Dustin Marx has also written a very good cookbook that might help you diagnose what's wrong by looking at the JMX Remote Exception stack trace.

Conclusion

As a conclusion, don't forget to check the FAQ which contains important information about common problems on Windows, Linux, and more common security configuration tricks.

Many thanks to all of you who are actively answering to posts in the Java and JMX forums, and contributing to the liveliness of the Java and JMX communities!


Cheers,
-- daniel

Comments:

Sir, I want to map a application running on the remote system on the internet. what technologies should I use for this? And how can I perform this task? do me a favour. Nikhil A. Shravane nikhil_shravane2004@rediffmail.com

Posted by guest on September 17, 2006 at 07:55 AM CEST #

If you control the remote system then you could arrange for your application to export its management interface through HTTPS.

JSR 262 - which is on-going, will standardize a WebService connector for JMX. You will find an early access of JSR 262 RI on java.net:
https://ws-jmx-connector.dev.java.net

Bear in mind however that this is a very early access - the WS Connector for JMX is still being discussed by the JSR 262 expert group, and the final version is likely to differ from the early access - with no guarantee of compatibility between them (the early access is just a means to get feedback from the community).

Another possibility would be for you to design an ad-hoc web interface for your application, using, for instance, NetBeans
http://www.netbeans.org/
to design and implement your WebService (client-side and server-side).

Yet another possibility could be to use proprietary JMX connectors over HTTPS - like for instance the legacy HTTP connector that comes with Java DMK 5.1:
http://java.sun.com/products/jdmk/index.jsp

If you don't control the remote system then there's nothing you can do - unless that remote system already offers a management interface as those described above.

best regards,
-- daniel

Posted by daniel on September 18, 2006 at 04:05 AM CEST #

Thanks Daniel, This information is really useful for me. I'll get back here. Regards

Posted by Nikhil A. Shravane on April 17, 2007 at 02:07 AM CEST #

Sorry, I'm one of the plenty developpers in trouble with remote JConsole.
Can you simply explain how run jconsole on a client machine to connect to an unmodifiable applicaton that run on a server. One and only one port on the server is open though firewall for mananing the application. Let's say we don't need security. The server adress is 1.2.3.4 and it's name is ww.simple.jconsole.net. and finally, the management port is the 8888.

Which are the flags for the server application ?
Which are the flags for the client ?

Posted by Willems on July 22, 2007 at 10:50 PM CEST #

Hi Willems, It was a bit long for answering within a comment, so I posted a follow-up entry here: Connecting Through Firewall Using JMX - Without modifying the server application.
Hope this will answer your question!
-- daniel

Posted by daniel on July 25, 2007 at 08:19 AM CEST #

Your page is very complete, but let me give you some more information

Jconsole 1.6 (and any software using jre1.6 + standard java connection ) against weblogic running 1.5 will not work, either vice-versa. Take a look.

http://bugs.sun.com/view_bug.do?bug_id=6614558

Posted by Alberto Navarro on November 09, 2007 at 01:41 AM CET #

@Alberto,
Hi Alberto, I am aware of this issue with the IIOP stack.
http://forum.java.sun.com/thread.jspa?threadID=5233754&tstart=0
As a temporary work around, can you configure the server to use JRMP instead of (or in addition to) IIOP?

Posted by daniel on November 09, 2007 at 02:46 AM CET #

I need some help understanding why jconsole is not connnecting to the local Java process. I am using JRE 1.6.0_03. I use jconsole -debug <java proc pid> to launch and this is the stack trace coming out:

java.io.IOException: Agent JAR loaded but agent failed to initialize
at sun.tools.jconsole.LocalVirtualMachine.loadManagementAgent(LocalVirtualMachine.java:251)
at sun.tools.jconsole.LocalVirtualMachine.startManagementAgent(LocalVirtualMachine.java:83)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:327)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:297)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:279)
Caused by: com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
at sun.tools.jconsole.LocalVirtualMachine.loadManagementAgent(LocalVirtualMachine.java:245)
... 4 more

Any suggestions on how to debug further? Thanks.

Posted by Ming Yang on August 22, 2008 at 11:32 AM CEST #

Hi Daniel,

This blog have really very good and important info. Thanks....!!!

Actually I am having problem accessing my application whivh is behind the NAT device.

I am having following code snippet:

----------------------------------------------
env.put( Context.INITIAL_CONTEXT_FACTORY, JBOSS_CONTEXT_FACTORY );
env.put( Context.URL_PKG_PREFIXES, JBOSS_URL_PKG_PREFIXES );
env.put( Context.PROVIDER_URL, host + ":" + port );

// Disable Discovery
env.put( NamingContext.JNP_DISABLE_DISCOVERY, "true" );

// Socket timeout, set to one minute.
env.put( TimedSocketFactory.JNP_SO_TIMEOUT, "60000" );

// Initial connect timeout, set to one minute.
env.put( TimedSocketFactory.JNP_TIMEOUT, "60000" );

Context context = new InitialContext( env );
adaptor_ = ( RMIAdaptor )context.lookup( "jmx/invoker/RMIAdaptor" );
host_ = host;
port_ = port;
}
catch( NamingException ne )
{
cleanup(); // clear security context in case of error

throw new JmxRemoteException( ne );
}
--------------------------------------------------

And I am getting the following exception and stack trace:

-------------------------------------------------
FINE: Creating JMXRemoteException for javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: 11.22.33.444; nested exception is:
java.net.ConnectException: Connection timed out: connect]
<R [RSMScheduler_Worker-132],07/24/08 04:14:50 UTC,STDERR> Jul 24, 2008 4:14:50 AM com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException JmxRemoteException
FINE: Creating JMXRemoteException for java.rmi.ConnectException: Connection refused to host: 11.22.33.444; nested exception is:
java.net.ConnectException: Connection timed out: connect
<R [RSMScheduler_Worker-132],07/24/08 04:14:50 UTC,STDERR> Jul 24, 2008 4:14:50 AM com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException JmxRemoteException
FINE: Creating JMXRemoteException for java.net.ConnectException: Connection timed out: connect
<R [RSMScheduler_Worker-132],07/24/08 04:14:50 UTC,STDERR> Jul 24, 2008 4:14:50 AM com.bmc.patrol.patsdk.solutions.jmx.JmxParamlet doExecute()
FINEST: Jmx remote exception:
com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException: javax.naming.CommunicationException : null
at com.bmc.patrol.patsdk.lib.jmx.JBossJmxClient.connect(JBossJmxClient.java:101)
at com.bmc.patrol.patsdk.solutions.jmx.JmxParamlet.doExecute(JmxParamlet.java:271)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException: java.rmi.ConnectException : Connection refused to host: 11.22.33.444; nested exception is:
java.net.ConnectException: Connection timed out: connect
... 9 more
Caused by: com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException: java.net.ConnectException : Connection timed out: connect
----------------------------------------------------

172.87.65.145 is the NAT address and 11.22.33.444 is the real IP address of the host that is behind the NAT device.
There is need to use NAT'ted addresses is a must in my environment. As a result, I can not make use of the real IP address. Only the NAT address is available.

As I have specified NAT address in the connection URL but in trace logs its giving error for real IP. Please suggest. How can I overcome this situation? What are the code changes i need to do for my requirement. Please suggest, its really urgent for me.

Thanks.
Nilesh

Posted by Nilesh on September 06, 2008 at 06:45 AM CEST #

Please suggest regarding the above

Posted by guest on September 07, 2008 at 11:46 PM CEST #

Hi,

You might need to pass the
-Djava.rmi.server.hostname=<nataddress> option on your server command line.

My guess is that the RMI stubs created by the server contain its "real" address - which would explain the errors you're seeing.

Hope this helps,

-- daniel

Posted by daniel on September 10, 2008 at 08:13 AM CEST #

Thanks for your article!
I have weird problem, hopefully someone will be able to help me to solve it.

My JMX monitoring application (Zenoss / ZenJMX) is located on server A. My Jboss is running on server B. There is a firewall between A and B, everything from A to B is allowed.
The problem: I have to run jconsole from server C (the same network as B) - w/o this my monitor doesn't work. I have to re-run jconsole after every jboss restart.
I tried to analyze sniffer logs, but w/o success...

TIA, Vitaly

Posted by Vitaly on December 02, 2008 at 01:19 AM CET #

ineed configuertion for simpie net workmengement
portocl and how to make connect between hosttohost

Posted by ayman siliman ali on October 15, 2009 at 03:05 AM CEST #

I doubt I will hear anything because this blog is so old, but... I guess I'm missing something fundamental - when I specify logging enabled (jconsole -J-Djava.util.logging.config.file=./Desktop/jmxLogging.properties), where does the log output go? I can't find a log file anywhere and nothing comes out on the terminal from which I invoked jconsole. My connection is failing and I don't know why. I sure could use some visibility into the problem. Thanks.

Posted by guest on May 14, 2011 at 02:36 PM CEST #

@guest: Have faith! :-) ---------------------------------------------------------------------------------------------------------------------------------------------------- It is possible to configure the log target in your logging.properties file. I usually configure mine to go to stderr. I believe you do that with the two lines below - which you should put first in your logging.properties file: ---------------------------------------------------------------------------------------------------------------------------------------------------- java.util.logging.ConsoleHandler.level = FINEST ---------------------------------------------------------------------------------------------------------------------------------------------------- java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter ---------------------------------------------------------------------------------------------------------------------------------------------------- Be aware however that jconsole is not very verbose :-( Hope this helps, ---------------------------------------------------------------------------------------------------------------------------------------------------- -- daniel

Posted by guest on May 24, 2011 at 05:14 PM CEST #

Hi Daniel, I’m impressed that you responded to this since the blog was so old. Am I the only one still using JMX? ;-) Anyway, I was able to get around the problem I wrote about on your blog. I don’t remember exactly, but I believe putting ‘handlers=java.util.logging.ConsoleHandler ‘ as the first line did the trick. The complete logging.properties file is below: handlers = java.util.logging.ConsoleHandler .level = FINEST java.util.logging.ConsoleHandler.level = FINEST java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter javax.management.level = FINEST javax.management.remote.level = FINEST Now I’m stuck with something different. I know that JMX is working because I can connect to my MX4J adaptor with no problem. But I still cannot connect using jconsole. Here's what I get: May 24, 2011 7:48:00 PM RMIConnector connect FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.253.96.93:8081/jmxrmi] connecting... May 24, 2011 7:48:00 PM RMIConnector connect FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.253.96.93:8081/jmxrmi] finding stub... May 24, 2011 7:49:16 PM RMIConnector connect FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.253.96.93:8081/jmxrmi] connecting stub... May 24, 2011 7:49:16 PM RMIConnector connect FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.253.96.93:8081/jmxrmi] getting connection... May 24, 2011 7:50:31 PM RMIConnector connect FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.253.96.93:8081/jmxrmi] failed to connect: java.rmi.ConnectException: Connection refused to host: 10.253.96.93; nested exception is: java.net.ConnectException: Operation timed out I'm using this as the service URL: service:jmx:rmi:///jndi/rmi://10.253.96.93:8081/jmxrmi And the JVM is started with the following defines: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8081 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.253.96.93 So if you can provide any insight into this, it would be appreciated as well. :-) Bill

Posted by guest on May 24, 2011 at 07:34 PM CEST #

Hi Bill, -------------------------------------------------------------------------------------------------------------------------------------------------------- Only two things come to my mind: -------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Can you ssh to 10.253.96.93? Maybe the host is not reachable. Or maybe the JVM failed to bind to port 8081? Try to start it with another port number. -------------------------------------------------------------------------------------------------------------------------------------------------------- 2. You could also try to pass 10.253.96.93:8081 on the jconsole command line: jconsole 10.253.96.93:8081 -------------------------------------------------------------------------------------------------------------------------------------------------------- I believe in that case jconsole will apply some special algorithmic to determine whether it should use SSL or not - so it might just do the trick. -------------------------------------------------------------------------------------------------------------------------------------------------------- -- daniel

Posted by guest on May 25, 2011 at 01:45 AM CEST #

Daniel, I'm able to ssh to the host. I can also telnet to port 8081. I tried passing the IP:port on the jconsole command line and it spewed TONS of logging to the console. Most of appears to be UI related however - I don't have time right now to sort through it all. It still did not connect, however. I suspect it has something to do with the way the network is set up (?). I'm on a VPN (don't know if that is relevant). Anyway, I will investigate further and let you know. Thanks!

Posted by guest on May 25, 2011 at 07:16 AM CEST #

Hi Daniel,

Im having a problem with Quartz manager,
I installed and login page appeared, i put the localhost and port 9520 (9510) as default but i could not log in, the error massage is "there was a problem in connecting to instance" i have read some guides, it said that we need anable JMX or Jconsole, please show me where i can find this ? or do you have anydocument about that pls share me

Thanks in Advances

QT

Posted by guest on August 24, 2012 at 09:31 AM CEST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Daniel Fuchs blogs on Scene Builder, JMX, SNMP, Java, etc...

The views expressed on this blog are those of the author and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today