Thread Dumps in Glassfish

We have totally changed the way that Thread Dumps are collected in Glassfish. Forget everything about how it used to be done.
  • You need not run the servers in native mode. The same procedure works in native and non-native mode
  • Thread dumps are available for all 3 kinds of servers -- Domain Administration Servers, Node Agents and Server Instances
  • The new behavior is totally automatic, you don't need to do anything other than send a signal to the JVM to trigger it.

How to Trigger a Thread Dump in Windows

  • Get the PID of the server with jps.
  • Run sendsignal giving it the PID of the server.

How to Trigger a Thread Dump in UNIX and Linux

  • Get the PID of the server with jps.
  • Run kill -3 PID

Where did my Thread Dump Go?

The Thread Dump is appended to a file named jvm.log in the same directory as the normal server logfile. E.g. /gf/domains/domain1/logs/jvm.log

Important: the jvm.log file is deleted every time the server is restarted. If you want to keep the contents you will need to copy the file.

Comments:

Great Stuff, Byron. Thanks.

You should also explain the "asadmin generate-jvm-report" and "jstack" commands. The work that you've done is important because "jstack" is NOT available in JAVA SE 5.0 on Windows!

Thanks!

Posted by Kedar on April 23, 2007 at 02:39 PM PDT #

For improved diagnostics you might want to check out JXInsight's support for Glassfish and all other major application servers. We have a FREE development only edition available from our website. Some recent blog postings showing the integration. Sun SJAS/Glassfish Diagnostics http://blog.jinspired.com/?p=43 http://blog.jinspired.com/?p=44 Diagnostics Tags http://blog.jinspired.com/?p=52 JVMInsight Thread Dumps http://blog.jinspired.com/?p=19 regards, William Louth JXInsight Product Architect JInspired "Java EE tuning, tracing, testing and monitoring with JXInsight" http://www.jinspired.com

Posted by William Louth on April 26, 2007 at 12:51 AM PDT #

[Trackback] Most of run into bugs where tests "hang". Here are some nice tools and tips I found to obtain and analyze thread dumps.

Posted by Bhakti Mehta's Blog on May 25, 2007 at 06:09 AM PDT #

@Kedar - jstack is available on Windows since jdk6. Note that jstack in jdk6 is very different from the jstack utility in jdk5. The utility that shipped in jdk5 was suited for post mortem debugging of stack traces but was not always suitable for live process analysis. Another useful feature in the jdk6 version is the -l option to help finding deadlock involving monitors and ownable synchronizers (ie: code using j.u.c.locks).

Posted by Alan on May 25, 2007 at 05:27 PM PDT #

Thanks, Alan. I will read more about it. But frankly, in Java EE 5 implementations, we have to support Java SE 5 as the minimum supported platform.

That brings up a good question - What is a good way to backport good features of more recent JDK versions into previous versions? I know you guys do a great job of that, but still, we end up writing some code that becomes obsolete or invent hack (e.g. reflection, if we are run on more recent JDK). Is there a better suggestion? Another instance of this is java.net.IDN class that is available only in JDK 6. See: https://glassfish.dev.java.net/issues/show_bug.cgi?id=3371 for an interesting use case.

Posted by Kedar on July 25, 2007 at 05:10 PM PDT #

I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

Sarah

http://www.lyricsdigs.com

Posted by Sarah on March 09, 2009 at 07:24 PM PDT #

I tried SendSignal <glassfish-pid> on my WinXP box, but got nothing in jvm.log. It seems that the file is locked, so...

Btw, jstack <glassfish-pid> works nicely on my WinXP box. So no need to use SendSignal.exe

Posted by Per Lindberg on March 06, 2011 at 10:59 PM PST #

In GF 3.2 (not released yet) the -Xrs option has been added to the servers by default. This option results in the server ignoring signals. This is important to set if you want your server to survive your logging off \*at least on Windows).

So -- use jstack to get thread dumps!

Posted by Byron Nevins on March 07, 2011 at 12:57 AM PST #

Well, jstack is nice, but doesn't work for me on Linux:

> jstack 30760

30760: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Ok, so I retry with -F. But that does not work very well either:

> jstack -F 30760
Attaching to process ID 30760, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.1-b02
Deadlock Detection:

No deadlocks found.

Thread 16527: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)

Posted by Per Lindberg on March 07, 2011 at 05:12 PM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

ByronNevins

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today