X

Do Not Kill – Use JRockit JRCMD

Guest Author

moses_do_not_killOne extremely valuable skill you should learn as someone who works with Java is to understand a thread dump.  One of my customers this week had a situation where their web application was acting up and we were able to see some misbehaving code very quickly by looking inside their application to see what each thread was doing.  It might be intimidating to you at first, but I can’t emphasize enough how much reading thread dumps will help your troubleshooting skills.  This post will illustrate a simple way to do this with the JRockit JVM.  In a subsequent post I’ll give some pointers that would work on other JVM’s and you can always refer to my previous post on JVM tools

Now my first preference is to always use JRockit Mission Control, which is an Eclipse RCP based tool that ships with JRockit that tells you virtually everything that’s going on inside the JVM in a very easy-to-understand way.  But let’s say you are logged into a remote system that is running in production that doesn’t have graphics libraries and you need to be able to maneuver on the command line.  To get thread dumps quickly, cleanly and easily with JRockit it’s much easier to use the <JROCKIT_INSTALL>/bin/jrcmd command than kill –3 or Cntrl-Break.  This way output can be redirected to a file of your choosing without having to know a bunch of command line tricks.  There are so many other powerful commands that you can use from with JRCMD, but let’s start simple and I think you’ll appreciate how easy this is.

From any shell on a machine where a JRockit JVM is running simply call JRCMD with no arguments.  The output is a list of every process id and command line of each Java command on this system.

Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
D:\>cd Oracle
 
D:\Oracle>cd wls11g\jrockit_160_14_R27.6.5-32\bin
 
D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd
 
3792 jrockit.tools.jrcmd.JrCmd
 
6016 com.pointbase.net.netServer /noconsole /port:9082 /d:3 /pointbase.ini=D:\Oracle\wls11g\USER_P~1\domains\MEDREC~1\pointbase.ini
 
4512 weblogic.Server
 


If you want to see all the available commands for a particular process, simply add the process id and the “help” command as arguments:

D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 help
 
4512:
 
The following commands are available:
 
        kill_rmp_server
 
        start_rmp_server
 
        kill_management_server
 
        start_management_server
 
        checkjrarecording
 
        stopjrarecording
 
        startjrarecording
 
        print_object_summary
 
        memleakserver
 
        print_codegenlist
 
        print_class_summary
 
        run_optfile
 
        dump_codelayout
 
        dump_codelist
 
        dump_codemap
 
        print_utf8pool
 
        print_properties
 
        print_threads
 
        datadump_request
 
        runsystemgc
 
        runfinalization
 
        heap_diagnostics
 
        oom_diagnostics
 
        print_exceptions
 
        version
 
        timestamp
 
        command_line
 
        memprof
 
        sanity
 
        verbosity
 
        set_filename
 
        help
 
        print_memusage
 



Wow, there are a lot of commands available.  What if you do not know all the options for how to use a command?  For help with any specific command use 'help <command>'.  So here is how to see the help for the print_threads command:

D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 help print_threads
 
4512:
 
Print all threads with stacktraces.
 
        nativestack     - include native frames in the stacktrace (bool,
 
                          false)
 
        javastack       - print java stack frames (bool, true)
 
        monitors        - print lock information (bool, true)
 
        jvmmonitors     - include the jvm internal monitors (bool, false)
 
        internal        - print JRockit internal threads (bool, true)
 
        concurrentlocks - print java.util.concurrent locks (bool, false)
 
        compact         - print all threads with the same stacktrace together
 
                          (will not print nativestack or monitors) (bool,
 
                          false)
 



So let’s say the defaults are ok, we can now send the thread dump to a file easily by redirecting the output.  Often times more than 1 thread dump is useful, if they are several moments apart you can tell if a certain thread is “stuck” on a piece of code.

D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 print_threads > d:\temp\threaddump1.txt
 
D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 print_threads > d:\temp\threaddump2.txt
 



Let’s take a look at threaddump1.txt.  You’ll see every thread in the JVM and the code stack it took to get to that point.

4512:
 
===== FULL THREAD DUMP ===============
Mon Feb 08 14:18:55 2010
BEA JRockit(R) R27.6.5-32_o-121899-1.6.0_14-20091001-2107-windows-ia32
 
"Main Thread" id=1 idx=0x4 tid=7192 prio=5 alive, in native, waiting
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x15127DF0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:849)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x15127DF0[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:465)
    at weblogic/Server.main(Server.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"(Signal Handler)" id=2 idx=0x8 tid=7248 prio=5 alive, in native, daemon
 
"(GC Main Thread)" id=3 idx=0xc tid=7900 prio=5 alive, in native, native_waiting, daemon
 
"(GC Worker Thread 1)" id=? idx=0x10 tid=6548 prio=5 alive, in native, daemon
 
"(GC Worker Thread 2)" id=? idx=0x14 tid=6344 prio=5 alive, in native, daemon
 
"(Code Generation Thread 1)" id=4 idx=0x18 tid=3748 prio=5 alive, in native, native_waiting, daemon
 
"(Code Optimization Thread 1)" id=5 idx=0x1c tid=4364 prio=5 alive, in native, native_waiting, daemon
 
"(VM Periodic Task)" id=6 idx=0x20 tid=3024 prio=10 alive, in native, daemon
 
"(Attach Listener)" id=7 idx=0x24 tid=7988 prio=5 alive, in native, daemon
 
"Finalizer" id=8 idx=0x28 tid=5232 prio=8 alive, in native, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
    at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
    at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"Reference Handler" id=9 idx=0x2c tid=7412 prio=10 alive, in native, native_waiting, daemon
    at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
    at java/lang/ref/Reference.access$100(Reference.java:11)
    at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"(Sensor Event Thread)" id=10 idx=0x30 tid=7088 prio=5 alive, in native, daemon
 
"Timer-0" id=13 idx=0x34 tid=4852 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x14DDC788[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at java/util/TimerThread.mainLoop(Timer.java:483)
    ^-- Lock released while waiting: java/util/TaskQueue@0x14DDC788[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"Timer-1" id=14 idx=0x38 tid=8100 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x14DDCB40[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/util/TimerThread.mainLoop(Timer.java:509)
    ^-- Lock released while waiting: java/util/TaskQueue@0x14DDCB40[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=924 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x14F4A3A0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x14F4A3A0[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"weblogic.time.TimeEventGenerator" id=16 idx=0x40 tid=7480 prio=9 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x14E49080[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
    ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x14E49080[fat lock]
    at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"JMAPI event thread" id=17 idx=0x44 tid=7080 prio=5 alive, in native, native_waiting, daemon
 
"weblogic.timers.TimerThread" id=18 idx=0x48 tid=2812 prio=9 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/timers/internal/TimerThread@0x150324B8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:267)
    ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x150324B8[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=19 idx=0x4c tid=7476 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x14F4A478[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x14F4A478[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"Thread-7" id=22 idx=0x50 tid=4816 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x14DDD658
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=23 idx=0x54 tid=7536 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x58 tid=6252 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x5c tid=4064 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"VDE Transaction Processor Thread" id=28 idx=0x60 tid=4904 prio=2 alive, in native, waiting, daemon
    -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x15036280[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
    ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x15036280[fat lock]
    at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"DoSManager" id=30 idx=0x68 tid=7072 prio=6 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"Thread-11" id=31 idx=0x6c tid=3288 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x14DDD868
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=32 idx=0x70 tid=7216 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x14F4A550[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x14F4A550[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=33 idx=0x74 tid=7764 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x0BAA3938[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0BAA3938[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=45 idx=0xa4 tid=8104 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x089001E8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x089001E8[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"DynamicListenThread[MedRec Local Network Channel]" id=46 idx=0xa8 tid=7452 prio=9 alive, in native, daemon
    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390)
    ^-- Holding lock: java/net/SocksSocketImpl@0x09374A98[biased lock]
    at java/net/ServerSocket.implAccept(ServerSocket.java:453)
    at java/net/ServerSocket.accept(ServerSocket.java:421)
    at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
    at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:528)
    at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)
    at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:171)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"DynamicListenThread[Default]" id=47 idx=0xac tid=4092 prio=9 alive, in native, daemon
    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390)
    ^-- Holding lock: java/net/SocksSocketImpl@0x09380038[biased lock]
    at java/net/ServerSocket.implAccept(ServerSocket.java:453)
    at java/net/ServerSocket.accept(ServerSocket.java:421)
    at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
    at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:528)
    at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)
    at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:171)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"DynamicSSLListenThread[DefaultSecure]" id=48 idx=0xb0 tid=6196 prio=9 alive, in native, daemon
    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390)
    ^-- Holding lock: java/net/SocksSocketImpl@0x093978B8[biased lock]
    at java/net/ServerSocket.implAccept(ServerSocket.java:453)
    at javax/net/ssl/impl/SSLServerSocketImpl.accept()Ljava/net/Socket;(Unknown Source)
    at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:528)
    at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)
    at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:171)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"Java2D Disposer" id=49 idx=0xb4 tid=6356 prio=10 alive, in native, waiting, daemon
    -- Waiting for notification on: java/lang/ref/ReferenceQueue$Lock@0x09D79E60[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:118)
    ^-- Lock released while waiting: java/lang/ref/ReferenceQueue$Lock@0x09D79E60[fat lock]
    at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:134)
    at sun/java2d/Disposer.run(Disposer.java:125)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
"AWT-Windows" id=50 idx=0xbc tid=8000 prio=6 alive, in native, daemon
    at sun/awt/windows/WToolkit.eventLoop()V(Native Method)
    at sun/awt/windows/WToolkit.run(WToolkit.java:291)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
 
===== END OF THREAD DUMP ===============
 



484302 cover.indd The 2009 book Professional Oracle WebLogic Server has excellent information about how to read WLS thread dumps starting on page 686.  But for those of you that want Cliff Notes, STUCK threads are bad.  A thread that has the name STUCK in it’s name has been identified as not being returned to the thread pool for a configured amount of time, which defaults to 10 minutes.  In a subsequent post I’ll show you how to use JRockit Mission Control to look at threads which has some nice tooling for sorting through this detail.

Also check out some other posts related to this:

http://blogs.oracle.com/staffan/2008/08/useful_tool_jrcmd.html

http://blogs.oracle.com/staffan/2008/08/more_jrcmd_tricks.html

http://www.oracle.com/technology/pub/articles/dev2arch/2007/12/jrockit-tuning3.html

Join the discussion

Comments ( 9 )
  • Pete Monday, February 15, 2010
    James,
    This is a very useful post, thanks.
    One problem I do have with jrcmd is invoking it on a remote JVM.
    We have a series of admin commands which are windows batch files. We run these from the admin server and running a "dump.cmd" will use weblogic.Admin and connect to each of the JVM's in the cluster and issue a thread_dump command.
    weblogic.Admin needs to have a thread available for it to run, and in some cases this isn't possible - when things have gone really bad!
    JRCMD on the other hand will always get a thread dump (in my experience) but what I haven't figured out how to do is say, run a batch file from the admin server machine and this will issue a jrcmd against all members of the cluster. At the moment, I have to remotely log on to each server and issue the command.
    Hopefully that will make sense and if anyone has any suggestions....
    Pete
  • james.bayer Monday, February 15, 2010
    Well I have two good answers for you, one is a blog entry from a JRockit engineer Marcus Hirt with sample code how to issue JRCMD-like features on a remote machine:
    http://blogs.oracle.com/hirt/2008/08/jrcmd_goes_remote_r271_edition.html
    The second approach would be to use the -Xmanagement options to enable you to send remote commands to remote machines directly from JRockit Mission Control. I'll show this shortly in another post.
    http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html#wp1030433
    If you have questions in the meantime, definitely post in the JRockit OTN forum and you should get good advice:
    http://forums.oracle.com/forums/forum.jspa?forumID=561
    Good luck,
    James
  • Kevin Thursday, April 7, 2011
    Hi James,
    I just have a simple question, whether we can use JRCMD to take t he thread dump for SUN JVM/HOTSPOT ?
    Regards.
  • james.bayer Thursday, April 7, 2011
    Kevin,
    No you can't use JRCMD with Hotspot, you need to use other approaches such as jps (for retrieving PIDs) and jstack for getting thread dumps. These tools are in the JDK/bin directory.
    Thanks, James
  • subhash Thursday, August 25, 2011

    i can't use JRCMD when the weblogic instances (9.2 mp3) are running under NodeManager control on windows 2003 R2....any ideas??..thanks


  • james.bayer Thursday, August 25, 2011

    Subhash,

    When running under Windows Servers, it's likely a user permissions issue with the user that is running the service not being the same as the user running JRCMD. I suggest either make those the same OR use a remote network listener on JRockit by passing the following Java options (or similar): -Xmanagement:port=7091,ssl=false,authenticate=false which opens a port on 7091 on the machine where the Server is running that you can connect to as a defined connection. Cheers, James


  • subhash Thursday, August 25, 2011

    Thanks james; i appreciate your quick response.

    weblogic instance is running with 'system' account and i was using jrcmd with my domain account. It could be possible as you said.

    But about your other alternative work around, will it work with out having a JRockit Mission Control? I was using jrcmd all along with out having JRockit Mission Control in the env.


  • james.bayer Thursday, August 25, 2011

    Subhash,

    I think JRCMD only works against the local machine based off looking at some docs.

    http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/ctrlbreakhndlr.html#wp1001743

    In order to use the remote commands (even if talking to a local port) I think you have to use Mission Control, which is the JRockit/bin directory with the jrmc command. You could always ask in the OTN JRockit forum to see what they recommend.

    https://forums.oracle.com/forums/forum.jspa?forumID=561

    Thanks,

    James


  • Subhash Friday, August 26, 2011

    i am using JRMC against local machine only but still couldn't connect. but i guess it is known limitation from that document.

    When using jrcmd on Windows, you need to run the Java process and jrcmd from the same Windows station. If you run the Java process as a Windows service, and run jrcmd on your desktop, it will not work, since they are running in two separate environments.

    thanks anyway...i will have to think about buying a license for mission control.hope it should be useful. i appreciate your responses.


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.