Do Not Kill – Use JRockit JRCMD

moses_do_not_killOne extremely valuable skill you should learn as someone who works with Java is to understand a thread dump.  One of my customers this week had a situation where their web application was acting up and we were able to see some misbehaving code very quickly by looking inside their application to see what each thread was doing.  It might be intimidating to you at first, but I can’t emphasize enough how much reading thread dumps will help your troubleshooting skills.  This post will illustrate a simple way to do this with the JRockit JVM.  In a subsequent post I’ll give some pointers that would work on other JVM’s and you can always refer to my previous post on JVM tools

Now my first preference is to always use JRockit Mission Control, which is an Eclipse RCP based tool that ships with JRockit that tells you virtually everything that’s going on inside the JVM in a very easy-to-understand way.  But let’s say you are logged into a remote system that is running in production that doesn’t have graphics libraries and you need to be able to maneuver on the command line.  To get thread dumps quickly, cleanly and easily with JRockit it’s much easier to use the <JROCKIT_INSTALL>/bin/jrcmd command than kill –3 or Cntrl-Break.  This way output can be redirected to a file of your choosing without having to know a bunch of command line tricks.  There are so many other powerful commands that you can use from with JRCMD, but let’s start simple and I think you’ll appreciate how easy this is.

From any shell on a machine where a JRockit JVM is running simply call JRCMD with no arguments.  The output is a list of every process id and command line of each Java command on this system.

Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

D:\>cd Oracle

 

D:\Oracle>cd wls11g\jrockit_160_14_R27.6.5-32\bin

 

D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd

 

3792 jrockit.tools.jrcmd.JrCmd

 

6016 com.pointbase.net.netServer /noconsole /port:9082 /d:3 /pointbase.ini=D:\Oracle\wls11g\USER_P~1\domains\MEDREC~1\pointbase.ini

 

4512 weblogic.Server

 


If you want to see all the available commands for a particular process, simply add the process id and the “help” command as arguments:



D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 help

 

4512:

 

The following commands are available:

 

        kill_rmp_server

 

        start_rmp_server

 

        kill_management_server

 

        start_management_server

 

        checkjrarecording

 

        stopjrarecording

 

        startjrarecording

 

        print_object_summary

 

        memleakserver

 

        print_codegenlist

 

        print_class_summary

 

        run_optfile

 

        dump_codelayout

 

        dump_codelist

 

        dump_codemap

 

        print_utf8pool

 

        print_properties

 

        print_threads

 

        datadump_request

 

        runsystemgc

 

        runfinalization

 

        heap_diagnostics

 

        oom_diagnostics

 

        print_exceptions

 

        version

 

        timestamp

 

        command_line

 

        memprof

 

        sanity

 

        verbosity

 

        set_filename

 

        help

 

        print_memusage

 


Wow, there are a lot of commands available.  What if you do not know all the options for how to use a command?  For help with any specific command use 'help <command>'.  So here is how to see the help for the print_threads command:



D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 help print_threads

 

4512:

 

Print all threads with stacktraces.

 

        nativestack     - include native frames in the stacktrace (bool,

 

                          false)

 

        javastack       - print java stack frames (bool, true)

 

        monitors        - print lock information (bool, true)

 

        jvmmonitors     - include the jvm internal monitors (bool, false)

 

        internal        - print JRockit internal threads (bool, true)

 

        concurrentlocks - print java.util.concurrent locks (bool, false)

 

        compact         - print all threads with the same stacktrace together

 

                          (will not print nativestack or monitors) (bool,

 

                          false)

 


So let’s say the defaults are ok, we can now send the thread dump to a file easily by redirecting the output.  Often times more than 1 thread dump is useful, if they are several moments apart you can tell if a certain thread is “stuck” on a piece of code.



D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 print_threads > d:\temp\threaddump1.txt

 

D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32\bin>jrcmd 4512 print_threads > d:\temp\threaddump2.txt

 


Let’s take a look at threaddump1.txt.  You’ll see every thread in the JVM and the code stack it took to get to that point.



4512:

 

===== FULL THREAD DUMP ===============

Mon Feb 08 14:18:55 2010

BEA JRockit(R) R27.6.5-32_o-121899-1.6.0_14-20091001-2107-windows-ia32

 

"Main Thread" id=1 idx=0x4 tid=7192 prio=5 alive, in native, waiting

    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x15127DF0[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:849)

    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x15127DF0[fat lock]

    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:465)

    at weblogic/Server.main(Server.java:67)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"(Signal Handler)" id=2 idx=0x8 tid=7248 prio=5 alive, in native, daemon

 

"(GC Main Thread)" id=3 idx=0xc tid=7900 prio=5 alive, in native, native_waiting, daemon

 

"(GC Worker Thread 1)" id=? idx=0x10 tid=6548 prio=5 alive, in native, daemon

 

"(GC Worker Thread 2)" id=? idx=0x14 tid=6344 prio=5 alive, in native, daemon

 

"(Code Generation Thread 1)" id=4 idx=0x18 tid=3748 prio=5 alive, in native, native_waiting, daemon

 

"(Code Optimization Thread 1)" id=5 idx=0x1c tid=4364 prio=5 alive, in native, native_waiting, daemon

 

"(VM Periodic Task)" id=6 idx=0x20 tid=3024 prio=10 alive, in native, daemon

 

"(Attach Listener)" id=7 idx=0x24 tid=7988 prio=5 alive, in native, daemon

 

"Finalizer" id=8 idx=0x28 tid=5232 prio=8 alive, in native, native_waiting, daemon

    at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)

    at jrockit/memory/Finalizer.access$500(Finalizer.java:12)

    at jrockit/memory/Finalizer$4.run(Finalizer.java:159)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"Reference Handler" id=9 idx=0x2c tid=7412 prio=10 alive, in native, native_waiting, daemon

    at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)

    at java/lang/ref/Reference.access$100(Reference.java:11)

    at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"(Sensor Event Thread)" id=10 idx=0x30 tid=7088 prio=5 alive, in native, daemon

 

"Timer-0" id=13 idx=0x34 tid=4852 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: java/util/TaskQueue@0x14DDC788[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at java/util/TimerThread.mainLoop(Timer.java:483)

    ^-- Lock released while waiting: java/util/TaskQueue@0x14DDC788[fat lock]

    at java/util/TimerThread.run(Timer.java:462)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"Timer-1" id=14 idx=0x38 tid=8100 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: java/util/TaskQueue@0x14DDCB40[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/util/TimerThread.mainLoop(Timer.java:509)

    ^-- Lock released while waiting: java/util/TaskQueue@0x14DDCB40[fat lock]

    at java/util/TimerThread.run(Timer.java:462)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=924 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/work/ExecuteThread@0x14F4A3A0[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)

    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x14F4A3A0[fat lock]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"weblogic.time.TimeEventGenerator" id=16 idx=0x40 tid=7480 prio=9 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x14E49080[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)

    ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x14E49080[fat lock]

    at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"JMAPI event thread" id=17 idx=0x44 tid=7080 prio=5 alive, in native, native_waiting, daemon

 

"weblogic.timers.TimerThread" id=18 idx=0x48 tid=2812 prio=9 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/timers/internal/TimerThread@0x150324B8[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:267)

    ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x150324B8[fat lock]

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=19 idx=0x4c tid=7476 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/work/ExecuteThread@0x14F4A478[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)

    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x14F4A478[fat lock]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"Thread-7" id=22 idx=0x50 tid=4816 prio=5 alive, in native, parked, daemon

    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x14DDD658

    at jrockit/vm/Locks.park0(J)V(Native Method)

    at jrockit/vm/Locks.park(Locks.java:2517)

    at sun/misc/Unsafe.park(ZJ)V(Native Method)

    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)

    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)

    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)

    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)

    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=23 idx=0x54 tid=7536 prio=5 alive, in native, daemon

    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)

    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)

    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x58 tid=6252 prio=5 alive, in native, daemon

    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)

    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)

    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x5c tid=4064 prio=5 alive, in native, daemon

    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)

    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)

    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"VDE Transaction Processor Thread" id=28 idx=0x60 tid=4904 prio=2 alive, in native, waiting, daemon

    -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x15036280[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)

    ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x15036280[fat lock]

    at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"DoSManager" id=30 idx=0x68 tid=7072 prio=6 alive, in native, sleeping, native_waiting, daemon

    at java/lang/Thread.sleep(J)V(Native Method)

    at com/octetstring/vde/DoSManager.run(DoSManager.java:433)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"Thread-11" id=31 idx=0x6c tid=3288 prio=5 alive, in native, parked, daemon

    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x14DDD868

    at jrockit/vm/Locks.park0(J)V(Native Method)

    at jrockit/vm/Locks.park(Locks.java:2517)

    at sun/misc/Unsafe.park(ZJ)V(Native Method)

    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)

    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)

    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)

    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)

    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=32 idx=0x70 tid=7216 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/work/ExecuteThread@0x14F4A550[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)

    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x14F4A550[fat lock]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=33 idx=0x74 tid=7764 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/work/ExecuteThread@0x0BAA3938[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)

    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0BAA3938[fat lock]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=45 idx=0xa4 tid=8104 prio=5 alive, in native, waiting, daemon

    -- Waiting for notification on: weblogic/work/ExecuteThread@0x089001E8[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)

    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x089001E8[fat lock]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"DynamicListenThread[MedRec Local Network Channel]" id=46 idx=0xa8 tid=7452 prio=9 alive, in native, daemon

    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390)

    ^-- Holding lock: java/net/SocksSocketImpl@0x09374A98[biased lock]

    at java/net/ServerSocket.implAccept(ServerSocket.java:453)

    at java/net/ServerSocket.accept(ServerSocket.java:421)

    at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)

    at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:528)

    at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)

    at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:171)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"DynamicListenThread[Default]" id=47 idx=0xac tid=4092 prio=9 alive, in native, daemon

    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390)

    ^-- Holding lock: java/net/SocksSocketImpl@0x09380038[biased lock]

    at java/net/ServerSocket.implAccept(ServerSocket.java:453)

    at java/net/ServerSocket.accept(ServerSocket.java:421)

    at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)

    at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:528)

    at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)

    at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:171)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"DynamicSSLListenThread[DefaultSecure]" id=48 idx=0xb0 tid=6196 prio=9 alive, in native, daemon

    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390)

    ^-- Holding lock: java/net/SocksSocketImpl@0x093978B8[biased lock]

    at java/net/ServerSocket.implAccept(ServerSocket.java:453)

    at javax/net/ssl/impl/SSLServerSocketImpl.accept()Ljava/net/Socket;(Unknown Source)

    at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:528)

    at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)

    at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:171)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"Java2D Disposer" id=49 idx=0xb4 tid=6356 prio=10 alive, in native, waiting, daemon

    -- Waiting for notification on: java/lang/ref/ReferenceQueue$Lock@0x09D79E60[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:118)

    ^-- Lock released while waiting: java/lang/ref/ReferenceQueue$Lock@0x09D79E60[fat lock]

    at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:134)

    at sun/java2d/Disposer.run(Disposer.java:125)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

"AWT-Windows" id=50 idx=0xbc tid=8000 prio=6 alive, in native, daemon

    at sun/awt/windows/WToolkit.eventLoop()V(Native Method)

    at sun/awt/windows/WToolkit.run(WToolkit.java:291)

    at java/lang/Thread.run(Thread.java:619)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

===== END OF THREAD DUMP ===============

 


484302 cover.indd The 2009 book Professional Oracle WebLogic Server has excellent information about how to read WLS thread dumps starting on page 686.  But for those of you that want Cliff Notes, STUCK threads are bad.  A thread that has the name STUCK in it’s name has been identified as not being returned to the thread pool for a configured amount of time, which defaults to 10 minutes.  In a subsequent post I’ll show you how to use JRockit Mission Control to look at threads which has some nice tooling for sorting through this detail.

Also check out some other posts related to this:

http://blogs.oracle.com/staffan/2008/08/useful_tool_jrcmd.html

http://blogs.oracle.com/staffan/2008/08/more_jrcmd_tricks.html

http://www.oracle.com/technology/pub/articles/dev2arch/2007/12/jrockit-tuning3.html

Comments:

James, This is a very useful post, thanks. One problem I do have with jrcmd is invoking it on a remote JVM. We have a series of admin commands which are windows batch files. We run these from the admin server and running a "dump.cmd" will use weblogic.Admin and connect to each of the JVM's in the cluster and issue a thread_dump command. weblogic.Admin needs to have a thread available for it to run, and in some cases this isn't possible - when things have gone really bad! JRCMD on the other hand will always get a thread dump (in my experience) but what I haven't figured out how to do is say, run a batch file from the admin server machine and this will issue a jrcmd against all members of the cluster. At the moment, I have to remotely log on to each server and issue the command. Hopefully that will make sense and if anyone has any suggestions.... Pete

Posted by Pete on February 14, 2010 at 06:03 PM PST #

Well I have two good answers for you, one is a blog entry from a JRockit engineer Marcus Hirt with sample code how to issue JRCMD-like features on a remote machine: http://blogs.oracle.com/hirt/2008/08/jrcmd_goes_remote_r271_edition.html The second approach would be to use the -Xmanagement options to enable you to send remote commands to remote machines directly from JRockit Mission Control. I'll show this shortly in another post. http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html#wp1030433 If you have questions in the meantime, definitely post in the JRockit OTN forum and you should get good advice: http://forums.oracle.com/forums/forum.jspa?forumID=561 Good luck, James

Posted by james.bayer on February 14, 2010 at 10:07 PM PST #

Hi James, I just have a simple question, whether we can use JRCMD to take t he thread dump for SUN JVM/HOTSPOT ? Regards.

Posted by Kevin on April 06, 2011 at 06:33 PM PDT #

Kevin, No you can't use JRCMD with Hotspot, you need to use other approaches such as jps (for retrieving PIDs) and jstack for getting thread dumps. These tools are in the JDK/bin directory. Thanks, James

Posted by james.bayer on April 07, 2011 at 04:02 PM PDT #

i can't use JRCMD when the weblogic instances (9.2 mp3) are running under NodeManager control on windows 2003 R2....any ideas??..thanks

Posted by subhash on August 25, 2011 at 07:40 AM PDT #

Subhash,

When running under Windows Servers, it's likely a user permissions issue with the user that is running the service not being the same as the user running JRCMD. I suggest either make those the same OR use a remote network listener on JRockit by passing the following Java options (or similar): -Xmanagement:port=7091,ssl=false,authenticate=false which opens a port on 7091 on the machine where the Server is running that you can connect to as a defined connection. Cheers, James

Posted by james.bayer on August 25, 2011 at 08:18 AM PDT #

Thanks james; i appreciate your quick response.
weblogic instance is running with 'system' account and i was using jrcmd with my domain account. It could be possible as you said.
But about your other alternative work around, will it work with out having a JRockit Mission Control? I was using jrcmd all along with out having JRockit Mission Control in the env.

Posted by subhash on August 25, 2011 at 08:30 AM PDT #

Subhash,

I think JRCMD only works against the local machine based off looking at some docs.
http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/ctrlbreakhndlr.html#wp1001743

In order to use the remote commands (even if talking to a local port) I think you have to use Mission Control, which is the JRockit/bin directory with the jrmc command. You could always ask in the OTN JRockit forum to see what they recommend.
https://forums.oracle.com/forums/forum.jspa?forumID=561

Thanks,

James

Posted by james.bayer on August 25, 2011 at 08:34 AM PDT #

i am using JRMC against local machine only but still couldn't connect. but i guess it is known limitation from that document.

When using jrcmd on Windows, you need to run the Java process and jrcmd from the same Windows station. If you run the Java process as a Windows service, and run jrcmd on your desktop, it will not work, since they are running in two separate environments.

thanks anyway...i will have to think about buying a license for mission control.hope it should be useful. i appreciate your responses.

Posted by Subhash on August 26, 2011 at 05:54 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

James Bayer Image
I was formerly a Product Manager on the WebLogic Server team based out of Oracle HQ. You can find my new blog at http://iamjambay.com.
Follow Me on Twitter
Oracle WebLogic

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today