Friday May 17, 2013

Troubleshooting Tools Part 3 - Heap Dumps

A Heap Dump is a snapshot of the Java process heap memory at a given time.  It's a useful tool to troubleshoot memory leaks and other memory related issues.  Heap dumps are to memory what thread dumps are to java threads.

There are different ways to dump the java heap used by a Java application such as WebLogic Server.  Heap dumps can be triggered automatically by JVMs such as HotSpot or JRockit when an OutOfMemory event occurs.

Example:

java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\myApp\hprof-dumps myApp  

This command will generate a heap dump file in the defined heap dump directory when an OOM is thrown.  

The file will be in HPROF binary format (*.hprof).  If no path is defined the file will be created in the directory from which the java process was launched.  The format of the file with JRockit is jrockit_pid,hprof, pid being the process ID of this JVM instance.  With HotSpot the format of the file will be java_pidpid.hprof.

Example:

java.lang.OutOfMemoryError: Java heap space

Dumping heap to D:\java_pid4148.hprof ...

Heap dump file created [24921731 bytes in 1.400 secs]

For WebLogic Server these properties, HeapDumpOnOutOfMemoryError and HeapDumpPath (optional), can be added directly to the startup script that contain the Java command that launches WLS.  On Windows the script (with default name startWebLogic.cmd) lives in the domain bin directory and the parameters can be directly added to the existing JAVA_OPTIONS.

With a JRockit VM process the following command can also be used to generate a heap dump.

Example:

jrcmd pid hprofdump filename=name_of_dump_file

However if you are using this command against a non JRockit process the following error will come up:

java.io.IOException: Command failed in target VM

You can also manually generate a heap dump with tools such as jmap or VisualVM.

Example:

jmap -heap:live,format=b,file=filename pid

This command will generate a heap dump file in binary format and will only count live objects (the live option is optional). Live objects are used by applications and generally reside in the Young Generation part of the heap.

As illustrated in the screenshot below, generating a heap dump with VisualVM is very simple.  VisualVM is shipped as part of the HotSpot JVM  (/bin/jvisualvm.exe).  However, VisualVM can connect to java processes that started with other JVM such as JRockit.


Once generated, the heap dump will be parsed as shown below:

VisualVM enables you to browse heap dumps as shown above with the Classes view where you can easily identify the number and percentage of instances referred by each class.

jhat can also be used to analyze heap dump files.

Example:

jhat dumpfile

Reading from dumpfile...
Dump file created Fri May 17 15:25:41 PDT 2013
Snapshot read, resolving...
Resolving 580390 objects...
Chasing references, expect 116 dots.............................................
.......................................................................
Eliminating duplicate references................................................
....................................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

Then you can open http://localhost:7000/ to navigate through the content of the dump file to identify objects with high number of occurrences.

Finally, as shown below, the OpenSource Eclipse MAT offers a very rich GUI that includes histograms and leak hunter actions, overview and leak suspect reports and much more.


Previous posts on troubleshooting tools:

Troubleshooting Tools Part 1 - VisualVM

Troubleshooting Tools Part 2 - jstack

Friday Jan 18, 2013

Troubleshooting Tools Part 1 - VisualVM

A variety of free troubleshooting and debugging tools exist but which ones are really useful when analyzing issues with WebLogic Server?  In this new blog post series, I will talk about some of the best tools that are available, easy to use, free, and very effective in identifying a bunch of issues.  In this first post I will cover VisualVM.

VisualVM most recent version, or 1.3.5, was released on 11/13/2012.  This tool consists of an user-friendly visual interface for monitoring running JVMs.  Multiple instances of WLS or Java can be monitored at the same time by just connecting to running local or remote JVMs.  VisualVM needs to run on Oracle Sun JDK 6+, the JDK, not the JRE. VisualVM can be started as follow to specify the JDK to run it it on.

visualvm --jdkhome "JDK_location", or visualvm --jdkhome "d:\jdk160_24"

Alternatively, /etc/visualvm.conf can be modified to specify a desired JDK.  After launch, the list of local running Java instances will be displayed on the left panel under applications.  In Remote, you can define the host of other running Java instances.  By selecting a running server and clicking on open, you should get a screen similar to the following:

In Start Page, online documentation of VisualVM can be easily accessed.  Tags next to Start Page are specific to open JVM instances.  In my case, you will find a Weblogic Server instance with its process id (pid).

Underneath, Overview reports all JVM settings as well as many system properties including Weblogic specific properties.

Monitor gives access to realtime monitoring of CPU, Heap, Classes and Threads.  You can also force a garbage collection (GC) or issue a heap dump for further analysis.

Threads reports all threads activity in realtime but also allows to take thread dumps.  All thread dumps will be attached to the relevant application process on the left panel and saved in C:\Users\<user_name>\AppData\Local\Temp\visualvm.dat.  Example: C:\Users\lgoldszt\AppData\Local\Temp\visualvm.dat\localhost_6352\threaddump-1358534350679.tdump.  Thread dumps can be further diagnosed with ThreadLogic.

In Samplers, you can record CPU or memory usage activity.  This is neat to debug potential memory leak and high CPU usage threads.

Finally, Profiler allows to identify where most of the time is being spent and which objects consume most of the memory.

Snapshots of memory can be taken and compared with others, see File - Compare Memory Snapshots.  

Also VisualVM comes with a bunch of easy to install plugins as you can see below:

VisualVM doesn't have an MBeans tab like JConsole.  However, this can be very easily changed by installing the available VisualVM-MBeans plugin.  

With its extensive features, especially including those offered by individual plugin, VisualVM is the tool of choice and combine characteristics of tools such as JConsole, jstat, jmap, or jstack.

Among other choices, the download page gives you the opportunity to use the VisualVM Eclipse launcher plugin.

For further information and demos please visit VisualVM demo and features

About

The official blog for Oracle WebLogic Server fans and followers!

Stay Connected

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
5
6
7
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today