Navigating Through Diagnostic Data Options

In this new post I will talk about diagnostic data and more importantly what data and source of data are needed to troubleshoot and resolve some specific common problems.

WebLogic Server along with the JVM offers various ways to collect logs, debug traces, diagnostic images, thread dumps, and much more. However, useful data can easily get buried with lots of noise, unneeded data that won't help resolving a particular problem.

Performance Issues

We will start with performance issues such as long running requests or server hangs.  The best way to narrow down where the issue lies is to dump the java threads and parse them through a thread dump analyzer tool such as ThreadLogic.  Looking at the server log files, or enabling some debugging features would be premature since the first step is to identify the thread(s) that are stuck to determine what is being executed and relevant JEE resources. The log could also alert about stuck threads once they have been stuck for over the maximum allowed time (see MaxStruckThreadTime) but unless thread dumps are examined, it won't be possible to determine possible patterns of issues and threads' relationship around locked objects.

Memory Leaks

Another common performance issue involves memory leaks.  If you encounter OutOfMemory java.lang exceptions you should issue a heap dump and analyze it with a tool such as VisualVM or the Eclipse MAT.  See my post on Heap Dumps analysis for more details. Heap dumps are snapshots of memory certain times.  Heap dumps are to the Java process memory (non native) what thread dumps are to Java process threads.  Enabling -verbose:gc is also an effective way to monitor heap usage at runtime.  The output of the GC stats can be redirected to a file so that not to clog the server logs.  This can be done by setting -Xloggc with HotSpot JVM.  It's recommended to include -XX:+PrintGCDetails, XX:+PrintGCDateStamps (for Java 6 Update 4 and later JVMs) and -XX:+PrintGCTimeStamps to log time stamps and detailed GC activities as well.  Troubleshooting guide for HotSpot VM details these flags among others useful flags for diagnosis purposes.

Applications, 3rd parties, and configuration issues

The following chart summarizes what could be collected to debug each of the listed WLS component.  This list is by no means exhaustive.  Many more debug flags could be used for these listed components.  This chart contains examples of useful debug options and links to additional resources.

For the following, Redirect stdout and stderr logging should be enabled.  This can be done via the Administration Console in the advanced section of the servers logging tab or at startup in the java command line with -Dweblogic.StdoutDebugEnabled=true.  In addition, the severity level of the log file should be set to debug.

WLS comp. or 3rd party resource

Examples of Debug options and flags

MyOracleSupport  related resources

(login required)

 EJB/Web Serv.



To log SOAP requests and response messages

Trace in servers log files 

Troubleshooting EJBs Issues

How to enable Debugging for Weblogic EJB

How To Debug WLS Application Container Problems

Troubleshooting Oracle Web Services 11g



To log info on load, store events and transaction records into servers log files

See Debugging JMS for more JMS debugging options

Troubleshooting JMS Info Center



Trace in servers log files 

Troubleshooting SSL Security

-Dweblogic.debug.DebugReplication for high level replication info

-Dweblogic.debug.DebugClusterAnnouncements to log info on announcement, StateDump and attributes messages sent or received by multicast

Trace in servers log files  

Debug RJVM or Intra-Cluster Communication Problems

weblogic.JTAXA & weblogic.JTA2PC for runtime issues, example:

JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA, weblogic.JTA2PC"

Trace in servers log files  

Investigating Transaction Problems


to print information about JDBC methods invoked with their arguments, return values, and thrown exceptions

See JDBC Debugging Scopes under Monitoring JDBC Resources

Trace in servers log files  

Investigating JDBC Issues

JDBC Debugging Scopes

Oracle JDBC Driver (*)

-Doracle.jdbc.Trace=true to enable jdbc logging


Trace is file defined in oracle.jdbc.LogFile

Enable Oracle JDBC Driver Debug in WLS
Deployment (Classloader) Set the following options to true to report which classes are getting loaded







Trace in servers log files

Deployment Problems - Enabling Debug

Proxy plug-in

Debug="ALL" in the proxy configuration file

(iisproxy.ini, httpd.conf or obj.conf)

Trace in file set in WLLogFile defined inside proxy configuration file

Common Diagnostic Process for Proxy Plug-In Problems

(*)  Enabling JDBC Driver debug flags is verbose so it's recommended to set logging properties to appropriate levels such as FINE, SEVERE, FINEST, INFO etc. based upon the debugging requirement.


Diagnosing and troubleshooting also require a good understanding of the Weblogic domain environment, its configuration, patching level, JVM parameters and also an easy navigation through various logs and configuration files.  For this, RDA is a great diagnosing tool because it collects all this data indiscriminately.  One of the numerous benefits of RDA is that it doesn't require any complex setting and can collect data from each distinctive log file, xml repository and scripts.  So, RDA reports are of great assistance while working directly with Oracle support because that allow the engineer assigned to a service request to quickly understand the environment under which a specific issue is occurring.  This generally helps expedite the resolution of the SR.


The Weblogic Diagnostic Framework can be used to collect metrics, setup watch and notifications, and to define instrumentation.  With WLS 12.1.2, you can set the level (Low, Medium or High) of built-in modules that will gather metrics, or disable them completely.  WLDF collects runtime statistics on JDBC, JTA and JVM and more.  You can then use the Monitoring Dashboard to view and navigate through the collected data.  WLDF is not an alternative to debug options.  Debug options activate tracing that is coded as part of WLS whereas WLDF collects and monitors runtime mbeans values and can activate notifications with defined rules.  They are used for different purposes.  

The WLDF built-in configurations (Low, Medium, & High) can be cloned into a new configuration, and used as templates for creating custom WLDF configurations.  The Low volume built-in is enabled by default in production mode.  One new feature with WLS 12.1.2 is Runtime control which gives the ability to activate or deactivate diagnostic system modules dynamically at runtime wihtout making any domain configuration change.


JFR recording data can also be captured through WLDF diagnostic images based on WLDF watch rules so that JVM runtime system information can be analyzed along with recording of WLS components diagnostic data.  Once extracted from the diagnostic image capture, the JFR file can be analyzed using Java Mission Control.


WLST diagnostic commands can be used to extract data from the diagnostic archive (event and harvested metric data) in either XML form or CSV, based on the function you use.  You can also dump the diagnostics data from a harvester to a local file.

The following WLST functions allow to capture an image and to download image files from a remote system over WLST:

One major difference between the diagnostic framework and the debug options is that debugs are enabled so that an issue can be reproduced and traced whereas the diagnostic image capture is used as a server-level dump for post-failure analysis in a similar way heap dumps are used post memory leak failures.


I would recommend using -XX:+PrintGCDateStamps (Java SE 6 update 4 and later) which displays a human readable absolute time of the GC activity instead of -XX:+PrintGCTimeStamps which displays the number of seconds since the JVM was started. The seconds are only relevant if you know when the JVM was started, which can be hard to work with if all you have is the GC log file (e.g. -Xloggc:<file_name>).

Posted by George Oprica on September 15, 2014 at 11:56 AM PDT #

Very good point, post updated, thanks!

Posted by guest on September 15, 2014 at 12:05 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed

The official blog for Oracle WebLogic Server fans and followers!

Stay Connected


« March 2015