Monday Nov 18, 2013

Navigating Through Diagnostic Data Options

In this new post I will talk about diagnostic data and more importantly what data and source of data are needed to troubleshoot and resolve some specific common problems.

WebLogic Server along with the JVM offers various ways to collect logs, debug traces, diagnostic images, thread dumps, and much more. However, useful data can easily get buried with lots of noise, unneeded data that won't help resolving a particular problem.

Performance Issues

We will start with performance issues such as long running requests or server hangs.  The best way to narrow down where the issue lies is to dump the java threads and parse them through a thread dump analyzer tool such as ThreadLogic.  Looking at the server log files, or enabling some debugging features would be premature since the first step is to identify the thread(s) that are stuck to determine what is being executed and relevant JEE resources. The log could also alert about stuck threads once they have been stuck for over the maximum allowed time (see MaxStruckThreadTime) but unless thread dumps are examined, it won't be possible to determine possible patterns of issues and threads' relationship around locked objects.

Memory Leaks

Another common performance issue involves memory leaks.  If you encounter OutOfMemory java.lang exceptions you should issue a heap dump and analyze it with a tool such as VisualVM or the Eclipse MAT.  See my post on Heap Dumps analysis for more details. Heap dumps are snapshots of memory certain times.  Heap dumps are to the Java process memory (non native) what thread dumps are to Java process threads.  Enabling -verbose:gc is also an effective way to monitor heap usage at runtime.  The output of the GC stats can be redirected to a file so that not to clog the server logs.  This can be done by setting -Xloggc with HotSpot JVM.  It's recommended to include -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps to log time stamps and detailed GC activities as well.  Troubleshooting guide for HotSpot VM details these flags among others useful flags for diagnosis purposes.

Applications, 3rd parties, and configuration issues

The following chart summarizes what could be collected to debug each of the listed WLS component.  This list is by no means exhaustive.  Many more debug flags could be used for these listed components.  This chart contains examples of useful debug options and links to additional resources.

For the following, Redirect stdout and stderr logging should be enabled.  This can be done via the Administration Console in the advanced section of the servers logging tab or at startup in the java command line with -Dweblogic.StdoutDebugEnabled=true.  In addition, the severity level of the log file should be set to debug.

WLS comp. or 3rd party resource

Examples of Debug options and flags

MyOracleSupport  related resources

(login required)

 EJB/Web Serv.

-Dweblogic.webservice.verbose=true

-Dweblogic.webservice.client.verbose=true

To log SOAP requests and response messages

Trace in servers log files 

Troubleshooting EJBs Issues

How to enable Debugging for Weblogic EJB

How To Debug WLS Application Container Problems

Troubleshooting Oracle Web Services 11g

 JMS

-Dweblogic.debug.DebugJMSStore

To log info on load, store events and transaction records into servers log files

See Debugging JMS for more JMS debugging options


Troubleshooting JMS Info Center

 SSL

-Dssl.debug=true

-Dweblogic.security.SSL.verbose=true 

Trace in servers log files 

Troubleshooting SSL Security
Clustering

-Dweblogic.debug.DebugReplication for high level replication info

-Dweblogic.debug.DebugClusterAnnouncements to log info on announcement, StateDump and attributes messages sent or received by multicast

Trace in servers log files  

Debug RJVM or Intra-Cluster Communication Problems
JTA

weblogic.JTAXA & weblogic.JTA2PC for runtime issues, example:

JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA, weblogic.JTA2PC"

Trace in servers log files  

Investigating Transaction Problems
JDBC

-Dweblogic.debug.DebugJDBCSQL=true

to print information about JDBC methods invoked with their arguments, return values, and thrown exceptions

See JDBC Debugging Scopes under Monitoring JDBC Resources

Trace in servers log files  

Investigating JDBC Issues

JDBC Debugging Scopes

Oracle JDBC Driver (*)

-Doracle.jdbc.Trace=true to enable jdbc logging

Create OracleLog.properties

Trace is file defined in oracle.jdbc.LogFile

Enable Oracle JDBC Driver Debug in WLS
Deployment (Classloader) Set the following options to true to report which classes are getting loaded

-Dweblogic.utils.classloaders.GenericClassLoader.Verbose

-Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose

-Dweblogic.utils.classloaders.ClasspathClassFinder

-Dweblogic.utils.classloaders.DefaultFilteringClassLoader.Verbose

-Dweblogic.utils.classloaders.FilteringClassLoader.Verbose

-Dweblogic.utils.classloaders.FilteringClassLoader.ResourceDump

Trace in servers log files

Deployment Problems - Enabling Debug


Proxy plug-in

Debug="ALL" in the proxy configuration file

(iisproxy.ini, httpd.conf or obj.conf)

Trace in file set in WLLogFile defined inside proxy configuration file

Common Diagnostic Process for Proxy Plug-In Problems

(*)  Enabling JDBC Driver debug flags is verbose so it's recommended to set logging properties to appropriate levels such as FINE, SEVERE, FINEST, INFO etc. based upon the debugging requirement.

RDA

Diagnosing and troubleshooting also require a good understanding of the Weblogic domain environment, its configuration, patching level, JVM parameters and also an easy navigation through various logs and configuration files.  For this, RDA is a great diagnosing tool because it collects all this data indiscriminately.  One of the numerous benefits of RDA is that it doesn't require any complex setting and can collect data from each distinctive log file, xml repository and scripts.  So, RDA reports are of great assistance while working directly with Oracle support because that allow the engineer assigned to a service request to quickly understand the environment under which a specific issue is occurring.  This generally helps expedite the resolution of the SR.

WLDF

The Weblogic Diagnostic Framework can be used to collect metrics, setup watch and notifications, and to define instrumentation.  With WLS 12.1.2, you can set the level (Low, Medium or High) of built-in modules that will gather metrics, or disable them completely.  WLDF collects runtime statistics on JDBC, JTA and JVM and more.  You can then use the Monitoring Dashboard to view and navigate through the collected data.  WLDF is not an alternative to debug options.  Debug options activate tracing that is coded as part of WLS whereas WLDF collects and monitors runtime mbeans values and can activate notifications with defined rules.  They are used for different purposes.  

The WLDF built-in configurations (Low, Medium, & High) can be cloned into a new configuration, and used as templates for creating custom WLDF configurations.  The Low volume built-in is enabled by default in production mode.  One new feature with WLS 12.1.2 is Runtime control which gives the ability to activate or deactivate diagnostic system modules dynamically at runtime wihtout making any domain configuration change.

JFR

JFR recording data can also be captured through WLDF diagnostic images based on WLDF watch rules so that JVM runtime system information can be analyzed along with recording of WLS components diagnostic data.  Once extracted from the diagnostic image capture, the JFR file can be analyzed using Java Mission Control.

WLST

WLST diagnostic commands can be used to extract data from the diagnostic archive (event and harvested metric data) in either XML form or CSV, based on the function you use.  You can also dump the diagnostics data from a harvester to a local file.

The following WLST functions allow to capture an image and to download image files from a remote system over WLST:

One major difference between the diagnostic framework and the debug options is that debugs are enabled so that an issue can be reproduced and traced whereas the diagnostic image capture is used as a server-level dump for post-failure analysis in a similar way heap dumps are used post memory leak failures.

Tuesday Feb 05, 2013

Troubleshooting Tools Part 2 - jstack

In my latest blog post I talked about VisualVM, a very effective monitoring and troubleshooting tool.  Many other easy-to-use and free tools can be used while working with Java SE.  First of all, and since there is no need to re-invent the wheel, you can access documentation of such tools for Windows at Quick Troubleshooting Tips on Windows for Java SE 6 and for Solaris and Linux at Troubleshooting Tools on Solaris OS and Linux for Java SE 6

These tools include jstack, jinfo, jconsole, jmap, jstat, jhat and a few others.

jstack can be use to dump the java threads.  While working with WebLogic Server, you can dump the java threads with various tools such as WLST.  jstack can do just that but differs from <ctrl>+<break> or kill -3 since it doesn't dump a heap summary along with the threads.  

In addition, jstack can print lock information with -l option and the list of synchronizers that may be exclusively owned by a thread.  Such information can be very useful while monitoring concurrent access by threads.

jstack comes as part of the JDK installed with WebLogic Server.  With instances of WLS running, you can open a new shell, execute the setDomain script and run jstack or any tool that's part of the JDK package. Thread dumps taken by jstack can be redirected to files, e.g. jstack <pid> > myTD.txt.

PIDs can be obtained on Solaris/Linux with the ps (report process status) command, e.g. ps -ef | grep java, and from the task managed on Windows.  In task manager you might need to go to View and Select Columns to add PID or process identifiers.

Friday Jan 18, 2013

Troubleshooting Tools Part 1 - VisualVM

A variety of free troubleshooting and debugging tools exist but which ones are really useful when analyzing issues with WebLogic Server?  In this new blog post series, I will talk about some of the best tools that are available, easy to use, free, and very effective in identifying a bunch of issues.  In this first post I will cover VisualVM.

VisualVM most recent version, or 1.3.5, was released on 11/13/2012.  This tool consists of an user-friendly visual interface for monitoring running JVMs.  Multiple instances of WLS or Java can be monitored at the same time by just connecting to running local or remote JVMs.  VisualVM needs to run on Oracle Sun JDK 6+, the JDK, not the JRE. VisualVM can be started as follow to specify the JDK to run it it on.

visualvm --jdkhome "JDK_location", or visualvm --jdkhome "d:\jdk160_24"

Alternatively, /etc/visualvm.conf can be modified to specify a desired JDK.  After launch, the list of local running Java instances will be displayed on the left panel under applications.  In Remote, you can define the host of other running Java instances.  By selecting a running server and clicking on open, you should get a screen similar to the following:

In Start Page, online documentation of VisualVM can be easily accessed.  Tags next to Start Page are specific to open JVM instances.  In my case, you will find a Weblogic Server instance with its process id (pid).

Underneath, Overview reports all JVM settings as well as many system properties including Weblogic specific properties.

Monitor gives access to realtime monitoring of CPU, Heap, Classes and Threads.  You can also force a garbage collection (GC) or issue a heap dump for further analysis.

Threads reports all threads activity in realtime but also allows to take thread dumps.  All thread dumps will be attached to the relevant application process on the left panel and saved in C:\Users\<user_name>\AppData\Local\Temp\visualvm.dat.  Example: C:\Users\lgoldszt\AppData\Local\Temp\visualvm.dat\localhost_6352\threaddump-1358534350679.tdump.  Thread dumps can be further diagnosed with ThreadLogic.

In Samplers, you can record CPU or memory usage activity.  This is neat to debug potential memory leak and high CPU usage threads.

Finally, Profiler allows to identify where most of the time is being spent and which objects consume most of the memory.

Snapshots of memory can be taken and compared with others, see File - Compare Memory Snapshots.  

Also VisualVM comes with a bunch of easy to install plugins as you can see below:

VisualVM doesn't have an MBeans tab like JConsole.  However, this can be very easily changed by installing the available VisualVM-MBeans plugin.  

With its extensive features, especially including those offered by individual plugin, VisualVM is the tool of choice and combine characteristics of tools such as JConsole, jstat, jmap, or jstack.

Among other choices, the download page gives you the opportunity to use the VisualVM Eclipse launcher plugin.

For further information and demos please visit VisualVM demo and features

About

The official blog for Oracle WebLogic Server fans and followers!

Stay Connected

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
5
6
7
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today