Monday Nov 18, 2013

Navigating Through Diagnostic Data Options

In this new post I will talk about diagnostic data and more importantly what data and source of data are needed to troubleshoot and resolve some specific common problems.

WebLogic Server along with the JVM offers various ways to collect logs, debug traces, diagnostic images, thread dumps, and much more. However, useful data can easily get buried with lots of noise, unneeded data that won't help resolving a particular problem.

Performance Issues

We will start with performance issues such as long running requests or server hangs.  The best way to narrow down where the issue lies is to dump the java threads and parse them through a thread dump analyzer tool such as ThreadLogic.  Looking at the server log files, or enabling some debugging features would be premature since the first step is to identify the thread(s) that are stuck to determine what is being executed and relevant JEE resources. The log could also alert about stuck threads once they have been stuck for over the maximum allowed time (see MaxStruckThreadTime) but unless thread dumps are examined, it won't be possible to determine possible patterns of issues and threads' relationship around locked objects.

Memory Leaks

Another common performance issue involves memory leaks.  If you encounter OutOfMemory java.lang exceptions you should issue a heap dump and analyze it with a tool such as VisualVM or the Eclipse MAT.  See my post on Heap Dumps analysis for more details. Heap dumps are snapshots of memory certain times.  Heap dumps are to the Java process memory (non native) what thread dumps are to Java process threads.  Enabling -verbose:gc is also an effective way to monitor heap usage at runtime.  The output of the GC stats can be redirected to a file so that not to clog the server logs.  This can be done by setting -Xloggc with HotSpot JVM.  It's recommended to include -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps to log time stamps and detailed GC activities as well.  Troubleshooting guide for HotSpot VM details these flags among others useful flags for diagnosis purposes.

Applications, 3rd parties, and configuration issues

The following chart summarizes what could be collected to debug each of the listed WLS component.  This list is by no means exhaustive.  Many more debug flags could be used for these listed components.  This chart contains examples of useful debug options and links to additional resources.

For the following, Redirect stdout and stderr logging should be enabled.  This can be done via the Administration Console in the advanced section of the servers logging tab or at startup in the java command line with -Dweblogic.StdoutDebugEnabled=true.  In addition, the severity level of the log file should be set to debug.

WLS comp. or 3rd party resource

Examples of Debug options and flags

MyOracleSupport  related resources

(login required)

 EJB/Web Serv.



To log SOAP requests and response messages

Trace in servers log files 

Troubleshooting EJBs Issues

How to enable Debugging for Weblogic EJB

How To Debug WLS Application Container Problems

Troubleshooting Oracle Web Services 11g



To log info on load, store events and transaction records into servers log files

See Debugging JMS for more JMS debugging options

Troubleshooting JMS Info Center



Trace in servers log files 

Troubleshooting SSL Security

-Dweblogic.debug.DebugReplication for high level replication info

-Dweblogic.debug.DebugClusterAnnouncements to log info on announcement, StateDump and attributes messages sent or received by multicast

Trace in servers log files  

Debug RJVM or Intra-Cluster Communication Problems

weblogic.JTAXA & weblogic.JTA2PC for runtime issues, example:

JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA, weblogic.JTA2PC"

Trace in servers log files  

Investigating Transaction Problems


to print information about JDBC methods invoked with their arguments, return values, and thrown exceptions

See JDBC Debugging Scopes under Monitoring JDBC Resources

Trace in servers log files  

Investigating JDBC Issues

JDBC Debugging Scopes

Oracle JDBC Driver (*)

-Doracle.jdbc.Trace=true to enable jdbc logging


Trace is file defined in oracle.jdbc.LogFile

Enable Oracle JDBC Driver Debug in WLS
Deployment (Classloader) Set the following options to true to report which classes are getting loaded







Trace in servers log files

Deployment Problems - Enabling Debug

Proxy plug-in

Debug="ALL" in the proxy configuration file

(iisproxy.ini, httpd.conf or obj.conf)

Trace in file set in WLLogFile defined inside proxy configuration file

Common Diagnostic Process for Proxy Plug-In Problems

(*)  Enabling JDBC Driver debug flags is verbose so it's recommended to set logging properties to appropriate levels such as FINE, SEVERE, FINEST, INFO etc. based upon the debugging requirement.


Diagnosing and troubleshooting also require a good understanding of the Weblogic domain environment, its configuration, patching level, JVM parameters and also an easy navigation through various logs and configuration files.  For this, RDA is a great diagnosing tool because it collects all this data indiscriminately.  One of the numerous benefits of RDA is that it doesn't require any complex setting and can collect data from each distinctive log file, xml repository and scripts.  So, RDA reports are of great assistance while working directly with Oracle support because that allow the engineer assigned to a service request to quickly understand the environment under which a specific issue is occurring.  This generally helps expedite the resolution of the SR.


The Weblogic Diagnostic Framework can be used to collect metrics, setup watch and notifications, and to define instrumentation.  With WLS 12.1.2, you can set the level (Low, Medium or High) of built-in modules that will gather metrics, or disable them completely.  WLDF collects runtime statistics on JDBC, JTA and JVM and more.  You can then use the Monitoring Dashboard to view and navigate through the collected data.  WLDF is not an alternative to debug options.  Debug options activate tracing that is coded as part of WLS whereas WLDF collects and monitors runtime mbeans values and can activate notifications with defined rules.  They are used for different purposes.  

The WLDF built-in configurations (Low, Medium, & High) can be cloned into a new configuration, and used as templates for creating custom WLDF configurations.  The Low volume built-in is enabled by default in production mode.  One new feature with WLS 12.1.2 is Runtime control which gives the ability to activate or deactivate diagnostic system modules dynamically at runtime wihtout making any domain configuration change.


JFR recording data can also be captured through WLDF diagnostic images based on WLDF watch rules so that JVM runtime system information can be analyzed along with recording of WLS components diagnostic data.  Once extracted from the diagnostic image capture, the JFR file can be analyzed using Java Mission Control.


WLST diagnostic commands can be used to extract data from the diagnostic archive (event and harvested metric data) in either XML form or CSV, based on the function you use.  You can also dump the diagnostics data from a harvester to a local file.

The following WLST functions allow to capture an image and to download image files from a remote system over WLST:

One major difference between the diagnostic framework and the debug options is that debugs are enabled so that an issue can be reproduced and traced whereas the diagnostic image capture is used as a server-level dump for post-failure analysis in a similar way heap dumps are used post memory leak failures.

Wednesday Mar 13, 2013

Potential Issues With Weblogic Server Lok Files

Certain configuration files's access sometime need to be restricted in order to only allow one Weblogic Server instance or user to access them at any specific time.  Files with .lok extension are created for this purpose.  While browsing your domain directory structure, you might find config.lok under the domain config directory.  This file exists to ensure that config.xml is being updated serially.  Edit.lok's purpose is to ensure that only one user at a time can edit the domain configuration.  Edit.lok lies in the domain directory.  Other lok files include EmbeddedLDAP.lok and server_name.lok.

Each running server creates its own server_name.lok (in \servers\server_name\tmp) and own EmbeddedLDAP.lok (in \servers\server_name\ldapfiles) while starting up.  The process ID (PID) that is holding the lock on these files is the same that the process which is running server_name.  This can easily be verified with the fuser Unix command or with handle on Windows.  Both files have a size of 0 byte.

MOS note 943790.1 describes these 4 files in more details.

In addition to creating these two files, the administration server also creates config.lok, also with a size of 0 byte, and edit.lok.  However edit.lok stores data such as encoded user information and various timestamps.  This file is updated as changes are being made or has the timestamp of the latest update made in the administraction console.  It's highly recommended not to delete any of these files while servers in your domain are running.  During the shutdown process, the server should delete its own EmbeddedLDAP.lok and server_name.lok and leave edit.lok and config.lok as they are.

So what can go wrong?

A server can fail to start with the critical error code BEA-000362 and a nested ManagementException reporting that it's unable to obtain lock on server_name.lok and that this server might already be running.  This issue can come up if the server didn't previously shut down gracefully and thus couldn't unlock (delete) its lok files, or simply because a process running this server still exists. At this point, and if no process is currently running this server, server_name.lok and EmbeddedLDAP.lok would need to be deleted manually for the server to startup successfully.

If EmbeddedLDAP.lok is present while a server is booting then the startup will fail with the emergency error code BEA-000342 and a nested ServiceFailureException containing Could not obtain an exclusive lock to the embedded LDAP data files directory.  One EmbeddedLDAP.lok exists for each server in the domain whereas only one edit.lok and one config.lok exist for the whole domain.

While attempting simultaneous deployments, you might encounter the warning code BEA-149124 with the message The domain edit lock is owned by another session in exclusive mode - hence this deployment operation cannot proceed. When multiple deployment tools are being used, only one can acquire a lock on the domain configuration at a given time until a deployment is complete.  The usenonexclusivelock parameter can be used with the wldeploy ant task or with weblogic.Deploy if this warning is considered an inconvenience.

Friday Jan 18, 2013

Troubleshooting Tools Part 1 - VisualVM

A variety of free troubleshooting and debugging tools exist but which ones are really useful when analyzing issues with WebLogic Server?  In this new blog post series, I will talk about some of the best tools that are available, easy to use, free, and very effective in identifying a bunch of issues.  In this first post I will cover VisualVM.

VisualVM most recent version, or 1.3.5, was released on 11/13/2012.  This tool consists of an user-friendly visual interface for monitoring running JVMs.  Multiple instances of WLS or Java can be monitored at the same time by just connecting to running local or remote JVMs.  VisualVM needs to run on Oracle Sun JDK 6+, the JDK, not the JRE. VisualVM can be started as follow to specify the JDK to run it it on.

visualvm --jdkhome "JDK_location", or visualvm --jdkhome "d:\jdk160_24"

Alternatively, /etc/visualvm.conf can be modified to specify a desired JDK.  After launch, the list of local running Java instances will be displayed on the left panel under applications.  In Remote, you can define the host of other running Java instances.  By selecting a running server and clicking on open, you should get a screen similar to the following:

In Start Page, online documentation of VisualVM can be easily accessed.  Tags next to Start Page are specific to open JVM instances.  In my case, you will find a Weblogic Server instance with its process id (pid).

Underneath, Overview reports all JVM settings as well as many system properties including Weblogic specific properties.

Monitor gives access to realtime monitoring of CPU, Heap, Classes and Threads.  You can also force a garbage collection (GC) or issue a heap dump for further analysis.

Threads reports all threads activity in realtime but also allows to take thread dumps.  All thread dumps will be attached to the relevant application process on the left panel and saved in C:\Users\<user_name>\AppData\Local\Temp\visualvm.dat.  Example: C:\Users\lgoldszt\AppData\Local\Temp\visualvm.dat\localhost_6352\threaddump-1358534350679.tdump.  Thread dumps can be further diagnosed with ThreadLogic.

In Samplers, you can record CPU or memory usage activity.  This is neat to debug potential memory leak and high CPU usage threads.

Finally, Profiler allows to identify where most of the time is being spent and which objects consume most of the memory.

Snapshots of memory can be taken and compared with others, see File - Compare Memory Snapshots.  

Also VisualVM comes with a bunch of easy to install plugins as you can see below:

VisualVM doesn't have an MBeans tab like JConsole.  However, this can be very easily changed by installing the available VisualVM-MBeans plugin.  

With its extensive features, especially including those offered by individual plugin, VisualVM is the tool of choice and combine characteristics of tools such as JConsole, jstat, jmap, or jstack.

Among other choices, the download page gives you the opportunity to use the VisualVM Eclipse launcher plugin.

For further information and demos please visit VisualVM demo and features

Monday Oct 20, 2008

Where To Download Old WebLogic/AquaLogic Versions

[Read More]

Wednesday Sep 17, 2008

Oracle Lifetime Support and WebLogic Server

[Read More]

The official blog for Oracle WebLogic Server fans and followers!

Stay Connected


« July 2014