Monday May 13, 2013

My Take on Getting Started With WebLogic Scripting Tool (WLST)


WebLogic Scripting Tool (WLST) has been a long established favourite for experienced WebLogic Server administrators. Over the past two / three weeks I have spent time getting to know WLST - and by getting to know I mean developing a script to produce a solution rather than randomly running three or four commands :-). For the final result of my efforts, see my previous blog entry

Get an Offline or Online WebLogic Domain Summary Using WLST! 

This is a short summary of the resources I used. 

Initial reading and understanding

If you are never encountered WLST, spend a few minutes reading:

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
3.5.1 Getting Started Using the Oracle WebLogic Scripting Tool (WLST)

Oracle Fusion Middleware WebLogic Scripting Tool Command Reference 11g
Overview of WLST Command Categories

Unsurprisingly, you will also find yourself using the Command Reference as a look up for available commands, functionality and syntax.

WLST Code Snippets and Examples

(There are many more WLST postings (questions, examples, discussions) to be found on the web.)


WLST is based on the Java scripting interpreter, Jython and as such supports standard Jython features such as local variables, conditional variables, and flow-control statements. Therefore, to achieve something more interesting and exciting you must prepared to learn (by doing) a little Jython. Fortunately, the Jython documentation is comprehensive and provides some good examples:

Use an editor which recognizes Jython / Python

Jython (like it's "parent" language Python) is very sensitive to syntax errors. In particular, the language requires the use of indents when coding conditions (e.g IF statements) and loops. Attempt to write Jython using a plain text editor, and before long stress levels will rise. You can make life easier by using an editor which recognizes Jython. Editors like Oracle Eclipse ** and SCITE assist the developer by for example:

  • auto-formatting lines of code
  • color coding of syntax
  • providing option to show line numbers

** Eclipse requires the Python and Jython Interpreters to be configured. Instructions can be found here.

My Oracle Support Documents

And finally, if you have access to My Oracle Support, here are a few knowledge articles which provide some nice WLST examples:

  • How To Debug WLST Jython Scripts? [ID 1360744.1]
  • WLST sample script To Obtain Managed Server Status [ID 1373532.1]
  • WLST Script for monitoring and getting statistics of WebLogic JMS Destinations. [ID 1327086.1]
  • How to get JDBC and Connection Pool data from a WLST or shell script [ID 1453818.1]
  • How to Startup or Shutdown Multiple WebLogic Managed Servers With a WLST Script [ID 1383150.1]
  • How To Take Thread Dumps With WLST [ID 1274713.1]

Tuesday May 07, 2013

Get an Offline or Online WebLogic Domain Summary Using WLST!


Further to blog entry "WLST Offline Script to Create Summary of WebLogic Domain", I have created a solution which extends the capability to collecting an online summary if a connection to the domain's Admin Server can be established.

Search the web, and you will find many WLST examples which summarize managed server status and performance, JMS, JDBC and more. In this respect, my efforts are nothing new. However, I thought I would share as my scripts have the following "bells and whistles".

1. If you are unable to connect to the domain's Admin Server, an offline summary of the domain will automatically be obtained

2. The script renders the collected data in HTML, using some javascript to give the final result some interactive formatting for ease of reading. See screenshot snippet below:

Screenshot : Snippet of WLST Summarizer HTML Output

Screenshot showing WLST Summarizer HTML output

3. If a managed server has a "FAILED" health state or there is 1 or more thread hogs in its JVM thread pool, three thread dumps will be captured (15 seconds apart).

Download Sample Script

You can grab the sample script by clicking here.

Instructions to Run:

1. After download, extract the zip to the machine hosting the WebLogic environment. You should have three directories along with a readme.txt

  • output
  • Sample_Output
  • scripts

2. In the scripts directory, find the start wrapper script (Unix) or startWLSTDomainSummarizer.cmd (MS Windows). Open the appropriate file in an editor and change the environment variable settings to suit your system.

Example - startWLSTDomainSummarizer.cmd

set WL_HOME=D:\product\FMW11g\wlserver_10.3
set DOMAIN_HOME=D:\product\FMW11g\user_projects\domains\MyDomain
set WLST_OUTPUT_PATH=D:\WLSTDomainSummarizer\output\
set WLST_OUTPUT_FILE=WLST_Domain_Summary_Via_MBeans.html

call "%WL_HOME%\common\bin\wlst.cmd"

Note: The WLST_OUTPUT_PATH directory value must have a trailing slash. If there is no trailing slash, the script will error and not continue. 

3. Run the shell / command line wrapper script. It should launch WLST and kick off "". This initial python script will hit you with some prompts e.g.

Is your domain Admin Server up and running and do you have the connection details? (Y /N ):
Enter connection URL to Admin Server e.g t3:// :
Enter weblogic username:
Enter weblogic username password (function prompt 1):

(Note: the value typed in for password will not be echoed back to the console).

Depending on your responses it will call "" or "". 

4. If the scripts run successfully, you should get a HTML summary in the specified output directory. 

5. For the HTML to render correctly, ensure the .js and .css files provided (review the output directory created by the zip file extraction) are accessible. For example, to view the HTML locally (without using a web server), place the HTML output, jquery-ui.js, spry.js and wlstsummarizer.css in the same directory.


This is a sample script. I have tested it against WebLogic Server 10.3.6 domains on MS Windows and Unix.  I cannot guarantee that the script will run error free or produce the expected output on your system. If you have any feedback add a comment to the blog. I will endeavour to fix any problems with my WLST code.


Spry (Adobe) :

Friday Apr 19, 2013

WLST Offline Script to Create Summary of WebLogic Domain


I have always believed it is important to find out the bigger picture early on in a problem investigation. For example, if a managed server is exhibiting problematic behaviour, as well as I looking at the logs, I like to know the context - for example:

  • Is the server part of a cluster?
  • How many servers / clusters are in the Weblogic Domain?
  • Is the domain spread across multiple machines?
  • Are System Resources such JDBC and JMS in play?

Weblogic Server scripting ( WLST ) has been around for many years. Consequently there are a large number of examples to be found in blogs, websites and forums. The vast majority assume you can run the script in online mode. What if, however, a WLST online connection cannot be established? Ok, sure we can manually peruse the DOMAIN_HOME/config/config.xml. It is not a friendly a solution partly because some of the configuration is held in other xml files referenced by the config.xml. There has to be another way, I thought, after all WebLogic Server holds domain configuration in MBeans. This got me thinking about WLST.

WLST's readDomain Function

We can use WLST's readDomain function, in offline mode, to load the domain MBean configuration hierarchy / tree. It is as simple this:

Example: Unix Machine:


Now you are ready to navigate the "tree" using "cd" and "ls" style syntax


drw-   AnyMachine
drw-   AppDeployment
drw-   Cluster
drw-   EmbeddedLDAP
drw-   FileStore
drw-   JDBCSystemResource
drw-   JMSServer
drw-   JMSSystemResource
drw-   Library
drw-   MigratableTarget
drw-   Security
drw-   SecurityConfiguration
drw-   Server
drw-   ShutdownClass
drw-   StartupClass
drw-   WLDFSystemResource
-rw-   Active                                        false
-rw-   AdminServerName                               AdminServer
-rw-   AdministrationMBeanAuditingEnabled            false
-rw-   AdministrationPort                            9002
.... etc
wls:/offline/MyDomain/cd ('Server')
drw-   AdminServer
drw-   MyManagedServer
drw-   MyManagedServer01
wls:/offline/MyDomain/Server/cd ('MyManagedServer')
drw-   DataSource
drw-   NetworkAccessPoint
drw-   SSL
drw-   ServerDiagnosticConfig

-rw-   AcceptBacklog                                 300
-rw-   AdminReconnectIntervalSeconds                 10
-rw-   AdministrationPort                            0
-rw-   AdministrationPortEnabled                     false
-rw-   AdministrationProtocol                        null
-rw-   JavaCompilerPostClassPath                     null
-rw-   JavaCompilerPreClassPath                      null
-rw-   JavaStandardTrustKeyStorePassPhraseEncrypted  null
-rw-   JdbcLoginTimeoutSeconds                       0
-rw-   KeyStores                                     null
-rw-   ListenAddress                       
-rw-   ListenDelaySecs                               0
-rw-   ListenPort                                    7021
-rw-   ListenPortEnabled                             true

... etc
wls:/offline/MyDomain/Server/MyManagedServer/foo = get('ListenPort')
wls:/offline/MyDomain/Server/MyManagedServer/print foo



Sample Script and Output

To write a script, which automates the capture and print out of attributes useful to a summary, is challenging because the functions to traverse and search the MBean tree are limited compared to online mode. You are pretty much stuck with the cd(), ls() and get(). However, with a bit of coding creativity with IF and FOR loop syntax you can programmatically explore the tree and return output in a nice format - in my sample I have chosen HTML tables.

Sample Output

To view sample output, go here

Screenshot below shows snipper of sample output.

Download Sample Script

You can grab the sample script by clicking here.

Instructions to Run:

1. Edit the environment variables in the start wrapper script (Unix) or startWLSofflineCollection.cmd (MS Windows) to suit your system

Example -

DOMAIN_HOME=/oracle/middleware/user_projects/domains/MyDomain; export DOMAIN_HOME
WLST_OUTPUT_FILE=WLST_MBean_Config_Summary.html; export WLST_OUTPUT_FILE

Note: The WLST_OUTPUT_PATH directory value must have a trailing slash. If there is no trailing slash script will error and not continue.

2. Then run the wrapper script. It should launch WLST and run the script. The script will collect MBean values and render them in a HTML file.
(Once the script has invoked the WLST shell, the execution of the python script should not take longer than 5 to 10 seconds.)


This is a sample script. I have tested it against WebLogic Server 10.3.6 domains on MS Windows and Unix.  I cannot guarantee that the script will run error free or produce the expected output on your system. That said, a number of colleagues have also run the script and with their feedback I have ironed out a few problems. If you have any feedback add a comment to the blog. I will endeavour to fix any problems with my WLST code. 


I found the following blogs particularly useful.

Java / Oracle SOA blog

WLST By Examples

Some Pitfalls (which required coding around)

Checking whether Configuration MBean Hierarchy Exists

If a service or functionality, such as Clusters, JDBC, JMS, Filestore are not configured, the related MBean hierarchy will not be visible when listing at domain level. Example: If Cluster, JDBC, JMS and Filestore have not been configured, the ls() will show:


drw-   AnyMachine
drw-   AppDeployment
drw-   EmbeddedLDAP
drw-   Library
drw-   MigratableTarget
drw-   Security
drw-   SecurityConfiguration
drw-   Server
drw-   ShutdownClass
drw-   StartupClass
drw-   WLDFSystemResource

A programmatic attempt to cd to a directory not listed will return an error e.g.

Error: cd() failed. Do dumpStack() to see details.

To work-around, the script contains a custom findMBean function. The function places the contents of the ls() in an array using the returnmap parameter. We can then loop through the array and check whether a given string - the MBean directory name exists e.g.

def findMBean(v_pattern):
        # get a listing of everything in the current directory
    mydirs = ls(returnMap='true');

        v_compile_pattern = java.util.regex.Pattern.compile(v_pattern);

    found = 'Nope not here';

    for mydir in mydirs:
        x = java.lang.String(mydir);
        v_matched = v_compile_pattern.matcher(x);
        if v_matched.find():
            found = 'true';

        return found;

AnyMachine v Machines

In WebLogic Server 10.3, if a domain has one or more machines configured (i.e nodemanager related machines) the ls() at domain level will show


drw-   AnyMachine

A cd ('AnyMachine'), however, will fail. The correct cd syntax is

wls:/offline/MyDomain/cd ('Machines')

Identifying Target Information

Features such as JDBC System Resources, Clusters may or may not have targets. An attempt to cd to an MBean directory or get an MBean attribute which does not exist will throw an error and prevent the script from continuing. To work-around we can use TRY / EXCEPT syntax. Example - snippet from sample script which deals with the capture of JDBC System Resource targets.

            v_any_targets = '';
            v_jdbc_target = get('Target');

            # Even if the get fails, the variable is assigned a value of none, set the flag variable accordingly
            if v_jdbc_target == 'None':
                v_any_targets = 'None';
                v_no_of_targets = 1;
                # If the get has succeeded then set flag accordingly and obtain length of array returned by the get
                # The array length will be used to determine the HTML rowspan value

                v_any_targets ='Use v_jdbc_target';
                v_no_of_targets = len(v_jdbc_target);


            # Setting flag and rowspan variable here as well .. belt and braces
            v_any_targets = 'None';
            v_no_of_targets = 1;

            # The exception will still display to standard out, which may cause alarm
            # So adding this message telling the user the exception is expected and can be ignored
            print "IGNORE this exception";   

Wednesday Apr 17, 2013

Critical Patch Update for Oracle Fusion Middleware - CPU April 2013

The latest Critical Patch Update (CPU) has been released for Oracle products. Start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all "Critical Patch Updates" released to date, along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities 

On this page you will find the link to the

Oracle Critical Patch Update Advisory - April 2013

The advisory lists the support documents that cover the patch availability for all Oracle products.

For Oracle Fusion Middleware, go to:

Patch Set Update and Critical Patch Update April 2013 Availability Document [ID 1525152.1]

If you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

  • Critical Patch Update April 2013 Oracle Fusion Middleware Known Issues  [ID 1525130.1]
  • Critical Patch Update April 2013 Database Known Issues [ID 1523844.1]
And lastly, for an informal summary of what the Critical Patch Update fixes, check out the blog posts by "Oracle Software Security Assurance" team

April 2013 Critical Patch Update Released

Wednesday Apr 03, 2013

One Stop Shop For Your Middleware Product Certification Questions


A certification is a combination of Oracle and third-party products and operating systems that Oracle has tested and should work together.

In more detail, product certifications cover:

  • Platform/OS
  • Client, Browser and Java (JRE)
  • Database, Java (JDK)
  • Web Server Plug-Ins
  • Third Party Adapters and Interoperability
  • Network - IPv6
  • Oracle Virtual Assembly Builder

This month we are excited to announce two initiatives designed to help you quickly find the answers to your product certification questions:

"Get Proactive" My Oracle Support Knowledge Article

Review and bookmark this knowledge article

Get Proactive with Fusion Middleware : Find Product Certifications [ID 1532687.1]

The article contains:

  • Viewlets which provides hints and tips for finding product certification data
  • Product Breakdown to drill down to the location of product specific certifications.
  • FAQs
  • Information and links to related resources
Screenshot 1: ID 1532687.1

Screenshot of the new Get Proactive Certification Note

And a New Support Community!

Still have questions / issues. No need to create a Service Request. Instead post your Middleware product certification question to the Middleware Certification support community! A dedicated team of Oracle Support Engineers are monitoring the community and will respond to your enquiry in timely manner.

Screenshot 2: Middleware Certification Support Community

Snapshot of Cert Community

Note: A valid My Oracle Support license / login is required to access the knowledge article and support community.

Monday Feb 04, 2013

Fusion Middleware Performance Problems? Take Advantage of DMS


Why is my application running slowly?
How can I access configuration data and related metrics?
I need to a tool to help me tune my Fusion Middleware solution?

If you are asking these types of questions then take some time out to find out more about DMS. 

What is DMS?

Yes, another acronym .. sorry ... translation : "Dynamic Monitoring Service." It is a service which enables Oracle Fusion Middleware components to provide data regarding the component's performance, state and ongoing behavior. DMS measures and reports metrics, trace events and system performance and provides a context correlation service for these components.

Benefits include

  • Comprehensive set of performance metrics
  • Multiple tools for easier viewing and analysis of the metrics - including:
    • Fusion Middleware (EM) Control
    • DMS Spy Servlet
    • WLST
    • Remote Diagnostic Agent
    • Diagnostic Framework
  • Event Tracing
    • configure live tracing with no restarts
    • trace issues within a given time period or collect specific data at a specific time for a specific set of criteria

Oracle Fusion Middleware Performance and Tuning Guide 11g
Chapter 6 Oracle Dynamic Monitoring Service Performance Tuning

is the place to find out more.

Important to Note:

The majority of Oracle Fusion Middleware 11g WebLogic based products incorporate DMS in their product code. As such there is no special set up required. DMS works out of the box. For WebLogic based products Diagnostic Framework is provided via JRF (Java Required Files). If a WebLogic Managed Server does not have JRF applied, DFW will NOT be available. Domains created with standalone WebLogic Server will not, by default, have JRF applied. Refer to:

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
19 Scaling Your Environment
19.3.1 Applying Oracle JRF Template to a Managed Server or Cluster

If you have a login to My Oracle Support, also check out the knowledge article:

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

DMS and Fusion Middleware (EM) Control

As stated above there are many ways to integrate with DMS. In this blog post, I am going to focus on Fusion Middleware (EM) Control as I think it tackles the task rather well. In Fusion Middleware Control, the DMS metrics for a component (e.g. a WebLogic managed server) are accessed via the Performance Summary option.

Screenshot : Performance Summary Menu Option

The Performance Summary will show a default selection of metrics for the component - some displayed in chart form, others listed in tables. However, within a matter of minutes you can change the metric selection and configure a metric view which suits your purpose or analysis.

For example; the screenshot below shows charts which plots, by way of comparison, the CPU and Memory usage of two WebLogic Managed Servers residing in the same domain.

Screenshot: WebLogic Managed Server - CPU and Memory Comparison

How is this done? Well, first click the button "Show Metric Pallette". 

Screenshot : Show Metric Pallette

The metric pallette allows you to pick and choose from a wide range of metrics. It is worth exploring. In this example, CPU and Memory Usage under the "JVM Metrics" has been chosen for WC_Spaces1.

Screenshot : WC_Spaces1

Then we scrolled down and selected the same metrics for WC_Spaces2 which is found under "Related Targets". To plot metrics on the same chart, the trick is to drag the metric from the pallette onto the existing chart which is showing just the one value. It's a bit difficult to show this in a screenshot :-) Just give it a go.

Screenshot: WC_Spaces2

You can also play around with the metric time period by enabling the Slider, or by specifying default time selection periods (15 minutes, 30 minutes, 1 hour), or by specifying your own start and end date / times.

Screenshot: Metric Time Period

And a final tip. Try using the Overlay menu option to build a chart which shows a data comparison - for example another Oracle WebLogic Server or Yesterday's data. (You will only see the latter option if you have been logged into Fusion Middleware (EM) Control for over 24 hours.)

Screenshot: Overlay

I hope this gives a flavor of what Dynamic Monitoring Service has to offer and that it can help you resolve performance related issues.

Friday Jan 11, 2013

Guide to Finding Fusion Middleware "End of Support" Information

When does support end for my Fusion Middleware version? Such a simple question and frequently asked. But there is more to the answer than you might expect. Furthermore, finding the full detail can be challenging.

In this blog entry, I will describe the navigation path through the relevant documents - all of which need to be read not only to obtain the "end of support" dates, but to also gain an understanding of Oracle Lifetime Support and Error Correction (bug fixing) policies.

The starting point is the Oracle Lifetime Support page

This page lays out the Oracle Lifetime Support terms, definitions and benefits. Once these are understood, go to the product specific Lifetime Support documents listed on the page. The direct link to the Fusion Middleware document is:

This is a weighty tome (60 + pages) which contains the actual "end of support" dates. The challenge is to pick out the dates for the different Middleware products which form your chosen technology stack / solution. On page 8, there is a table labelled - "Fusion Middleware Releases". While some dates are listed in this table, for 10.1.3, 11g and 12c you are referred to the footnotes e.g.

Screenshot of rows in Fusion Middleware Release Table on Page 8

To quote the footnote

"... For specific dates refer to the respective sections in this document. To find a section, refer to the Table of Contents."

Therefore, be prepared to jump to different sections to find the dates which are specific to the products you have chosen for your Middleware solution.

There is one other very important footnote on this page, to quote:

"For more-detailed information on bug fix and patch release policies, please refer to the “Error Correction Support Policy” on My Oracle Support ."

This statement leads you down the path to find "Error Correction Support" dates for given patch set version numbers. Establishing the "Error Correction Support" dates for your solution is key because it will inform your plans for patching / maintenance. 

For Fusion Middleware, you can cut to the chase and go straight to:

Error Correction Support Dates for Oracle Fusion Middleware 11g (11.1.1/11.1.2) [ID 1290894.1]

Thursday Dec 13, 2012

How to Run Apache Commands From Oracle HTTP Server 11g Home

Every now and then you come across a problem when there is nothing in the "troubleshooting manual" which can help you. Instead you need to think outside the box. This happened to me two or three years back. Oracle HTTP Server (OHS) 11g did not start. The error reported back by OPMN was generic and gave no clue, and worse the HTTP Server error log was empty, and remained so even after I had increased the OPMN and HTTP Server log levels. After checking configuration files, operating system resources, etc I was still no nearer the solution.

And then the light bulb moment!

OHS is based on Apache - what happens if I attempt to start HTTP Server using the native apache command. Trouble was the OHS 11g solution has its binaries and configuration files in separate "home" directories

  • ORACLE_HOME contains the binaries
  • ORACLE_INSTANCE contains the configuration files

How to set the environment so that native apache commands run without error? Eventually, with help from a colleague, the knowledge article

How to Start Oracle HTTP Server 11g Without Using opmnctl [ID 946532.1]

was born!

To be honest, I cannot remember the exact cause and solution to that OHS problem two or three years ago. But, I do remember that an attempt to start HTTP Server using the native apache command threw back an error to the console which led me to discover the culprit was some unusual filesystem fault.

The other day, I was asked to review and publish a new knowledge article which described how to use the apache command to dump a list of static and shared loaded modules. This got me thinking that it was time
[ID 946532.1] was given an update. The result

How To Run Native Apache Commands in an Oracle HTTP Server 11g Environment [
ID 946532.1]


  • Title change
  • Improved environment setting scripts
    • Interactive, should be no need to manually edit the scripts (although readers are welcome to do so)
    • Automatically dump out some diagnostic information
  • Inclusion of some links to other troubleshooting collateral

To view the knowledge article you need a My Oracle Support login. For convenience, you can obtain the scripts via the links below.

MS Windows:

Wrapper cmd script - calls main cmd script [After download, remove the ".txt" file extension]
Main cmd script - sets OHS 11g environment to run Apache commands [After download, remove the ".txt" file extension]


Shell script - sets OHS 11g environment to run Apache commands on Unix

Please note: I cannot guarantee that the scripts held in the blog repository will be maintained. Any enhancements or faults will applied to the scripts attached to the knowledge article.

Lastly, to find out more about native apache commands, refer to the Apache Documentation

apachectl - Apache HTTP Server Control Interface

httpd - Apache Hypertext Transfer Protocol Server

Tuesday Nov 20, 2012

RDA and Fusion Middleware Diagnostic Framework Integration

Further to my last blog entry "FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!" I have spent some time exploring how Remote Diagnostic Agent (RDA) integrates with Diagnostic Framework.

Remote Diagnostic Agent, by default, collects the information about the last 10 incidents which have been captured in the Automatic Diagnostic Repository (ADR). This information can be located via RDA's Start Page Menu system. See screenshot below.

Screenshot - Viewing Diagnostic Framework Incident Information in a RDA Package

Note: In the next release of RDA - version 4.30 - the Diagnostic Repository menu label will have it's own position in the weblogic managed server sub menu hierarchy rather than be a child menu item of the logs menu

Diagnostic Framework is also capable of launching RDA engine and including the output in an incident package. This is achieved via the command "IPS GENERATE PACKAGE". The RDA output is written to

DOMAIN_HOME/servers/<server name>/adr/diag/ofm/<domain name>/<server name>/

The RDA collected files are best viewed (as shown in the screenshot above) by opening the RDA Start Page - "DFW__start.htm" - from this directory in a browser. If you do not have a browser available on the host machine, zip the contents of the rda directory and transfer and extract to a machine which has a browser. 

The explanation of the integration goes a little deeper. If you want to know more, read My Oracle Support document:

Understanding RDA and FMW 11g Diagnostic Framework Integration [Document 1503644.1]

Friday Nov 02, 2012

FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!


There is nothing more frustrating than a problem that "cannot be reproduced". Logs, configuration files have been analysed but there just isn't enough information to establish the root cause. The issue maybe closed, but you are left with the feeling that the problem will raise its ugly head again in the future. Trouble is, to resolve such issues you need to capture diagnostic data at the exact time the incident occurs. Step forward Fusion Middleware Diagnostic Framework! 

Diagnostic Framework monitors WebLogic Managed Servers and delivers "Automatic capture of diagnostic data upon first failure". To quote from

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

"When a critical error occurs ... the Diagnostic Framework automatically collects diagnostics, such as thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps ... The data is stored in a file-based repository and is accessible with command-line utilities."

In other words the data collected upon first failure - especially the thread and image dumps - provides a snapshot of the system as or immediately after the problem occurs. The table below shows the type of WebLogic Server issues which fall into the scope of Diagnostic Framework

How to Configure Diagnostic Framework?

Depending on your Fusion Middleware product choice you may not need to do anything! Diagnostic Framework is automatically installed, configured and initiated for any WebLogic Domain which has the Oracle Java Required Files (JRF) template applied. This template is applied by default whenever you configure WebLogic Managed Servers for products such as

  • Portal / Forms / Reports / Discoverer
  • Identity Management ( OID , OAM , OIM etc)
  • WebCenter
  • SOA

Check your WebLogic Domain directory structure. If you have an "adr" sub directory under


then JRF template has been applied and Diagnostic Framework will be in play.

Should the "adr" sub directory not exist, review the advice given in My Oracle Support article

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

If you are working with a standalone WebLogic Server solution and applying Oracle JRF is not acceptable, consider using WLDF - WebLogic Diagnostic Framework. (Fusion Middleware Diagnostic Framework makes use of WLDF under the covers.) Couple of useful links about WLDF are listed below

How to Get Started With Diagnostic Framework

To be frank, the Fusion Middleware Administrator's Guide is the best place to start your learning

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

A lot of reading here,  but if you are in hurry and just want to get the right information to Oracle Support to help resolve your issue, check out the next section below.

How to Upload Diagnostic Framework Incident Data to Oracle Support

Some Background Information

There are three interfaces to the Repository:

  1. Enterprise Manager Cloud Control (Support Workbench)
  2. WLST (Command Line)
  3. ADRCI (Command Line)

The Enterprise Manager Cloud Control does provide a nice GUI interface to search, view and package diagnostic framework incidents. However, this software is not to be confused with Fusion Middleware (EM) Control. Cloud Control (formerly known as Grid Control) is part of the Enterprise Manager media package. EM Cloud Control has it's own install and configuration story. Therefore, for the benefit of those yet to install and play with Cloud Control, I am going to describe how to use the command line tools.

Ideally, you would only need to one command line interface, but currently I suggest using both - mainly due to the fact that ADRCI SHOW INCIDENTS does not reveal the description behind the Diagnostic Framework error code.



WLST and ADRCI are case sensitive when it comes to handling parameter values. If you make a mistake, expect an unfriendly syntax error message.

1) Find the incident


The managed server which you are troubleshooting must be up and running. If the managed server is down, ensure the domain's Admin Server is accessible. If you cannot connect to the Admin Server or the Managed Server the example WLST commands will not work.

a) Launch WLST 

Note: Use the WLST which resides in the "oracle_common" directory (not WL_HOME/common/bin) otherwise you will get a syntax error like the one below

Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: listIncidents


b) Connect to the managed server or the admin server e.g.

wls:/offline> connect('weblogic','welcome1','t3://localhost:7020')

c) Run the command

wls:/mydomain/serverConfig> listIncidents()

This will list the incidents for the server to which you have connected. If you have connected to the Admin Server and want to list the incidents for a managed server within the domain, use the command

wls:/mydomain/serverConfig> listIncidents(adrHome='diag\ofm\mydomain\mymanagedserver'

Example output

Incident Id     Problem Key              Incident Time
        1       DFW-99998 [java.lang.NullPointerException]
Fri Nov 02 10:38:46 GMT 2012

 The piece highlighted in bold is the description you do not see when using the ADRCI 'SHOW INCIDENT' command.

Make a note of the incident id. You are ready to move to step 2

2. Package the incident

a) Set up the environment - example commands below are for Unix

cd <DOMAIN_HOME>/bin
. ./


If you want ADRCI to run a Remote Diagnostic Agent collection (recommended) at generate package time, point ORACLE_HOME at oracle_common

ORACLE_HOME=$MW_HOME/oracle_common; export ORACLE_HOME

To prevent ADRCI from running RDA at generate package time, point ORACLE_HOME at WL_HOME/server/adr directory. 


b) Launch adrci



adrci> SET BASE /oracle/middleware/user_projects/domains/
adrci> SET HOMEPATH diag/ofm/mydomain/mymanagedserver

d)  Optionally run SHOW INCIDENTS e.g.

ADR Home = /oracle/middleware/user_projects/domains/mydomain/

   INCIDENT_ID                   1
   STATUS                        ready
   CREATE_TIME                   2012-11-02 10:38:46.468000 +00:00
   PROBLEM_ID                    1
   CLOSE_TIME                    <NULL>
   FLOOD_CONTROLLED              none
   ERROR_FACILITY                DFW
   ERROR_NUMBER                  99998
   ERROR_ARG1                    <NULL>
   ERROR_ARG2                    <NULL>
   ERROR_ARG3                    <NULL>
   ERROR_ARG4                    <NULL>
   ERROR_ARG5                    <NULL>
   ERROR_ARG6                    <NULL>
   ERROR_ARG7                    <NULL>
   ERROR_ARG8                    <NULL>
   ERROR_ARG9                    <NULL>
   ERROR_ARG10                   <NULL>
   ERROR_ARG11                   <NULL>
   ERROR_ARG12                   <NULL>
   ECID                          5162744c6a2eea5e:155ff445:13ac0aae7cb:-8000-000
   IMPACTS                       0
1 rows fetched

e)  Create a logical package



Created package 1 based on incident id 1, correlation level typical

f) Generate the package

IPS GENERATE PACKAGE package_number IN path


Generated package 1 in file /tmp/, mode complete


If the generate package command hangs, ADRCI may be experiencing an issue when running RDA. To avoid such trouble, exit ADRCI and point the ORACLE_HOME environment variable at WL_HOME/server/adr

3) Upload the package zip to Oracle Support via your Service Request

a) Log into My Oracle Support and locate your Service Request

b) Click on "Add Attachments

c) And upload the zip file

Wednesday Oct 31, 2012

How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?


There are many ways to generate a thread dump of a WebLogic Managed Server. For example, take a look at:

Taking Thread Dumps - [an excellent blog post on the Middleware Magic site]


Different ways to take thread dumps in WebLogic Server (Document 1098691.1)

There is another method - use Remote Diagnostic Agent! The solution described below is not documented, but it is relatively straightforward to execute. One advantage of using RDA to collect the thread dumps is RDA will also collect configuration, log files, network, system, performance information at the same time.


1. Not familiar with Remote Diagnostic Agent? Take a look at my previous blog "Resolve SRs Faster Using RDA - Find the Right Profile"

2. Choose a profile, which includes the WebLogic Server data collection modules (for example the profile "WebLogicServer"). At RDA setup time you should see the prompt below:

S301WLS: Collects Oracle WebLogic Server Information
Enter the location of the directory where the domains to analyze are located
(For example in UNIX, <BEA Home>/user_projects/domains or <Middleware
Hit 'Return' to accept the default

For a successful WLS connection, ensure that the domain Admin Server is up and

Data Collection Type:
  1  Collect for a single server (offline mode)
  2  Collect for a single server (using WLS connection)
  3  Collect for multiple servers (using WLS connection)

Enter the item number
Hit 'Return' to accept the default (1)
> 2

Choose option 2 or 3.

Note: Collect for a single server or multiple servers using WLS connection means that RDA will attempt to connect to execute online WLST commands against the targeted server(s). The thread dumps are collected using the WLST function - "threadDumps()". If WLST cannot connect to the managed server, RDA will proceed to collect other data and ignore the request to collect thread dumps. If in the final output you see no Thread Dump menu item, then it's likely that the managed server is in a state which prevents new connections to it. If faced with this scenario, you would have to employ alternative methods for collecting thread dumps.

3. The RDA setup will create a setup.cfg file in the RDA_HOME directory. Open this file in an editor. You will find the following parameters which govern the number of thread dumps and thread dump interval.

#N.Number of thread dumps to capture
#N.Thread dump interval

The example lines above show the default settings. In other words, RDA will collect 10 thread dumps at 5000 millisecond (5 second) intervals. You may want to change this to something like:

#N.Number of thread dumps to capture
#N.Thread dump interval

However, bear in mind, that such change will increase the total amount of time it takes for RDA to complete its run.

4. Once you are happy with the setup.cfg, run RDA. RDA will collect, render, generate and package all files in the output directory.

5. For ease of viewing, open up the RDA Start html file - "xxxx__start.htm". The thread dumps can be found under the WLST Collections for the target managed server(s). See screenshots below

Screenshot 1:RDA Start Page - Main Index

Screenshot of RDA Start Page

Screenshot 2: Managed Server Sub Index

Screenshot of Managed Server Sub Index

Screenshot 3: WLST Collections

Screenshot of WLST Collections - pointing out Thread Dump menu option

Screenshot 4: Thread Dump Page - List of dump file links

Screenshot of Thread Dump Page - list of dump file links

Screenshot 5: Thread Dump Dat File Link

Screenshot of what you see when you click a dump file link

Additional Comment:

A) You can view the thread dump files within the RDA Start Page framework, but most likely you will want to download the dat files for in-depth analysis via thread dump analysis tools such as:

If you are new to thread dump analysis - take a look at this recorded Support Advisor Webcast 

Oracle WebLogic Server: Diagnosing Performance Issues through Java Thread Dumps
[Slidedeck from webcast in PDF format]

B) I have logged a couple of enhancement requests for the RDA Development Team to consider:

  1. Add timestamp to dump file links, dat filename and at the top of the body of the dat file
  2. Package the individual thread dumps in a zip so all dump files can be conveniently downloaded in one go.

Wednesday Oct 17, 2012

Critical Patch Update For Oracle Fusion Middleware - CPU October 2012

The latest Critical Patch Update (CPU) has been released for Oracle products. Start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all "Critical Patch Updates" released to date, along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities 

On this page you will find the link to the

Oracle Critical Patch Update Advisory - October 2012

The advisory lists the support documents that cover the patch availability for all Oracle products.

From an Oracle Fusion Middleware perspective, you can cut to the chase by using the links below which take you to the appropriate sections in

Patch Set Update and Critical Patch Update October 2012 Availability Document [ID 1477727.1]

  1. Oracle Fusion Middleware 11g Release 2 
  2. Oracle Fusion Middleware 11g Release 1
  3. Oracle Application Server 10g Release 3
The #anchor links above should work in Firefox and IE provided you have already logged into My Oracle Support within the same browser session. For some reason, Chrome always takes you to the top of the document :-/


Error Correction Support for Oracle Identity Management 10g, version 10.1.4.x ended in December 2011. For this reason, there is no section which is specific to this version. However, Error Correction Support remains in place, until end of this year, for the Oracle Identity Management 10.1.4.x components

    • Single Sign On (SSO)
    • Delegated Administration Services (OIDDAS)

provided you are using them as part of a Single Sign-On solution (OID 11g + SSO / OIDDAS for a Portal / Forms / Reports and Discoverer 11.1.1.x architecture.   

As such there are security related patches available for Fusion Middleware Single Sign On. You will find the patch numbers listed in the sections for, and

And finally, if you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

Critical Patch Update October 2012 Oracle Fusion Middleware Known Issues (Doc ID 1455408.1)

Critical Patch Update October 2012 Database Known Issues (Doc ID 1477865.1)

Monday Sep 10, 2012

Resolve SRs Faster Using RDA - Find the Right Profile


Remote Diagnostic Agent (RDA) is an excellent command-line data collection tool that can aid troubleshooting / problem solving. The tool covers the majority of Oracle's vast product range, and its data collection capability is comprehensive. RDA collects data about

  • the operating system and environment, including
    • environment variable, kernel settings
    • network
    • o/s performance
    • o/s patches
    • and much more
  • the Oracle Products installed, including
    • patches
    • logs and debug
    • metrics
    • configuration
    • and much more

In effect, RDA can obtain a snapshot of an Oracle Product and its environment. Oracle Support encourages the use of RDA because it greatly reduces service request resolution time by minimizing the number of requests from Oracle Support for more information. RDA is designed to be as unobtrusive as possible; it does not modify systems in any way. It collects useful data for Oracle Support only and a security filter is provided if required.

RDA can be downloaded via this My Oracle Support document

Remote Diagnostic Agent (RDA) 4 - Getting Started [ID 314422.1]

Find and Use the Right RDA Profile

One problem of any tool / utility, which covers a large range of products, is knowing how to target it against only the products you wish to troubleshoot. RDA does not have a GUI. Nor does RDA have an intelligent mechanism for detecting and automatically collecting data only for those Oracle products installed. Instead, you have to tell RDA what to do.

There is a mind boggling large number of RDA data collection modules which you can configure RDA to use. It is easier, however, to setup RDA to use a "Profile". A profile consists of a list of data collection modules and predefined settings. As such profiles can be used to diagnose a problem with a particular product or combination of products.

How to run RDA with a profile?

( <rda> represents the command you selected to run RDA (for example,, rda.cmd,, and perl

1. Use the embedded spreadsheet to find the RDA profile which is appropriate for your problem / chosen Oracle Fusion Middleware products.

2. Use the following command to perform the setup

<rda> -S -p <profile_name> 

3. Run the data collection


Run the data collection. If you want to perform setup and run in one go, then use a command such as the following:

<rda> -vnSCRP -p <profile name>

For more information, refer to:

Remote Diagnostic Agent (RDA) 4 - Profile Manual Pages [ID 391983.1]

Additional Hints / Tips:

1. Be careful! Profile names are case sensitive.

2. When profiles are not used, RDA considers all existing modules by default. For example, if you have downloaded RDA for the first time and run the command

<rda> -S

you will see prompts for every RDA collection module many of which will be of no interest to you. Also, you may, in your haste to work through all the questions, forget to say "Yes" to the collection of data that is pertinent to your particular problem or product. Profiles avoid such tedium and help ensure the right data is collected at the first time of asking.

3. RDA has a security filter which removes potentially sensitive data such as

  • Domain names
  • Group names
  • Host names
  • IPv4 and IPv6 addresses
  • LDAP domain components
  • Network masks
  • User names

You can switch on at RDA setup by appending "-Security" to the profile value

<rda> -vnSCRP -p <profile name>-Security


<rda> -vnSCRP -p FM11g_WlsWebTier-Security

If you already have a setup.cfg, you can enable the security filter by running the command

<rda> -X Filter enable 

Monday Aug 13, 2012

Find Oracle FMw 11g Other Certification Information - Client/Browser, 3rd Party, Product Interoperability

Here is the fourth and final viewlet in a series of viewlets which discuss how to find Oracle Fusion Middleware certification information. The viewlet below explores the process of searching for Oracle Fusion Middleware 11g Client, 3rd Party and Interoperability certification information. The objective is to help you find the answers to questions such as

  • Which versions of WebLogic Server are compatible with the latest FMw 11g patchset?
  • Which browsers are certified e.g. Is Forms / Reports 11g certified to work with IE 9?
  • Is it supported to integrate FMW 11g products with Websphere?
  • Does FMW 11g support IPv6?

[Note - the viewlets have no sound and use the swf (FlashPlayer) format. It will take approximately 10 to 15 minutes to work through the show.

If you are unable to use Flash Player, click here to grab a PDF version of the presentation.]

If you have not done so already, check out the previous viewlets in the series

Friday Aug 03, 2012

How to Find Oracle Application Server 10g Certification Information?

Here is the third in a series of viewlets which discuss how to find Oracle Fusion Middleware certification information. The viewlet below covers Oracle Application Server 10g - with hints / tips relevant to searching for certification information related to versions

  • Oracle Application Server 10g Release 3 (10.1.3) - including SOA Suite
  • Oracle Application Server 10g Release 2 (10.1.2)
  • Oracle Identity Management 10g Release 3 (10.1.4)

[Note - the viewlets have no sound and use the swf (FlashPlayer) format. It will take approximately 10 to 15 minutes to work through the show.

If you are unable to use Flash Player, click here to grab a PDF version of the presentation.]

If you have not done so already, check out the other viewlets in the series


This is the blog of the Oracle Fusion Middleware Proactive Support Delivery Team. Here we will provide information about our activities, publications, product related information and more. Feedback welcome.

Follow OracleMWSupport on Twitter


« April 2014