Friday Apr 19, 2013

WLST Offline Script to Create Summary of WebLogic Domain

Introduction

I have always believed it is important to find out the bigger picture early on in a problem investigation. For example, if a managed server is exhibiting problematic behaviour, as well as I looking at the logs, I like to know the context - for example:

  • Is the server part of a cluster?
  • How many servers / clusters are in the Weblogic Domain?
  • Is the domain spread across multiple machines?
  • Are System Resources such JDBC and JMS in play?

Weblogic Server scripting ( WLST ) has been around for many years. Consequently there are a large number of examples to be found in blogs, websites and forums. The vast majority assume you can run the script in online mode. What if, however, a WLST online connection cannot be established? Ok, sure we can manually peruse the DOMAIN_HOME/config/config.xml. It is not a friendly a solution partly because some of the configuration is held in other xml files referenced by the config.xml. There has to be another way, I thought, after all WebLogic Server holds domain configuration in MBeans. This got me thinking about WLST.


WLST's readDomain Function

We can use WLST's readDomain function, in offline mode, to load the domain MBean configuration hierarchy / tree. It is as simple this:

Example: Unix Machine:

WL_HOME=/oracle/middleware/wlserver_10.3
DOMAIN_HOME=/oracle/middleware/user_projects/domains/MyDomain
$WL_HOME/common/bin/wlst.sh
wls:/offline/readDomain('/oracle/middleware/user_projects/domains/MyDomain')

Now you are ready to navigate the "tree" using "cd" and "ls" style syntax

wls:/offline/MyDomain/ls()

drw-   AnyMachine
drw-   AppDeployment
drw-   Cluster
drw-   EmbeddedLDAP
drw-   FileStore
drw-   JDBCSystemResource
drw-   JMSServer
drw-   JMSSystemResource
drw-   Library
drw-   MigratableTarget
drw-   Security
drw-   SecurityConfiguration
drw-   Server
drw-   ShutdownClass
drw-   StartupClass
drw-   WLDFSystemResource
-rw-   Active                                        false
-rw-   AdminServerName                               AdminServer
-rw-   AdministrationMBeanAuditingEnabled            false
-rw-   AdministrationPort                            9002
.... etc
wls:/offline/MyDomain/cd ('Server')
wls:/offline/MyDomain/Server/ls()
drw-   AdminServer
drw-   MyManagedServer
drw-   MyManagedServer01
wls:/offline/MyDomain/Server/cd ('MyManagedServer')
wls:/offline/MyDomain/Server/MyManagedServer/ls()
drw-   DataSource
drw-   NetworkAccessPoint
drw-   SSL
drw-   ServerDiagnosticConfig

-rw-   AcceptBacklog                                 300
-rw-   AdminReconnectIntervalSeconds                 10
-rw-   AdministrationPort                            0
-rw-   AdministrationPortEnabled                     false
-rw-   AdministrationProtocol                        null
...
-rw-   JavaCompilerPostClassPath                     null
-rw-   JavaCompilerPreClassPath                      null
-rw-   JavaStandardTrustKeyStorePassPhraseEncrypted  null
-rw-   JdbcLoginTimeoutSeconds                       0
-rw-   KeyStores                                     null
-rw-   ListenAddress                                 127.0.0.1
-rw-   ListenDelaySecs                               0
-rw-   ListenPort                                    7021
-rw-   ListenPortEnabled                             true

... etc
wls:/offline/MyDomain/Server/MyManagedServer/foo = get('ListenPort')
wls:/offline/MyDomain/Server/MyManagedServer/print foo

7021

 

Sample Script and Output

To write a script, which automates the capture and print out of attributes useful to a summary, is challenging because the functions to traverse and search the MBean tree are limited compared to online mode. You are pretty much stuck with the cd(), ls() and get(). However, with a bit of coding creativity with IF and FOR loop syntax you can programmatically explore the tree and return output in a nice format - in my sample I have chosen HTML tables.

Sample Output

To view sample output, go here

Screenshot below shows snipper of sample output.


Download Sample Script

You can grab the sample script by clicking here.

Instructions to Run:

1. Edit the environment variables in the start wrapper script startWLSofflineCollection.sh (Unix) or startWLSofflineCollection.cmd (MS Windows) to suit your system

Example - startWLSofflineCollection.sh

WL_HOME=/oracle/middleware/wlserver_10.3
DOMAIN_HOME=/oracle/middleware/user_projects/domains/MyDomain; export DOMAIN_HOME
WLST_OUTPUT_PATH=/temp/wlst/; export WLST_OUTPUT_PATH
WLST_OUTPUT_FILE=WLST_MBean_Config_Summary.html; export WLST_OUTPUT_FILE
${WL_HOME}/common/bin/wlst.sh CollectWLSConfigOffline.py

Note: The WLST_OUTPUT_PATH directory value must have a trailing slash. If there is no trailing slash script will error and not continue.

2. Then run the wrapper script. It should launch WLST and run the script. The script will collect MBean values and render them in a HTML file.
(Once the script has invoked the WLST shell, the execution of the python script should not take longer than 5 to 10 seconds.)

Disclaimer

This is a sample script. I have tested it against WebLogic Server 10.3.6 domains on MS Windows and Unix.  I cannot guarantee that the script will run error free or produce the expected output on your system. That said, a number of colleagues have also run the script and with their feedback I have ironed out a few problems. If you have any feedback add a comment to the blog. I will endeavour to fix any problems with my WLST code. 

Credits

I found the following blogs particularly useful.

Java / Oracle SOA blog
[http://biemond.blogspot.co.uk/2010/02/invoking-fmw-application-mbeans-in.html]

WLST By Examples
[http://wlstbyexamples.blogspot.co.uk/]


Some Pitfalls (which required coding around)

Checking whether Configuration MBean Hierarchy Exists

If a service or functionality, such as Clusters, JDBC, JMS, Filestore are not configured, the related MBean hierarchy will not be visible when listing at domain level. Example: If Cluster, JDBC, JMS and Filestore have not been configured, the ls() will show:

wls:/offline/MyDomain/ls()

drw-   AnyMachine
drw-   AppDeployment
drw-   EmbeddedLDAP
drw-   Library
drw-   MigratableTarget
drw-   Security
drw-   SecurityConfiguration
drw-   Server
drw-   ShutdownClass
drw-   StartupClass
drw-   WLDFSystemResource

A programmatic attempt to cd to a directory not listed will return an error e.g.

Error: cd() failed. Do dumpStack() to see details.

To work-around, the script contains a custom findMBean function. The function places the contents of the ls() in an array using the returnmap parameter. We can then loop through the array and check whether a given string - the MBean directory name exists e.g.

def findMBean(v_pattern):
        # get a listing of everything in the current directory
    mydirs = ls(returnMap='true');

        v_compile_pattern = java.util.regex.Pattern.compile(v_pattern);

    found = 'Nope not here';

    for mydir in mydirs:
        x = java.lang.String(mydir);
        v_matched = v_compile_pattern.matcher(x);
        if v_matched.find():
            found = 'true';

        return found;

AnyMachine v Machines

In WebLogic Server 10.3, if a domain has one or more machines configured (i.e nodemanager related machines) the ls() at domain level will show

wls:/offline/MyDomain/ls()

drw-   AnyMachine

A cd ('AnyMachine'), however, will fail. The correct cd syntax is

wls:/offline/MyDomain/cd ('Machines')

Identifying Target Information

Features such as JDBC System Resources, Clusters may or may not have targets. An attempt to cd to an MBean directory or get an MBean attribute which does not exist will throw an error and prevent the script from continuing. To work-around we can use TRY / EXCEPT syntax. Example - snippet from sample script which deals with the capture of JDBC System Resource targets.

try:
            v_any_targets = '';
            v_jdbc_target = get('Target');

            # Even if the get fails, the variable is assigned a value of none, set the flag variable accordingly
            if v_jdbc_target == 'None':
                v_any_targets = 'None';
                v_no_of_targets = 1;
            else:   
                # If the get has succeeded then set flag accordingly and obtain length of array returned by the get
                # The array length will be used to determine the HTML rowspan value

                v_any_targets ='Use v_jdbc_target';
                v_no_of_targets = len(v_jdbc_target);

        except:

            # Setting flag and rowspan variable here as well .. belt and braces
            v_any_targets = 'None';
            v_no_of_targets = 1;

            # The exception will still display to standard out, which may cause alarm
            # So adding this message telling the user the exception is expected and can be ignored
            print "IGNORE this exception";   

Wednesday Oct 31, 2012

How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?

Introduction

There are many ways to generate a thread dump of a WebLogic Managed Server. For example, take a look at:

Taking Thread Dumps - [an excellent blog post on the Middleware Magic site]

or 

Different ways to take thread dumps in WebLogic Server (Document 1098691.1)

There is another method - use Remote Diagnostic Agent! The solution described below is not documented, but it is relatively straightforward to execute. One advantage of using RDA to collect the thread dumps is RDA will also collect configuration, log files, network, system, performance information at the same time.

Instructions

1. Not familiar with Remote Diagnostic Agent? Take a look at my previous blog "Resolve SRs Faster Using RDA - Find the Right Profile"

2. Choose a profile, which includes the WebLogic Server data collection modules (for example the profile "WebLogicServer"). At RDA setup time you should see the prompt below:

-------------------------------------------------------------------------------
S301WLS: Collects Oracle WebLogic Server Information
-------------------------------------------------------------------------------
Enter the location of the directory where the domains to analyze are located
(For example in UNIX, <BEA Home>/user_projects/domains or <Middleware
Home>/user_projects/domains)
Hit 'Return' to accept the default
(/oracle/11AS/Middleware/user_projects/domains)
> 

For a successful WLS connection, ensure that the domain Admin Server is up and
running.

Data Collection Type:
  1  Collect for a single server (offline mode)
  2  Collect for a single server (using WLS connection)
  3  Collect for multiple servers (using WLS connection)

Enter the item number
Hit 'Return' to accept the default (1)
> 2

Choose option 2 or 3.

Note: Collect for a single server or multiple servers using WLS connection means that RDA will attempt to connect to execute online WLST commands against the targeted server(s). The thread dumps are collected using the WLST function - "threadDumps()". If WLST cannot connect to the managed server, RDA will proceed to collect other data and ignore the request to collect thread dumps. If in the final output you see no Thread Dump menu item, then it's likely that the managed server is in a state which prevents new connections to it. If faced with this scenario, you would have to employ alternative methods for collecting thread dumps.

3. The RDA setup will create a setup.cfg file in the RDA_HOME directory. Open this file in an editor. You will find the following parameters which govern the number of thread dumps and thread dump interval.

#N.Number of thread dumps to capture
WREQ_THREAD_DUMP=10
#N.Thread dump interval
WREQ_THREAD_DUMP_INTERVAL=5000

The example lines above show the default settings. In other words, RDA will collect 10 thread dumps at 5000 millisecond (5 second) intervals. You may want to change this to something like:

#N.Number of thread dumps to capture
WREQ_THREAD_DUMP=10
#N.Thread dump interval
WREQ_THREAD_DUMP_INTERVAL=30000

However, bear in mind, that such change will increase the total amount of time it takes for RDA to complete its run.

4. Once you are happy with the setup.cfg, run RDA. RDA will collect, render, generate and package all files in the output directory.

5. For ease of viewing, open up the RDA Start html file - "xxxx__start.htm". The thread dumps can be found under the WLST Collections for the target managed server(s). See screenshots below

Screenshot 1:RDA Start Page - Main Index

Screenshot of RDA Start Page

Screenshot 2: Managed Server Sub Index

Screenshot of Managed Server Sub Index

Screenshot 3: WLST Collections

Screenshot of WLST Collections - pointing out Thread Dump menu option

Screenshot 4: Thread Dump Page - List of dump file links

Screenshot of Thread Dump Page - list of dump file links

Screenshot 5: Thread Dump Dat File Link

Screenshot of what you see when you click a dump file link

Additional Comment:

A) You can view the thread dump files within the RDA Start Page framework, but most likely you will want to download the dat files for in-depth analysis via thread dump analysis tools such as:

If you are new to thread dump analysis - take a look at this recorded Support Advisor Webcast 

Oracle WebLogic Server: Diagnosing Performance Issues through Java Thread Dumps
[Slidedeck from webcast in PDF format]

B) I have logged a couple of enhancement requests for the RDA Development Team to consider:

  1. Add timestamp to dump file links, dat filename and at the top of the body of the dat file
  2. Package the individual thread dumps in a zip so all dump files can be conveniently downloaded in one go.

Monday May 21, 2012

Which JDK is my FMW 11g WebLogic Domain Configured to Use?

Introduction

If you are the administrator who ran the Fusion Middleware Installer or Configuration Wizard, the question in the title may seem a little daft. But, in fact the advice below may become handy should you face one of the following situations:

  • You have been asked to look after a Middleware Home which was previously built by someone else
  • You are configuring a new FMW 11g AS Instance, and unexpectedly hitting errors or hanging at the WebLogic Domain Creation time. For example, see:

    FMW 11g "IDM" or "Portal/Forms/Reports/Discoverer" Configuration Wizards on 64bit Platforms Hang at 0% 'Creating Domain'  [ID 865462.1]

    This support document describes what happens if you inadvertently attempt to configure an AS Instance with a mixture of 64bit (Oracle Home Binaries - ORACLE_HOME) and 32bit JDK (WebLogic Binary Home - WL_HOME)

There are two parts to the question:

  1. What version of JDK will the Configuration Wizard, by default, configure a new WebLogic Domain to use?
  2. Which JDK and version is being used by an existing WebLogic Domain?

JDK Used at WebLogic Domain Creation Time

The answer is in the default WebLogic environment script:

Unix

WL_HOME/common/bin/commEnv.sh

MS Windows

WL_HOME/common/bin/commEnv.cmd

(e.g WL_HOME is typically - MW_HOME/wlserver_10.3/common/bin/commEnv.sh)

Look for lines like the ones below (examples are from a WLS install on a Unix machine)

#JAVA_USE_64BIT, true if JVM uses 64 bit operations
JAVA_USE_64BIT=true
if [ -z "${JAVA_HOME}" -o -z "${JAVA_VENDOR}" ]; <---- This line means if 
JAVA_HOME OR JAVA_VENDOR is NOT set, so if either of them are empty 
the settings in the "then" part of the statement are applied. 
If JAVA_VENDOR and JAVA_HOME are both set then the if statement is skipped
then
      # Set up JAVA HOME
      JAVA_HOME="/oracle/SOAWC/jdk1.6.0_24"
      # Set up JAVA VENDOR, possible values are
      #Oracle, HP, IBM, Sun ...
      JAVA_VENDOR=Sun
      # PRODUCTION_MODE, default to the development mode
      PRODUCTION_MODE=""
fi

Important to Note: It is possible to set JAVA_HOME, JAVA_VENDOR environment variables prior to launching the Installer or Configuration Wizard. Therefore you cannot always assume that commEnv.sh is the source of these environment variable values. In the same session from which you launch the Installer or Configuration Wizard double check whether JAVA_HOME and JAVA_VENDOR have values e.g.

Unix:

echo $JAVA_HOME
echo $JAVA_VENDOR

MS Windows:

set $JAVA_HOME
set $JAVA_VENDOR 

If either of these environment variables are not set then you know that the values will be derived from the commEnv.sh.

The JAVA_HOME and JAVA_VENDOR values do not tell you whether the JDK is 32bit or 64bit. To find out this detail you need to run a switch to the bin directory where the JDK resides e.g

cd /oracle/SOAWC/jdk1.6.0_24/bin 

and then run the java version command. To quote from:

How to tell if Weblogic Server is running in 32-bit or 64-bit mode [ID 1066808.1]

"To identify if you have a 32-bit or 64-bit JDK installed you can run"

java -version

which will give you more information.

By running

java -d64 -version

you can test if the 64-bit mode is supported by your JVM. You will get an error message if this is a 32-bit only JVM. 32/64-bit-hybrid JVMs exist for some platforms. In this case you would have to include the -d64 flag to tell the JVM to run in 64-bit mode."

Which JDK is my existing WebLogic Domain using?

Typically, the JDK used by a WebLogic Domain will be the same as specified by commEnv.sh. However, you must check the domain's environment variable setting file:

Unix:

DOMAIN_HOME/bin/setDomainEnv.sh

MS Windows:

DOMAIN_HOME/bin/setDomainEnv.cmd

(e.g A DOMAIN_HOME might look something like MW_HOME/user_projects/domains/myDomain/)

Couple of examples (taken from installs on a Unix machine)

A Portal / Forms / Reports and/or Discoverer WebLogic Domain; the setDomainEnv has these lines

BEA_JAVA_HOME="/oracle/app/product/oracle/FMW11gR1PS1/jrockit_160_14_R27.6.5-32"
export BEA_JAVA_HOME

SUN_JAVA_HOME="/oracle/app/product/oracle/FMW11gR1PS1/jdk160_14_R27.6.5-32"
export SUN_JAVA_HOME

    if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
        JAVA_HOME="${BEA_JAVA_HOME}"
        export JAVA_HOME
    else
    if [ "${JAVA_VENDOR}" = "Sun" ] ; then
        JAVA_HOME="${SUN_JAVA_HOME}"
        export JAVA_HOME
    else
        JAVA_VENDOR="Oracle"
        export JAVA_VENDOR
        JAVA_HOME="/oracle/app/product/oracle/FMW11gR1PS1/jrockit_160_14_R27.6.5-32"
        export JAVA_HOME
    fi
    fi

A FMW 11g SOA Suite, the setDomainEnv has these lines

    if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
            JAVA_HOME="${BEA_JAVA_HOME}"
            export JAVA_HOME
    else
            if [ "${JAVA_VENDOR}" = "Sun" ] ; then
                    JAVA_HOME="${SUN_JAVA_HOME}"
                    export JAVA_HOME
            else
                    JAVA_VENDOR="Sun"
                    export JAVA_VENDOR
                    JAVA_HOME="/oracle/SOAWC/jdk1.6.0_24"
                    export JAVA_HOME
            fi
    fi

The JAVA_VENDOR environment variable is not normally set prior to running setDomainEnv.sh, in which case the JAVA_VENDOR will default to the "else" part of the statement (highlighted in Red)

The commEnv (.sh or .cmd) is invoked later on in the setDomainEnv script. In the commEnv script the if statement

if [ -z "${JAVA_HOME}" -o -z "${JAVA_VENDOR}" ]; 

is bypassed because both JAVA_VENDOR and JAVA_HOME have been set.

Again, the setDomainEnv.sh does not tell you whether the specified JAVA_HOME is 32bit or 64bit. The only way to find this information is to run the "java -version" as previously described.

Additional Useful Reference:

How To Change Type of JDK (Sun / JRockit) for FMW 11g Domain (ID 1058804.1)

Note - however I have added a comment to this support document as I believe the advice does not take into account the possibility of JAVA_VENDOR and JAVA_HOME being set before commEnv is launched.

About

This is the blog of the Oracle Fusion Middleware Proactive Support Delivery Team. Here we will provide information about our activities, publications, product related information and more. Feedback welcome.

Follow OracleMWSupport on Twitter

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today