James Bayer's Blog

For those of you that attended the Chicago event Jan 19th, the slides for the Application Grid Overview, Coherence and JPA with TopLink Grid have been posted.

Oracle Technology Network Developer Day - Application Grid Development

There are other cities still to come in the next several weeks.

http://events.oracle.com/search/search?start=&pageHitCount=10&group=Events&keyword=application+grid+development

Cheers,

James

If part of your application goes bad, you do not want issues to cascade to the rest of your application or the rest of the server.  Based on my recent discovery of Michael Nygard’s architecture writings, this post will be somewhat applicable to his Bulkhead pattern (see page 39) of his excellent presentation.   I’ve been told that his ReleaseIt! book is phenomenal by peers I respect, so it’s already on it’s way to my bookshelf.  WebLogic has a way to put a leash on a JSP with only a few minor deployment descriptor edits – read on for an example on how to not let one resource intensive JSP bring you down.

Actual Customer Problem

One of my customers has an interesting problem with a long-running JSP.  They use a 3rd party java library exposed via a JSP that brings back large amounts of content from a backend file system.  This can take tens of minutes to process each request.  Within the same application (WAR file) they also have many short-lived requests hitting normal JSPs like a traditional web application.  In some cases they were having stability problems when too many threads were busy (STUCK threads) on the long-running JSP, taking up lots of server memory and starving threads from the fast JSPs.  In some cases, they would have to restart the WLS instance to recover stability.  The reason - by default the application components and even the rest of WebLogic Server are all using threads from the same default Work Manager – which is used by WebLogic Server’s self-tuning thread pool.

Potential Solution Using Custom Work Manager and Constraint

In 9.x and above of WebLogic, you have the ability to use a WorkManager with different types of constraints to control the shared threads in WebLogic Server.  Here is a detailed article on Work Managers on OTN.

Work Managers can be used to put some constraints on a troublesome JSP such as the max number of threads and the max number of requests that are being serviced and queued (called a capacity constraint) simply by editing the web.xml and weblogic.xml deployment descriptors.  Requests over the capacity constraint will result in a 503 error code.  Using web.xml a specialize page can be shown in this case that provides a helpful message as to why this condition exists instead of the default 503 page and the application can stay up while limited access to the JSPs that hog server resources.

Simple Example

2 JSPs - normal.jsp and veryslow.jsp - the veryslow.jsp has a 30 second java.lang.Thread.sleep()

A simple Filter that applies to both JSPs that simply logs to standard out – This is included just to show that Filters are not affected by Work Managers – you don’t have to worry about a Russian Nested Doll issue to target a work manager successfully at a JSP, even if it has Filters.

503error.html page (for the informational message in the overload condition).  I included this to show you that this can be a custom pretty page and does not have to be the ugly Internal Server Error page that you see by default.

weblogic.xml with a custom Work Manager defined in  with constraints:

<?xml version="1.0" encoding="UTF-8"?>

<wls:weblogic-web-app xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd">

    <wls:weblogic-version>10.3.2</wls:weblogic-version>

    <wls:context-root>WorkManagerWeb</wls:context-root>

     <wls:work-manager>

        <wls:name>myCustomWorkManager</wls:name>

        <wls:max-threads-constraint>

            <wls:name>5MaxThreads</wls:name>

            <wls:count>5</wls:count>

        </wls:max-threads-constraint>

        <wls:capacity>

            <wls:name>7MaxCapacity</wls:name>

            <wls:count>7</wls:count>

        </wls:capacity>

    </wls:work-manager>

</wls:weblogic-web-app>

web.xml – this simply shows how you can target a custom Work Manager directly at a JSP/Servlet – It is probably more common to target a Work Manager at an entire web application – but we have a very targeted use case in this example.  In EJB modules, work managers are targeted per EJB or MDB.  The important part is looking at the init-param of the veryslow.jsp.  The 503 error code page is also defined here.

<?xml version="1.0" encoding="UTF-8"?>

<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">

  <display-name>WorkManagerWeb</display-name>

  <welcome-file-list>

    <welcome-file>normal.jsp</welcome-file>

  </welcome-file-list>

  <error-page>

    <error-code>503</error-code>

    <location>/503error.html</location>

  </error-page>

  <filter>

    <display-name>BarFilter</display-name>

    <filter-name>BarFilter</filter-name>

    <filter-class>foo.BarFilter</filter-class>

  </filter>

  <filter-mapping>

    <filter-name>BarFilter</filter-name>

    <url-pattern>*.jsp</url-pattern>

  </filter-mapping>

  <servlet>

    <servlet-name>VerySlowJSP</servlet-name>

    <jsp-file>veryslow.jsp</jsp-file>

    <init-param>

         <param-name>wl-dispatch-policy</param-name>

         <param-value>myCustomWorkManager</param-value>

    </init-param>

   </servlet>

</web-app>

Here’s an illustration with a max thread constraint of 5 and a capacity constraint of 7 showing what will happen when 10 requests hit veryslow.jsp at the same time.

Here is a JMeter report using 10 threads instantaneously hitting normal.jsp which uses the default Work Manager from WLS.  Notice that all finish executing right away which is exactly as we expect because there are no relevant constraints in the default Work Manager.

Here is a JMeter report showing 10 requests instantaneously hitting veryslow.jsp which uses the custom Work Manager with the constraints discussed before.  In this case we see 3 rejected requests (the last 3 in), 5 that complete in 30 seconds, and 2 that complete in 60 seconds.  This is exactly what we expect given our constraints since 2 threads will wait not only for their 30 second sleep, but also for the 30 second sleep for the thread that they are waiting on.

When you go to the browser and hit veryslow.jsp while the veryslow test is being run, you get the 503error.html page defined in web.xml to come up.

Here’s the access.log from the test – notice the 503 codes:

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 503 348 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 503 348 

127.0.0.1 - - [30/Dec/2009:11:35:32 -0600] "GET /WorkManagerWeb/normal.jsp HTTP/1.1" 200 304 

127.0.0.1 - - [30/Dec/2009:11:35:32 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 503 348 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396 

127.0.0.1 - - [30/Dec/2009:11:35:31 -0600] "GET /WorkManagerWeb/veryslow.jsp HTTP/1.1" 200 396

Here’s the Standard Out log – notice that both JSP’s use the Filter and it is not affected by using different work managers for JSPs.

Dec 30, 2009 11:33:20 AM - BarFilter [ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:20 AM - BarFilter [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:20 AM - BarFilter [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:20 AM - BarFilter [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - BarFilter [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - Normal JSP hit [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:21 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:51 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:51 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - BarFilter [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - BarFilter [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:33:52 AM - VerySlow JSP Starting to sleep [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:34:22 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'

Dec 30, 2009 11:34:22 AM - VerySlow JSP awake [ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'

I’ve attached the code for this example that includes both the war file and my Oracle Enterprise Pack for Eclipse project.

As you can see, it’s pretty easy to put constraints on individual JSPs, Servlets, EJB’s, MDB’s, and entire web applications using only a few deployment descriptor edits.  That’s a very powerful feature when you need it to keep hogs from bogging down other parts of your application or your WLS instance.

I only yesterday added Google Analytics and Feedburner to this blog so I can better understand what entries are most popular.  If you use RSS/Atom to subscribe to this blog, please use the new Feedburner feeds so I can track the traffic and write stuff more people want to read about.

Atom Feed

RSS 2.0 Feed

I could definitely be more popular in Africa and South America :)

Cheers, James

*Update 1/10/2010* I realized after my post that I forgot a few things.  1) You should always have a catchy image at the top of a blog post to add some visual context to your post (see TechCrunch for a blog that almost always follows this).  So I’ve added a NodeManager architecture diagram.  2) My colleague Phil Aston (co-author of the book I recommend below) pointed out a much simpler way of configuring the classpath to use StartScriptEnabled=true in nodemanager.properties which I refer to below.

One of my customers is new to WebLogic and asked if I could instruct them how to automatically start their WebLogic Server instances when the OS is booted.  Node Manager can be used for that and more.  I put together a quick start guide that might help you get started quickly if you’re new to Node Manager like I was earlier today.  I’ve translated most of that documentation into a quick blog post.  I’ve definitely cut some corners (using demo certs, default path to the domain dir and node manager dir, etc).  If you want to get a feel for Node Manager quickly taking those grains of salt, read-on.  There is no included service for *nix systems, you’ll have to create an xinetd service one yourself and there are examples of that in the documentation and in the book referenced below.

Introduction

This is a short guide to getting started with WebLogic Server Node Manager, the process agent used to start and stop WebLogic Server instances. This is not a substitute for the complete documentation, but merely intended to get up and running quickly. Upon completiion of these steps, WebLogic Server processes should be automatically managed by Node Manager. When an OS instance is rebooted, WebLogic Server should return to the state it was in prior to the reboot (either started or stopped).

Official documentation:

http://download.oracle.com/docs/cd/E12839_01/web.1111/e13740/nodemgr_config.htm

Professional Oracle WebLogic Server book with real-world advice and best practices for Node Manager starting on page 560:

http://www.amazon.com/Professional-Oracle-WebLogic-Server-Programmer/dp/0470484306

The book is highly recommended as I mentioned I cut some corners here for simplicity.

Starting Point

The assumption is that the WebLogic Server installation has been done and that the Node Manager Windows Service was installed as a part of the installation.

Any paths reflected below should be updated to reflect your directory structures. In fact, it probably would not be a bad idea to find/replace some of the following with the relevant strings:

• WebLogic Server installation path: D:\Oracle\wls11g
• Hostname: jbayer-us
• Domain path: D:\Oracle\wls11g\user_projects\domains\my_domain

Uninstall / Reinstall NodeManager Service

This needs to be done if you installed Node Manager with the standard installer because by default Node Manager binds to localhost. So as more OS instances are added, it will be required to have Node Manager instances on the network communicate to each other. This can only happen if Node Manager binds to an network interface that is available remotely.

• open cmd prompt
• run D:\Oracle\wls11g\wlserver_10.3\server\bin\setWLSEnv.cmd
• run D:\Oracle\wls11g\wlserver_10.3\server\bin\uninstallNodeMgrSvc.cmd
• Make backup copy of installNodeMgrSvc.cmd
• edit installNodeMgrSvc.cmd to bind to hostname for remote starts
• set NODEMGR_HOST=jbayer-us
• run D:\Oracle\wls11g\wlserver_10.3\server\bin\installNodeMgrSvc.cmd
• start the service – mine is called “Oracle WebLogic NodeManager…” but on older versions it’s likely will start with “BEA …”
• starting the NodeManager process should create the nodemanager.properties file we will edit next

CrashRecovery and StartupScriptEnabled (updated 01/10/2010)

• We want to Enable NodeManager to restore servers to their last known state after a reboot.
• Open nodemanager.properties from the directory D:\Oracle\wls11g\wlserver_10.3\common\nodemanager\
• Change the CrashRecoverEnabled property from false to true:
• CrashRecoveryEnabled=true
• Also, because the domain’s \bin\startWebLogic script already has classpath configured, which is especially helpful for use with SmartUpdate that manages WebLogic patches as those classpath’s are not trivial to understand, I strongly recommend setting StartupScriptEnabled=true in nodemanager.properties
• stop/start node manager after making changes to nodemanager.properties

Set up the Machines

• Go to the domain's AdminServer console
• http://localhost:7001/console
• If you need to start it, run <your_domain_dir>\bin\startWebLogic.cmd
• With the console navigate to <domain_name> -> Environment -> Machines
• Create a new machine – I named mine after the hostname of the server - jbayer-us
• If you have more than one OS instance, create one machine for each.

• In console go to <domain_name> -> Environment -> Servers
• Create a new server, example: managedServer1
• Put in the listen address to the hostname you want to bind to, in my case jbayer-us
• Change the port to 8001or another available port so as not to conflict with the 7001 port of the AdminServer
• Assign managedServer1 it to the jbayer-us machine you just created - <domain_name> -> Enviornment -> Servers -> managedServer1
• There should be a dropdown you can use to select the machine
• Click the save button on this settings page (don't forget this step)
• Now activate changes if required

We need to assign the machine for the AdminServer, this cannot be done while the AdminServer is running.

• Stop AdminServer
• Open the domain config, example - D:\Oracle\wls11g\user_projects\domains\my_domain\config\config.xml
• Find the element used to assign the machine for managedServer1
• <machine>jbayer-us</machine>
• Copy that and paste that element right after the AdminServer <name/> element
• If it is not already set, also specify the AdminServer listen address to be the hostname to bind to, by default it is blank which means bind to all network interfaces.
• After these changes my config.xml snippet for the AdminServer looks like this:

<server>
<name>AdminServer</name>
<machine>jbayer-us</machine>
<listen-address>jbayer-us</listen-address>
</server>

• Start the AdminServer again with the domain directory’s bin\startWebLogic.cmd
• Check the console page <domain_name> -> Environment -> Servers to see that the servers are defined correctly as expected

Enroll the domain with Node Manager (updated 01/10/2010)  

• Navigate with a command prompt go to the domain directory
• run bin\setDomainEnv.cmd
• run java weblogic.WLST
• connect to the AdminServer with your credentials:
• connect('weblogic','welcome1','t3://jbayer-us:7001')
• Make sure you use forward slashes instead of backslashes in Windows:
• nmEnroll(domainDir='D:/Oracle/wls11g/user_projects/domains/my_domain',nmHome='D:/Oracle/wls11g/wlserver_10.3/common/nodemanager')
• Now go back to the web console - http://localhost:7001, for each server in the environment, go to the server start tab and put in the appropriate values.
• If you are using StartScriptEnabled=true in nodemanager.properties, then the classpath and jvm arguments from the script will be used first and the values you specify on the Server Startup tab will get added to the end, so you only need to enter values that are unique to each server in the classpath and arguments section.
• If you’re using Sun JDK instead of JRockit, then use –Xrs instead of –Xnohup refer to the official docs for more on this, but it has to do with handling OS signals properly.

Click the image to enlarge – actual text values below:

Java Home

D:\Oracle\wls11g\jrockit_160_14_R27.6.5-32

Java Vendor

Oracle

BEA Home

D:\Oracle\wls11g

Root Directory

D:\Oracle\wls11g\user_projects\domains\my_domain

Classpath is empty as the script values are sufficent

Arguments

-Xnohup

Security Policy File

D:\Oracle\wls11g\wlserver_10.3\server\lib\weblogic.policy

User Name

weblogic

Password

welcome1

Confirm Password

welcome1

• Save
• Now do the same thing for each managed server.
• Activate the changes if required

Node Manager Domain Username and Password

• Set the Node Manager username/password for the domain in the console
• Click <domain_name> in the console navigator
• Select Security tab
• Expand to the "Advanced" options about half-way down the page
• Choose a Node Manager username/password and put it in the username/password/confirm boxes your credentials, these could be unique from your user used to start the AdminServer, but I chose to keep mine the same weblogic/welcome1

Decrease Log Verbosity For Standard Out

• In console, go to each server's Logging -> General page
• Expand to the Advanced section half-way down the page
• Change the standard out notice level to "Critical"
• This log file that captures Standard Out does not roll over while a server is running, so it's really important to make sure this file doesn't get too large. Note that this is not the server log file, this is only standard out.

Have NodeManager Start and Stop the Servers

• If it is running, shutdown the AdminServer
• Start the AdminServer with NodeManager
• Go to domain dir
• run bin\setDomainEnv.cmd
• run java weblogic.WLST
• nmConnect(domainName='my_domain')
• nmStart('AdminServer')
• Go to console -> Environment -> Servers control tab and start the managed server from the console. It will send a command to Node Manager which actually performs the operation on the console's behalf.
• Gotcha! – If you have never started the managed server ever before and you try to start it from Node Manager, you might get an error.  In the server log it mjght say something like: Booting as admin server, but servername, managedServer1, does not match the admin server name, AdminServer  To get around this, simply start the Managed Server for the first time using either the Admin Console Servers->Control tab or the startManagedWebLogic.cmd script.  Subsequent nmStart commands should not have this issue any longer.

Test Killing and Restarting

If the servers are started successfully by Node Manager, try and kill a process from task manager and see if it restarts.

Looking at the Node Manager directory in the domain for each Managed Server, you will be able to see the state - below is a screenshot of managedServer1's files. The PID and .state files should tell you what you need to know to see if the server recovered.  If it did not, the .out file in the server log directory should hopefully give you a clue why.

If manual killing of the process restarts a Server then you are ready to test an operating system reboot. Reboot the machine without stopping the WebLogic Servers. Each server should be restored to the state that it was in when the OS was rebooted.

If you have questions, post them in the WLS General OTN forum as I’m not an expert on Node Manager – just someone who wanted have a quick start guide.  Cheers, James

Are you aware of the different tools available in the JVM you use to help troubleshoot and look under the hood?  Over the past couple of weeks several of my customers have been involved in several WebLogic support escalations with different JVM vendors.  It was an opportunity for me to learn a about the different tools – some unique to each JVM - and I want to document what I’ve found thus far.  I am by no means an expert in this area, but based on what I’ve seen there is not a high-level of awareness about the tools freely available.  I think you’ll find that there is a lot of innovation in JRockit Mission Control, and VisualVM is also a promising tool that you should get familiar with. 

*Updated 12/31/2009* Brian Peacock from IBM’s JVM team pointed me towards IBM’s JDK’s newer tools that look to all be available as plug-ins from the IBM Support Assistant and I was not aware of all of them during my original post.  Especially if you’re using the IBM JVM, make sure you’re aware of these powerful tools as they look to represent a big improvement over the older IBM JDK tools.

I purposely am not covering 3rd party commercial products as I am not familiar with them and the list is long.

*Updated 1/12/2010* Check out this nice OTN article covering both VisualVM and JRockit in more detail with screenhots. http://www.oracle.com/technology/pub/articles/heimburger-tuning.html

Dip Your Toe In The Water With JConsole or VisualVM

Sun has a detailed article about common symptoms and JVM tools to use for each, which I highly recommend.  But if the JVM is mostly a black-box to you, then trying to analyze some of the low-level detail that comes out of the JVM tools can be very intimidating at first and you can start by keeping it simple.  Two important questions to ask for JVM health are:

1) Is the JVM on a sustainable path?

2) What is the JVM doing?

The easiest way to get a quick answer to these questions that works across all JVM vendors and modern releases is JConsole because it ships with the JDK, so you know it’s there without a separate download.  The JDK documentation shows you how to use it on either a local or remote JVM.  Starting with JDK 1.6 update 7 there is a better tool called visualvm that ships with the JDK, but it can also be downloaded and used (with JDK 1.6) to monitor 1.4.2 and later JDKs.  I’m going to show visualvm because it’s simply newer and better.

VisualVM Monitor Tab

This snapshot is a stable and sustainable JVM (WebLogic Server on a JRockitJDK) where trend lines are relatively flat.  The memory utilization is low and returns to normal levels after each Garbage Collection.  The threads are not constantly increasing, and the CPU utilization is very low.  If you saw either the memory or threads going up constantly or saw extremely high CPU utilization, it would be worth investigating further.  Trend lines that slope up and to the right are signs of an upcoming problem.

Visual VM Threads Tab

The Threads tab shows a color coded timeline for all the threads by state and also has sub-tabs with views to sort by table columns and show extra detail.  You can also generate a full thread dump from this tab, which will open the current execution stack for each thread in the JVM in a new tab.  This reminds me a lot of the Mission Control Latency Analyzer.

More on WebLogic Threads

Support often asks for several thread dumps taken at 10-15 second intervals when Service Requests are filed.  By looking at the current point of execution of each thread over a period of time, you can identify whether some threads are stuck on something that should be fast, identify resource contention, etc.

If you’re using WebLogic Server, you do not necessarily have to worry about how each JVM has different ways to create a thread dump and potentially even different output formats.  Simply use the WLST threadDump() command regardless of which JVM or platform you are on and it should work the same to produce text output.  Max actually has a good example of how to do this on a periodic basis.

The 2009 book Professional Oracle WebLogic Server has excellent information about how to read WLS thread dumps starting on page 686.  But for those of you that want Cliff Notes, STUCK threads are bad.  A thread that has the name STUCK in it’s name has been identified as not being returned to the thread pool for a configured amount of time, which defaults to 10 minutes.

Digging Deeper with Specialized JDK Tools

Thread Dump Analysis

If you want to use the JVM tools that help decipher thread dumps, then you may need another technique as the format of the thread dump varies depending on the vendor.  These can help you find lock contention, bogged down resources, and other nasty problems.

Oracle JRockit – There is not a dedicated tool for Thread Dumps in JRockit as the format is human readable.  The JRockit documentation has a section describing thread dump analysis.  JRockit Mission Control also has some thread analysis both the live Console tool and the JRockit Runtime Analyzer recordings files.  For an entire mapping of Sun JDK tools to JRockit tools, look at this page in the docs.

Sun Hotspot – the jdk\bin\jstack command produces output suitable for Thread Dump Analyzer, a project on java.net that has many useful features for analyzing multiple thread dumps.

IBM J9 – The IBM JDK does not use a human readable format for it’s core files, so it’s especially important to know about the IBM Thread and Monitor Dump Analyzer for Java hosted on AlphaWorks.  The author published an article in JDJ covering how it works.  It’s a nice tool.

Heap Analysis

Oracle JRockit Mission Control has both the Memory Leak Detector and the JRockit Runtime Analyzer that excellent and easy to use visual tools.  On the command line, JRCMD also has a heap_diagnostics command that produces a human readable text file.  Paul Done has written about it recently.

Sun Hotspot – At the lowest level, there is a JVM tool called Java Heap Analysis Tool or jhat that ships with the JDK.  Additionally VisualVM also includes heap dump analysis features.  Eclipse Memory Analysis Tool (MAT) also supports both Sun and IBM JVM binary heaps as has a really nice report for finding Leak Suspects that doesn’t require advanced JVM knowledge to use.

IBM J9 – There are several tools available including the IBM Support Assistant (ISA) Plugins, the original is the Memory Dump Diagnostic for Java.  There are several beta plugins for ISA that offer more functionality.  Eclipse MAT now supports the IBM JDK also.  The oldest tool to my knowledge is the AlphaWorks project Heap Analyzer.

Garbage Collection Analysis

GCViewer claims some support for all 3 of the vendors verbose logging formats.  It is not clear whether all new JDK releases are forward compatible.

Oracle JRockit – Instead of the Sun -verbose:gc option, JRockit has an -Xverbose:memory command.  Additionally, the JRockit Runtime Analyzer files have detailed GC information presented in an easy to understand way.

Sun Hotspot - -verbose:gc on the command line and a visual tool named appropriately visualgc.

IBM J9 – AlphaWorks has a tool called IBM Pattern Modeling and Analysis Tool for Java Garbage Collector that parses the IBM JDK verbose gc output and makes very nice graphs.

Summary

Overall I really like JRockit Mission Control and use it most often.  It ships with the JRockit JDK so there is no separate download.  It integrates with Eclipse (including OEPE) and links me directly to the line of code in JDT.  It has a helpful network auto discovery protocol built-in that don’t require me to install remote daemons on other machines to find remote JVMs.  Perhaps most importantly to me, I have the most experience with it, find it the easiest to use and most consolidated.  Truthfully, I haven’t given VisualVM enough screen-time yet for a fair comparison and there is an Eclipse launcher for it that I have yet to try.  I also have to give IBM credit for really having a nice vision with the IBM Support Assistant even though I don’t think it’s capabilities or user experience quite match what Oracle and Sun have yet strictly from a JDK tooling perspective.  I’m also not sure it will ever be shipped with the IBM JDK.  IBM ISA is intended for a broader-than-java product support perspective and has a larger scope than just the JDK.  If you have any other JDK vendor tools that I should have covered but omitted please let me know.

In my last post about including static resources in shared libraries, I had a follow-up question about how to include multiple jar files in a library.  I alluded to it in my post, but here is a concrete example that should make it more clear.

Suppose I have 2 jars, that need to be deployed together, and I want to do that in a single shared library that my new web application will reference.

Foo.java – packaged in foo.jar

   1:  package com.oracle.otn.samples.wls;

   2:   

   3:  public class Foo 

   4:  {

   5:      public String toString()

   6:      {

   7:          return "Foo";

   8:      }

   9:  }

Bar.java – packaged in bar.jar – notice that Bar extends Foo, so unless both jars are available at runtime, we should expect to see an error.

   1:  package com.oracle.otn.samples.wls;

   2:   

   3:  public class Bar extends Foo {

   4:      

   5:      public String toString()

   6:      {

   7:          return super.toString() + "Bar";

   8:      }

   9:  }

To build a shared library for these, I simply construct a new packaging structure for this library that includes both foo.jar and bar.jar in the WEB-INF/lib of my shared library.  Of course I also need to configure a MANIFEST.MF.

Here is the file structure of my shared library I call manyjars.war.  The jars is in this library’s WEB-INF/lib directory will be added to the WEB-INF/lib of any application that references this library when the application is initialized.

   1:  d:\temp>jar -tf manyjars.war

   2:  META-INF/

   3:  META-INF/MANIFEST.MF

   4:  WEB-INF/

   5:  WEB-INF/lib/

   6:  WEB-INF/lib/bar.jar

   7:  WEB-INF/lib/foo.jar

Here is the MANIFEST.MF file:

   1:  Manifest-Version: 1.0

   2:  Specification-Title: ManyJars

   3:  Specification-Version: 1.0

   4:  Implementation-Title: ManyJars Implementation

   5:  Implementation-Version: 1.0

   6:  Implementation-Vendor: Oracle

   7:  Extension-Name: ManyJars

I can now deploy this shared library to WebLogic Server (or Oracle Enterprise Pack for Eclipse – OEPE - will do it for you if you configure the library there and refer to it in your application).  Here is how it looks in the console once it’s deployed.

In my new web application, I refer to the library in my weblogic.xml – again OEPE for insert this reference for you if you use the tooling:

   1:  <?xml version="1.0" encoding="UTF-8"?>

   2:  <wls:weblogic-web-app xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd">

   3:      <wls:weblogic-version>10.3.2</wls:weblogic-version>

   4:      <wls:context-root>FooBarWeb</wls:context-root>

   5:      <wls:library-ref>

   6:          <wls:library-name>ManyJars</wls:library-name>

   7:          <wls:specification-version>1.0</wls:specification-version>

   8:          <wls:exact-match>true</wls:exact-match>

   9:      </wls:library-ref>

  10:  </wls:weblogic-web-app>

So now if I have a simple index.jsp that refers to the Bar class we showed earlier, it should all work:

If I wanted to include these jars at the EAR level, the procedure would be similar, except that I would create an ear-based shared library packaging and instead of WEB-INF/lib, I would place the jars in APP-INF/lib.  Instead of referring to the shared library in weblogic.xml, I would use weblogic-application.xml.

You can download my files here if you want to try this.  Good luck.

In the OTN WLS General forum someone asked a question regarding using Shared Libraries to share static web resources among applications.  “Basically, we are trying to load jquery, css files, shared images, and what not into a shared library for access from other Web Applications.”  Shared libraries will handle this use case well.  I’ve put together a basic example using a simple image that is put into a shared library that is referenced by another web application.  The image in the screen shot below is from a shared library.  The web application that references it simply contains the JSP and a reference to the shared library in the weblogic.xml deployment descriptor.

Basic Steps

1. Assemble the resources (images, css, etc) in a base directory, with subdirectories if desired
2. In the main directory, create a META-INF/MANIFEST.MF file describing the library
3. Package the base directory as a WAR file
4. Deploy the WAR as a library to WLS
5. Refer to the Shared Library in a web application’s WEB-INF/weblogic.xml
6. Refer to the static resources from the library in the web application as if they are local to your application

Here is the link to the full documentation that describes all of the options for Shared Libraries in WebLogic.

Simple Example

I recommend taking a look at the shared libraries that ship with WebLogic Server to use as an example.  In WLS 10.3.x they are located here: <MIDDLEWARE_HOME>\wlserver_10.3\common\deployable-libraries

Here is what my META-INF/MANIFEST.MF file looks like for my example, I simply copied an existing MANFIEST.MF from jsf-1.2.war and made edits:

Manifest-Version: 1.0
Specification-Title: Images
Specification-Version: 1.0
Implementation-Title: Images Implementation
Implementation-Version: 1.0
Implementation-Vendor: Oracle
Extension-Name: images

Here is what the packaging of the library looks like after zipping it up as a WAR file:

C:\temp\images>jar -tf images.war
META-INF/
META-INF/MANIFEST.MF
oralogo_small.gif

As you can see, it’s only a simple WAR file with an oracle image in the base directory with a META-INF/MANIFEST.MF file describing the library.  To deploy this to the WLS, you can do that as normal other than noting that it is a library.  After doing that it will show up like this in the console:

If you’re using Oracle Enterprise Pack for Eclipse and set your Windows->Preferences->Server->Runtime Environments correctly, then the existing shared libraries should be available in Preferences->WebLogic->Shared Libraries and you can add the library you just created to the list.  This will allow you to edit the WEB-INF/weblogic.xml in the IDE with a dialog box, correctly specify the reference to the shared library, and deploy the shared library for you to the server if it has not been done already when you deploy the application.

weblogic.xml

<?xml version="1.0" encoding="UTF-8"?>
<wls:weblogic-web-app xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd">
    <wls:weblogic-version>10.3.2</wls:weblogic-version>
    <wls:context-root>ImagesWeb</wls:context-root>
    <wls:library-ref>
        <wls:library-name>images</wls:library-name>
        <wls:specification-version>1.0</wls:specification-version>
        <wls:exact-match>true</wls:exact-match>
    </wls:library-ref>
</wls:weblogic-web-app>

Now in the index.jsp, I can simply refer to the image as if it were in my base directory of the application like this:

<img src="oralogo_small.gif"/>

Conclusion

Shared libraries are usually used to share code such as jars and class files in WARs and EARs, but they can also be used to share static resources.  So let’s say I wanted to share a jar file, I would have a WEB-INF/lib directory in my shared library, and put any jars that I wanted my application’s to have in there.  If it is an EAR file, then I would use APP-INF/lib.

You can download this extremely simple example here.  I tested this with WLS 10.3.2 also known as 11g PS1.

Last week I wrote about my experiences playing with the Coherence Inbator’s Push Replication Pattern and showed some screenshots of the JMX instrumentation with JConsole and my post is already out of date.  This week there has been a new release of the Coherence Incubator projects.   Another new development is that you now learn directly from the lead of the Coherence Incubator projects as InfoQ just posted a video presentation from Brian Oliver discussing several of the projects and the rationale behind them.  One announcement I’m particularly fond of is the new Examples project.  This should help jump-start those getting started with the Coherence Incubator as in prior releases the examples were in separate places and not as complete.  Cheers to the Incubator team for continuing to improve and share these patterns.

Lately I've been able to do some Coherence work with some local customers and play with the Coherence Incubator projects.  This entry will showcase one of the examples for Push Replication for sharing data among separate Coherence clusters.

Inter-cluster Data Replication

Coherence clustering technology makes a lot of sense for customers to be able to scale applications horizontally and reliably with very fast predictable performance.  This is of course easy to do within one data center on a fast network.  One of the common challenges that customers have is around High Availability and Disaster Recovery and keeping data synchronized across data centers.  Normally Coherence is optimized to use UDP unicast or multicast, but what happens if the network is unreliable and/or has high latency which is common when networking multiple data centers?  The answer is Coherence TCP Extend, which addresses these challenges by using TCP and the result is that multiple Coherence clusters can communicate together.

Multiple data centers are not the only reason you need to keep multiple clusters in sync.  I recently came across a use case where it made sense to have separate Coherence clusters on different physical machines.  The JVM processes for each machine have a single digit millisecond SLA for round trip time per request, so they are very sensitive to any outside events and it was a requirement to isolate the processes as much as possible.  In this scenario one physical machine was Active and one was Passive for fail-over.  During load-testing on the Active machine we found that bringing up cluster members on the Passive machine in the middle of the test impacted performance when both machines were part of the same cluster.  Using a single Coherence cluster per machine ensures that the impact of cluster membership events are isolated to that machine only.

Leveraging the Coherence Incubator Push Replication pattern, multiple Coherence clusters can keep data in sync whether they are on the same network subnet with low-latency or hundreds of miles apart in separate data centers.  Let's take a look at the simplest example, which is Active - Passive scenario.  In this situation we'll use Push Replication to make sure that Cache Entry operations (insert/update/delete) in the Active cache are replicated in the Passive cache.

Run the Example

To run this example called ActivePassiveExample - which is included in the src distribution of the Push Replication Pattern, I used the following software:

• Coherence 3.5.2 for Java,
• Coherence Common 1.4.0
• Command Pattern 2.4.0
• Messaging Pattern 2.4.0
• Push Replication Pattern 2.4.0
• Apache Ant 1.7.1
• JMX RI 1.2.1 - this is optional, simply do not issue the "-http" on either of the jmx ant targets which will prevent the JMX HTTP Adapter from binding
• My build.xml and other example code

The Ant Approach

One of my colleagues Randy Stafford introduced me using an ant build.xml file to organize different types of Coherence cluster processes.  I found it to be a much simpler and cleaner approach than what I had been doing (multiple shell scripts or multiple Eclipse launch configurations).  This way it is simple to centralize the shared configuration to a set of properties that are used by all of the ant targets and each process can easily override one of the properties with a -DpropertyName=propertyValue at the command line.  Take a look at the build.xml file and let me know what you think.

Step by Step

• Set the build.properties paths according to your environment
• Set up shells with ANT_HOME and JAVA_HOME and the PATH, setAntEnv.txt is an example
• Run ant from those shells with the appropriate targets as described in build.xml.  Minimum for this example:
  ant compile (you can reuse this shell)
  ant run_active_cache_server
  ant run_passive_cache_server
  ant run_active_publisher
• Most likely you will also want to run the JMX and Console processes to see the values.
  ant run_active_jmx
  ant run_passive_jmx
  ant run_passive_console

Guided Tour

Once the JMX processes are running for Active and Passive, you can use jconsole to locally bind to the two MBeanConnector process.

To verify that the passive cache received all the updates you can execute the run_passive_console target and execute two commands to see the contents.

• cache passive-cache
• list

Notice that at the bottom of my output that the entry values are the last ten number leading up to 10000, which is what we expect looking at the sample publishing code.

     [java]   ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
     [java]   InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
     [java]   )
cache passive-cache
     [java] 2009-10-22 17:07:31.797/10.344 Oracle Coherence GE 3.5.1/461p2 <Info> (thread=Main Thread, member=3): Loaded cache configuration from "jar:file:/C:/Oracle/coherence-v3.5.1b461/coherence/lib/coherence.jar!/coherence-cache-config.
xml"
     [java] 2009-10-22 17:07:32.156/10.703 Oracle Coherence GE 3.5.1/461p2 <D5> (thread=DistributedCache, member=3): Service DistributedCache joined the cluster with senior service member 1
     [java] 2009-10-22 17:07:32.172/10.719 Oracle Coherence GE 3.5.1/461p2 <D5> (thread=DistributedCache, member=3): Service DistributedCache: received ServiceConfigSync containing 259 entries
     [java] <distributed-scheme>
     [java]   <scheme-name>example-distributed</scheme-name>
     [java]   <service-name>DistributedCache</service-name>
     [java]   <backing-map-scheme>
     [java]     <local-scheme>
     [java]       <scheme-ref>example-binary-backing-map</scheme-ref>
     [java]     </local-scheme>
     [java]   </backing-map-scheme>
     [java]   <autostart>true</autostart>
     [java] </distributed-scheme>
     [java] Map (?):
list
     [java] 2 = 9992
     [java] 3 = 9993
     [java] 1 = 9991
     [java] 0 = 10000
     [java] 5 = 9995
     [java] 4 = 9994
     [java] 6 = 9996
     [java] 8 = 9998
     [java] 9 = 9999
     [java] 7 = 9997
     [java] Map (passive-cache):

Let's look at the Active cluster processes.  You can see that there are caches defined for many of the Incubator patterns.  Below you can see that the publishing-active cache should have a size of 10 after running the run_active_publisher ant target.

Consider the role of some of the other caches:
DistributedCacheForCommandPattern - stores command contexts and co-located commands

DistributedCacheForMessages - stores messages (in this case - entries that need to be pushed to the other cluster)

DistributedCacheForSubscriptions - maintains durable subscribers for the publishers

You can learn more about these caches by looking at the Coherence Incubator Wiki and reviewing the cache-config xml files that correspond to the Incubator project they are named for.

Looking at the MBean for the PublishingService, you can find operations to suspend, resume, and drain messages that have yet to be published.

So if you suspend() the PublishingService, any changes to the publishing-active cache will be queued up as messages in the DistributedCacheForMessages coherence.messagingpattern.messages cache.  You can try this by executing the suspend() operation then the run_active_publisher target again.  You should then see the messaging cache full of the operations waiting to be replicated to the passive-cache.

Now if you execute the resume() operation on the PublisherService, the messages will replicate to the passive-cache.  Alternatively you could execute the drain() operation and all of the messages waiting to be replicated would be deleted and the passive-cache would never receive those updates.

On the Passive cluster, you can see the TCP Extend working by looking at the ConnectionManager and the Connection MBeans.  The passive process is listening in this case on port 20000 and has 1 active connection.

Conclusion

This should give you a quick idea of the capabilities of the Push Replication Pattern and some of the JMX enabled capabilities in the Coherence Incubator projects.  This is the most basic example and there are also samples for Hub-and-Spoke, Active-Active, and Federated patterns which are more complex.

I had some people contact me about my previous post on deployment plans that still had questions about a more complex example of using WebLogic Server deployment API support for JSR-88 Deployment Plans.  Steve Button – one of the WebLogic Server Product Managers -  provided a very helpful presentation on Deployment Plans that I will embed below. 

I have also posted a Deployment Plan Sample on http://samplecode.oracle.com.  The sample illustrates how to change a web.xml context parameter value from “development” to “qualityassurance” to “production” all without modifying the EAR file at all.  The presentation and sample should help you become more familiar with the capabilities.

UPDATED 9/9/2009 – Also there is a complete Oracle By Example – showing how to create and use a Deployment Plan step-by-step.