November 10, 2009

Developing custom MBeans to manage J2EE Applications (Part II)

This is the second part in a series of blogs, that demonstrate how to add management capability to your own application using JMX MBeans.

In Part I we did the bulk of the work. We saw:

  • How to implement a custom MBean to manage configuration associated with an application.
  • How to package the resulting code and configuration as part of the application's ear file.
  • How to register MBeans upon application startup, and unregistered them upon application stop (or undeployment).
  • How to use generic JMX clients such as JConsole to browse and edit our application's MBean.

In this second part, we will add descriptions to our:

  • MBean
  • MBean attributes
  • MBean operations
  • MBean operation parameters

We saw using JConsole, that default descriptions were generated for all of the above. However those default descriptions didn't provide any meaningful information to our MBean users. We will replace those default descriptions with custom descriptions that can be used by us human to understand the functionality provided by our MBean.

We will also add localization support to our MBean. Our goal is to ensure that our MBean's meta-data is localized based on the WebLogic Server it is deployed to. So if that server uses a French Locale, then we'd expect our MBeans descriptions to be French and not English.

The complete code sample and associated build files for part II are available as a zip file. The code has been tested against WebLogic Server 10.3.1 and JDK6. To build and deploy our sample application, please follow the instruction provided in Part I, as they also apply to part II's code and associated zip file.

Providing custom descriptions

In order to provide custom description we need to modify our MBean implementation to extend the StandardMBean class. Despite its name that class is used to implement both Standard and MXBeans.

We override the StandardMBean class many getDescription methods to provide our own custom descriptions. We use resources bundles to store our descriptions to ensure that those can be properly localized. The updated code for our MBean implementation is included below:

package blog.wls.jmx.appmbean;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.File;

import java.net.URL;

import java.util.Map;
import java.util.HashMap;
import java.util.Properties;
import java.util.PropertyResourceBundle;
import java.util.ResourceBundle;

import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.management.MBeanRegistration;
import javax.management.StandardMBean;
import javax.management.MBeanInfo;
import javax.management.MBeanAttributeInfo;
import javax.management.MBeanOperationInfo;
import javax.management.MBeanParameterInfo;

public class PropertyConfig extends StandardMBean implements
    PropertyConfigMXBean, MBeanRegistration {

    private String relativePath_ = null; 

    private Properties props_ = null;

    private File resource_ = null;

    private ResourceBundle resourceBundle_ = null;

    private static Map operationsParamNames_ = null;

    static {
        operationsParamNames_ = new HashMap();
        operationsParamNames_.put("setProperty", new String[] {"key", "value"});
        operationsParamNames_.put("getProperty", new String[] {"key"});
    }

    public PropertyConfig(String relativePath) throws Exception {
        super(PropertyConfigMXBean.class , true);
        props_ = new Properties();
        relativePath_ = relativePath;
    }

    public String setProperty(String key,
                              String value) throws IOException {

        String oldValue = null;

        if (value == null) {
            oldValue = String.class.cast(props_.remove(key));
        } else {
            oldValue = String.class.cast(props_.setProperty(key, value));      
        }

        save();
        return oldValue;
    }

    public String getProperty(String key) {
        return props_.getProperty(key);
    }

    public Map getProperties() {
        return (Map) props_;
    }

    private void load() throws IOException {
        
        InputStream is = new FileInputStream(resource_);
        try {
            props_.load(is);
        }
        finally {
            is.close();
        }
    } 

    private void save() throws IOException {
  
        OutputStream os = new FileOutputStream(resource_);

        try {
            props_.store(os, null);
        }
        finally {
            os.close();
        }
    }

    public ObjectName preRegister(MBeanServer server, ObjectName name)
        throws Exception {
       
        // MBean must be registered from an application thread
        // to have access to the application ClassLoader
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        URL resourceUrl = cl.getResource(relativePath_);
        resource_ = new File(resourceUrl.toURI());

        load();     

        return name;
    }

    public void postRegister(Boolean registrationDone) { }

    public void preDeregister() throws Exception {}

    public void postDeregister() {}

    private synchronized ResourceBundle getResourceBundle() {
   
        if ( resourceBundle_ == null ) {
            resourceBundle_ = 
                PropertyResourceBundle.getBundle(
                    "blog.wls.jmx.appmbean.MBeanDescriptions");
        }
       
        return resourceBundle_;
    } 

    protected String getDescription(MBeanAttributeInfo info) { 
        return getResourceBundle().getString("PropertyConfigMXBean.attribute." + 
            info.getName() ); 
    } 

    protected String getDescription(MBeanOperationInfo info) { 
        return getResourceBundle().getString("PropertyConfigMXBean.operation." + 
            info.getName() ); 
    }   

    protected String getDescription(MBeanInfo info) {
        return getResourceBundle().getString("PropertyConfigMXBean.mbean");
    }

    protected String getDescription(MBeanOperationInfo op,
                                    MBeanParameterInfo param,
                                    int sequence) {
        return getResourceBundle().getString(
            "PropertyConfigMXBean.operation." + 
            op.getName() + "." + 
            operationsParamNames_.get(op.getName())[sequence] ); 
    }

    protected String getParameterName(MBeanOperationInfo op,
                                      MBeanParameterInfo param,
                                      int sequence) {
        return operationsParamNames_.get(op.getName())[sequence];
    } 

}

Our default resource bundle class, that contains our English descriptions contains the following code:

package blog.wls.jmx.appmbean;

import java.util.ListResourceBundle;

public class MBeanDescriptions extends ListResourceBundle {
     protected Object[][] getContents() {
         return new Object[][] {
             {"PropertyConfigMXBean.mbean", 
              "MBean used to manage persistent application properties"},  
             {"PropertyConfigMXBean.attribute.Properties", 
              "Properties associated with the running application"},
             {"PropertyConfigMXBean.operation.setProperty", 
              "Create a new property, or change the value of an existing property"},
             {"PropertyConfigMXBean.operation.setProperty.key", 
              "Name that identify the property to set."},
             {"PropertyConfigMXBean.operation.setProperty.value", 
              "Value for the property being set"},
             {"PropertyConfigMXBean.operation.getProperty", 
              "Get the value for an existing property"}, 
             {"PropertyConfigMXBean.operation.getProperty.key", 
              "Name that identify the property to be retrieved"} 
        };
     }
 }

Our MBean is quite simple, and only exposes one attribute and two operations. This helps keep our resource bundle class quite small. For real world example that file will be much bigger. Note: We didn't override the getDescription method associated with our MBean constructor, to keep our sample small as this doesn't add much value to our discussion.

To add support for other languages, we only need to implement the corresponding resource bundle class. MBeanDescriptions_fr for French, and translate the description appropriately.

One interesting thing to note, is the name we used for our resource bundle keys. We didn't use arbitrary values, but we made sure we followed the following convention:

  • MBean description: <MBeanInterfaceClass>.mbean
  • MBean attribute description: <MBeanInterfaceClass>.attribute.<AttributeName>
  • MBean operation description: <MBeanInterfaceClass>.operation.<OperationName>
  • MBean operation parameter description: <MBeanInterfaceClass>.operation.<OperationName>.<ParameterName>
  • MBean constructor description: <MBeanInterfaceClass>.constructor.<ConstructorName>
  • MBean constructor parameter description: <MBeanInterfaceClass>.constructor.<ConstructorName>.<ParameterName>
We also purposely named our resource bundle class MBeanDescriptions and included it as part of the same package as our MBean. The above convention is used by the JDK 7 to localize MBean descriptions without requiring us to extend the StandardMBean class and override its many getDescription methods. Unfortunately JDK 6 doesn't support built-in JMX localization, so we have to write the above code. However we can anticipate the JDK 7 functionality (and possible early support by WebLogic) by using the above convention when specifying our MBean resource bundle and associated resource keys.

You might have noticed the following code in our updated MBean implementatrion:

    protected String getParameterName(MBeanOperationInfo op,
                                      MBeanParameterInfo param,
                                      int sequence) {
        return operationsParamNames_.get(op.getName())[sequence];
    } 
This is used to provide customized name to our operation parameters in place of the default generated 'po', 'p1', ... , 'pn' values.

The result of our hard work can be seen in the following JConsole screen shot:

app_mbean_part2_jconsole1.JPG

Consult Part I for information on how to use JConsole to browse/edit our MBean.

What's next?

What if our application is deployed to a WebLogic server running with an English Locale, and our management client wants to use a French Locale. Currently as things stand the Localization is performed based on the server Locale, and not based on the client's Locale. In the last part of this blog series, we will see how to associate a Locale with our JMX client connection, and we will update our MBean code to support client based localization.

November 9, 2009

Stateful JAX-WS with Coherence*Web


We describe a stateful JAX-WS web service with high availability provided by Coherence*Web [1]. Deployed to two separate WebLogic managed servers, the web service stores its state in http session cookies. Coherence*Web distributes and synchronizes http session data between both managed servers. As a result, two endpoints with synchronized web service state provide high availability. With one endpoint down, another could continue to serve the service request with current web service state.

Note that WebLogic clustering can also provide highly available http sessions. However, we want to demonstrate Coherence*Web as an alternative to clustering for high availability. Therefore, we do not use clustering here. Instead, we use Coherence*Web.

Setting up the stateful web service with Coherence*Web involves the following steps.

  • start Coherence*Web cache server 
  • create and configure managed servers
    • deploy Coherence*Web as a shared library to the managed servers
  • create and deploy the web service
    • stores state in http session
    • instrument the web service using Coherence*Web
    • deploy the web service to both managed servers
Start Coherence*Web Cache Server

Start Coherence*Web cache server using the command below [2].

java -server -Xms512m -Xmx512m 
-cp coherence/lib/coherence.jar:coherence/lib/coherence-web-spi.war
-Dtangosol.coherence.management.remote=true
-Dtangosol.coherence.cacheconfig=WEB-INF/classes/session-cache-config.xml
-Dtangosol.coherence.session.localstorage=true
com.tangosol.net.DefaultCacheServer

Create and Configure Managed Servers

Copy coherence/lib/coherence.jar [2] to $DOMAIN_HOME/lib.

Start WebLogic AdminServer using

$DOMAIN_HOME/bin/startWebLogic.sh

Create two managed servers Server-2 at port 7012 and Server-3 at port 7013 using the admin console. Start both servers using the commands below.

$DOMAIN_HOME/bin/startManagedWebLogic.sh Server-2
$DOMAIN_HOME/bin/startManagedWebLogic.sh Server-3

Deploy coherence\lib\coherence-web-api.war [2] to the servers as a shared library. 

Create and Deploy the Web Service

Install Oracle Enterprise Pack for Eclipse (OEPE) on Eclipse Galileo [4]. In Eclipse, create a web service using the following WSDL and implementation.

<definitions
 xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
 xmlns:tns="http://oracle.ws.demo/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns="http://schemas.xmlsoap.org/wsdl/"
 targetNamespace="http://oracle.ws.demo/"
 name="StatefulWSService">
<types>
<xsd:schema>
<xsd:import namespace="http://oracle.ws.demo/"
 schemaLocation="http://localhost:7001/Stateful_WS_Coherence/StatefulWSService?xsd=1" />
</xsd:schema>
</types>
<message name="addItem">
<part name="parameters" element="tns:addItem" />
</message>
<message name="addItemResponse">
<part name="parameters" element="tns:addItemResponse" />
</message>
<portType name="StatefulWS">
<operation name="addItem">
<input message="tns:addItem" />
<output message="tns:addItemResponse" />
</operation>
</portType>
<binding name="StatefulWSPortBinding" type="tns:StatefulWS">
<soap:binding transport="http://schemas.xmlsoap.org/soap/http"
style="document" />
<operation name="addItem">
<soap:operation soapAction="" />
<input>
<soap:body use="literal" />
</input>
<output>
<soap:body use="literal" />
</output>
</operation>
</binding>
<service name="StatefulWSService">
<port name="StatefulWSPort" binding="tns:StatefulWSPortBinding">
<soap:address
 location="http://localhost:7001/Stateful_WS_Coherence/StatefulWSService" />
</port>
</service>
</definitions>

StatefulWS.java and Item.java

package demo.ws.oracle;
import javax.jws.WebMethod;
import javax.jws.WebService;
import javax.xml.ws.WebServiceContext;
import javax.xml.ws.*;
import javax.xml.ws.handler.MessageContext;
import javax.jws.*;
import javax.annotation.Resource;
import java.util.*;
import javax.servlet.*;
import javax.servlet.http.*
@WebService
public class StatefulWS {
@Resource
WebServiceContext ctx;
@WebMethod()
public int addItem(String name) {
HttpServletRequest req = (HttpServletRequest)
ctx.getMessageContext().get(MessageContext.SERVLET_REQUEST);
HttpSession session = req.getSession(true);
if (session == null)
throw new WebServiceException("No HTTP Session found");
System.out.println("httpsession: id="+session.getId());

//Get the cart object from the HttpSession
List<Item> cart = (List<Item>)session.getAttribute("myCart");
if (cart == null)
{ cart = new ArrayList(); }
// Add the item to the cart
cart.add(new Item(name));
// Save the updated cart in the HTTPSession
session.setAttribute("myCart", cart);
// return the number of items in the stateful cart
return cart.size();
}
}
package demo.ws.oracle;
public class Item implements java.io.Serializable {
public String name;public Item(String name){
this.name = name;
}
}

Add coherence-web-api library into weblogic.xml [2].

<weblogic-web-app
 xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app
 http://xmlns.oracle.com/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd"
>
<weblogic-version>10.3.1</weblogic-version>
<context-root>Stateful_WS_Coherence</context-root>
<library-ref>
<library-name>coherence-web-spi</library-name>
<specification-version>1.0.0.0</specification-version>
<implementation-version>1.0.0.0</implementation-version>
<exact-match>false</exact-match>
</library-ref>
</weblogic-web-app>

Deploy the web service to both managed servers.

Test 

Run the test.

package demo.ws;
import demo.ws.oracle.*;
import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import javax.xml.namespace.QName;
import javax.xml.ws.BindingProvider;

public class StatefulWSClient {
public static void main(String[] args) throws Exception {
StatefulWSService service =
new
StatefulWSService(Thread.currentThread().getContextClassLoader()
.getResource("META-INF/wsdls/StatefulWSService.wsdl")
 , new QName("http://oracle.ws.demo/", "StatefulWSService"));
StatefulWS port = service.getStatefulWSPort();

((BindingProvider)port).getRequestContext()
.put(BindingProvider.SESSION_MAINTAIN_PROPERTY, true);

((BindingProvider)port).getRequestContext()
.put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
"http://stacd15.us.oracle.com:7012/Stateful_WS_Coherence/StatefulWSService");

System.out.println("calling " +
((BindingProvider)port).getRequestContext()
.get(BindingProvider.ENDPOINT_ADDRESS_PROPERTY));
for (int j=0; j<5; j++) {
try {
int response = port.addItem("abc");
System.out.println("Got " + response);
Thread.currentThread().sleep(3000);
} catch (Exception ex) {}
}

((BindingProvider)port).getRequestContext()
.put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
"http://stacd15.us.oracle.com:7013/Stateful_WS_Coherence/StatefulWSService");
System.out.println("calling " +
((BindingProvider)port).getRequestContext()
.get(BindingProvider.ENDPOINT_ADDRESS_PROPERTY));
for (int j=0; j<5; j++) {
try {
int response = port.addItem("abc");
System.out.println("Got " + response);
Thread.currentThread().sleep(3000);
} catch (Exception ex) {}
}
}
}

The test invokes the web service five times for each endpoint.  The output shows that the returned count increases, indicating synchronized web service state across both endpoints.

calling http://stacd15.us.oracle.com:7012/Stateful_WS_Coherence/StatefulWSService
Got 1
Got 2
Got 3
Got 4
Got 5
calling http://stacd15.us.oracle.com:7013/Stateful_WS_Coherence/StatefulWSService
Got 6
Got 7
Got 8
Got 9
Got 10

Both managed servers output the same http session id's.

httpsession: id=tS1DNs2XTJZuQNhNfsl4fvpuj0Qd2ZnmTmHFKtKlUX4EytukR9V0!-280396885!1257829391923
httpsession: id=tS1DNs2XTJZuQNhNfsl4fvpuj0Qd2ZnmTmHFKtKlUX4EytukR9V0!-280396885!1257829391923
httpsession: id=tS1DNs2XTJZuQNhNfs14fvpuj0Qd2ZnmTmHFKtKlUX4EytukR9V0!-280396885!1257829391923
httpsession: id=tS1DNs2XTJZuQNhNfsl4fvpuj0Qd2ZnmTmHFKtKlUX4EytukR9V0!-280396885!1257829391923

Reference
  1. Coherence*Web and WebLogic Server
  2. Installing Coherence*Web on the WebLogic Server 10.3
  3. Stateful JAX-WS
  4. Oracle Enterprise Pack for Eclipse

Continue reading "Stateful JAX-WS with Coherence*Web " »

November 2, 2009

Understanding Custom Context propagation

What is a Custom Work Context ?

The idea behind out-of-band context propagation in Weblogic server was to allow application developers to send information from a client to an application as an extra payload on the underlying transport protocol. Typical transport protocols that are supported for context propagation include Threads, JMS queues, SOAP, RMI etc. The use cases of context propagation are applications that need to carry information outside the application rather than inside the application itself. Diagnostics seems like a classic example that can ride on workcontext to propagate monitoring information from client to server and vice versa. For a quick refresher to work context propagation, please refer to latest Weblogic documentation. A few important APIs in the workcontext framework are -

1. weblogic.workarea.WorkContextMap - The main context propagation interface to tie applications with data and propagate that information via application requests. WorkContextMap is part of the client/application's JNDI environment. The map can be accessed through JNDI lookup by the name "java:comp/WorkContextMap".

2. weblogic.workarea.WorkContext - Developers could use this interface to marshal and unmarshal the user data that is passed along with the application. With the interface came a set of four implementing classes that could marshall 8 bit Ascii, Long, String and Serializable contexts.

Applications that started replying on this framework created custom contexts either by extending WorkContext or Serializable interface. We found some interesting use cases as these custom contexts were developed and put in use. The following sections document some of these use cases.

Using the correct classloader to load a custom WorkContext

Initially, the custom contexts were bundled as a part of the server through the Weblogic System classpath. As a result, the Context classloader (Weblogic System ClassLoader) for the Weblogic server could resolve these custom classes while unmarshalling the Work Context object. This scheme fell short of the fact that an application may not choose to bundle its custom context as a part of the System Loader in Weblogic. A specific application may choose to bundle the custom context as a part of the application (EAR or WAR) rather than the System classpath approach. Hence, instead of using SystemLoader.loadClass(classname) to resolve the custom context, the Server started relying on the Thread Context classloader to load the custom context. The Thread Context classloader is set correctly to the application loader if the unmarshalling of the context had to happen in an application loader scope.

Inter-operating custom work contexts across different Weblogic server versions

As Weblogic server got integrated into the Oracle Fusion Middleware Framework (11g), the context propagation framework faced a challenging problem with inter-operating custom work contexts created in 11g release with older versions of the Weblogic server. Once a newly created custom context passed from a newer version of Weblogic server to an older version, the Weblogic server had to convert the byte array into a work context object while de-serializing the context in the older server. An older version of the server would not know about newer work context object types. This resulted in ClassNotFoundExceptions.

The solution to this problem was easy. The application provider had to wrap their custom objects in a SerializableWorkContext. In effect, the SerializableWorkContext wraps a Java Serializable object (for example a custom Work Context) such that on de-serializing the work context, the server does not hit the ClassNotFoundExceptions. When the work context provider is present in the newer version of the server, it would firstly get the SerializableWorkContext from the map and then get the wrapped serializable custom object. The custom context provider knows if it can handle the new context. Hence, the serialization of the custom object gets controlled by the application provider rather than the work context framework.

Here is a snippet of code that creates a Serializable work context and puts it in the work context map for the client .

1. WorkContext serializedCustomContext = PrimitiveContextFactory.create(Serializable customContext)

2. Put the serialized custom context inside the workcontext map -

WorkContextMap map =
WorkContextHelper.getWorkContextHelper().getWorkContextMap();

map.put(MYAPP.CUSTOM_CONTEXT,serializedCustomContext);

On the receiving end, the custom context would be retrieved as -

1. WorkContextMap map =
WorkContextHelper.getWorkContextHelper().getWorkContextMap();

2. (SerializableWorkContext)WorkContext customContext = (SerializableWorkContext)map.get(MYAPP.CUSTOM_CONTEXT);

3. MyContext myCtx = (MyContext)((SerializableWorkContext)customContext).get();

Unless the application provider turned around and calls step (3) or customContext.getSerializable(), the custom object would not be converted from a byte array to an object instance. Hence, the workcontext framework would simply unmarshal the wrapped serialized object containing the payload instead of de-serializing the custom object itself. This mechanism shifts the control of de-serializing to the application provider who now has more control on when to convert the byte array into the custom work context.

If the application provider on a specific Weblogic server version cannot de-serialize the custom context, it would pass the wrapped object to a different server who may be able to read it fully. The mechanism is limited to Serializable work context types. Serialized work contexts come with a performance overhead. Hence an application provider should carefully evaluate the pros and cons of using a Serialized work context type.

So far so good. Now came the side effect of when to serialize while sending a custom context from a client.

When to serialize the custom work context before sending it from the client

There is a hidden side-effect to SerializedWorkContexts other than performance. A serialized work context is serialized the moment it is created. For example, if the client wraps its custom context in a Serializable context -

WorkContext customContext = PrimitiveContextFactory.create(Serializable context);

The custom work context is serialized at the point of creation (the create call). This is often a drawback to the application provider who may not have all the information it needs at the creation time to stuff in the custom context. Once the context is created, its too late to change it. Hence we needed a better mechanism to delay the serialization of the custom context. This allows the Serializable context data to be updated even after it is put in the WorkContextMap. From 11g release, the framework provided a new API to create a mutable Serializable object that in effect delayed the serialization of the work context -

WorkContext customContext = PrimitiveContextFactory.createMutable(Serializable context);

The mutable work context vastly improved the time window when clients could change the contents of the work context. Diagnostic frameworks often use this feature since they do not have all the information they need in the work context when creating it in the first place. However, they do have all the required information just before the request is send over the wire from the client to the server. This feature gave developers the opportunity to change the workcontext repeatedly before it was put on the wire.

We haven't talked about the inter-operability case with mutable work contexts. The Weblogic server itself needs to distinguish from a mutable (or delayed serialized work context) from the plain serialized work context. Hence the mutable flag or property had to be carried with the new type of work context so that the Server could take an appropriate action when it came across a mutable or non mutable Serializable work context.

When is a Work Context is really available to the application ?

It would be desirable for a work context send from a client to a server to be available to the application the moment the request is intercepted. However, the availability of the workcontext in the application is dependent on the client server communication protocol. For example, when a Web Services request is send over HTTP, the SOAP request is not unmarshalled by the server unless the protocol specific interceptors (or handlers who know what to do with the payload) have read the whole SOAP message. The interceptors don't fire at the first point of request interception. If applications have J2EE filters or security related policy checks that are done before the SOAP request is unmarshalled, the goodies in the workcontext payload are hidden to these consumers unless the SOAP envelope is fully unmarshalled later. This is a known limitation with how work contexts interact with the different protocols.

Unless the request is completely unmarshalled, a call to the WorkContextMap to fetch an already known work context will return null -

WorkContext context =
WorkContextHelper.getWorkContextHelper().getWorkContextMap().get(MYAPP.MY_CONTEXT);

It is therefore incumbent on frameworks who ride on Weblogic work contexts not to access the map during this window (intercept request --> unmarshall request). Though this window is small, its possible some piece of application code would like get hold of the incoming work context at the very first request interception point. Without that, they may face the dilemma of having to reconcile a work context created during this window and the real work context that reveals itself later. Such reconciliations are tough and come with even more serious inter-operability side effects. Hence, its recommended that the workcontext is only accessed from the map after the request is fully unmarshalled.

In Conclusion

Custom work contexts (mutable or un-mutable) is a handy feature for applications who intend to send extra data outside the application scope. The Weblogic server tries to hide all the underlying gory details of marshalling and unmarshalling the context associated with various protocols from the end user. The consumer simply needs to remember the "key" or "id" to the stuffed work context in the incoming request and pull the work context out of the map. Used wisely, this feature can save man hours with applications providers who intend to drive data outside their scope. The workcontext framework has also evolved over time to learn and adapt with changing needs of the application provider. Once used primarily as an internal Weblogic server feature, workcontext is now full integrated and deployed in applications running on Oracle Fusion Middleware 11g framework.

October 30, 2009

SSL Troubleshooting and Debugging

SSL Troubleshooting and Debugging

Due to the very nature of secure channel establishment, it is often difficult to even approach troubleshooting and debugging SSL related issues.

This brief article intends to illustrate the challenges, approaches and tools available for debugging these difficult scenarios.

SSL Description

Secure Socket Layer (SSL) is a protocol for providing a secure channel of communication between two computers. It makes provisions for data integrity, confidentiality and authentication. Authentication of the server - by the client - provides an assurance of the fact that the traffic has not been diverted to an attacking server. Mutual authentication requires the client to provide credentials to the server over the secure channel.

SSL Handshake Overview

In order to really be able to troubleshoot and debug SSL related issues, we need an understanding of what the protocol actually does on both the client and server sides. This understanding will enable us to quickly categorize the type of problem being encountered and hopefully a category of approaches for tracking down the root cause.

SSL Diagrams.jpg

We will touch on issues and troubleshooting approaches in the following categories:

1. Certificate Validation
2. Trust
3. Configuration

So let's briefly describe the protocol with a bit of focus on these three categories.

The client initiates the SSL connection by requesting a channel through the use of a ClientHello handshake message. This message contains the Cipher Suites that are configured to be supported by the client side and are available for the server to choose in creating the most secure channel configuration possible between the two machines. It also contains a random number to be used by the server in the generation of keys - this random number is a result of the configured or default RNG on a given platform.

The server side, in turn, responds with a ServerHello that includes the Cipher Suite selected by the server as the most appropriately secure suite for the channel. If a suitable cipher suite could not be selected from the list of supported suites provided by the client - the request for an SSL connection is denied by the server. It also includes a random number and the certificate that is to be used for authenticating the server to the client. This certificate must be validated by the client in order for it to be trusted as representing the identity asserted by it.


This validation is based on a number of possible factors (driven by configuration):

1.Whether it is expired
2.Whether it has been revoked
3.Whether it was issued by a trusted Certificate Authority
4.Whether the server name within the certificate matches the host name for the current connection

Where Things Can Go Wrong

There are a number of common scenarios that occur as a result of improperly configured environments, clients, servers and certificates that can be categorized into one or more of the afore mentioned categories.

The following are a few descriptions of these scenarios and what the approach to identifying the root cause might be.

Keystores and Truststores
- Categories: Configuration, Trust, Certificate Validation

The client (for mutual authentication) and server each present the other a certificate that represents the identity of the machine its running on. In order for either to present this certificate - it must be available within the appropriate Keystore.

Tip 1: Determine the default certificate for a machine as appropriate for your server and ensure that it exists within the configured Keystore and is available to the process that needs to present it to the corresponding partner process.

Tip 2: Ensure that the issuer of the presented certificate exists within the appropriate Truststore of the recipient process.

Supported Cipher Suites
- Categories: Configuration

As described earlier, the handshake involves the selection of the most secure Cipher Suite by the server from the list of supported suites presented by the client.

If there isn't a common Cipher Suite between the client and server, then there is no way for the two machines to establish a secure channel - as there is no common language that will be understood buy each party that provides the necessary protection offered by SSL.

Tip 3: Ensure that the appropriate Cipher Suites are enabled on the client and server sides in order to establish this common language for secure message exchange.

Tip 4: Utilize SSL debug information to determine which cipher suites has been selected
...

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <SSLTrustValidator returns: 0>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Trust status (0): NONE>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Performing hostname validation checks: stabd58>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: false>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <17849795 SSL3/TLS MAC>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <17849795 received HANDSHAKE>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <HANDSHAKEMESSAGE: ServerHelloDone>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm MD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RC4>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RSA>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <write HANDSHAKE, offset = 0, length = 70>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <write CHANGE_CIPHER_SPEC, offset = 0, length = 1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RC4>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HMACMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HMACMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1>

<Aug 19, 2009 2:34:57 PM PDT> <Debug> <SecuritySSL> <BEA-000000> <write HANDSHAKE, offset = 0, length = 16>

...


Tip 5: Utilize a tool such as SSLDump as necessary to see details of the handshake and application data message exchanges
...

11 1 0.0035 (0.0035) C>S SSLv2 compatible client hello

Version 3.1

cipher suites

TLS_RSA_WITH_RC4_128_MD5

SSL2_CK_RC4

TLS_RSA_WITH_RC4_128_SHA

TLS_DHE_DSS_WITH_RC4_128_SHA

TLS_ECDH_ECDSA_WITH_RC4_128_SHA

Unknown value 0x4e

Unknown value 0x2f

Unknown value 0x35

Unknown value 0x4b

Unknown value 0x4c

TLS_RSA_WITH_3DES_EDE_CBC_SHA

TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA

TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA

TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA

Unknown value 0x50

TLS_RSA_WITH_DES_CBC_SHA

TLS_DHE_DSS_WITH_DES_CBC_SHA

TLS_DHE_RSA_WITH_DES_CBC_SHA

TLS_ECDH_ECDSA_WITH_DES_CBC_SHA

Unknown value 0x4f

TLS_RSA_EXPORT1024_WITH_RC4_56_SHA

TLS_DHE_DSS_WITH_RC2_56_CBC_SHA

TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA

TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA

TLS_RSA_EXPORT_WITH_RC4_40_MD5

SSL2_CK_RC4_EXPORT40

TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_DH_anon_WITH_3DES_EDE_CBC_SHA

TLS_DH_anon_WITH_RC4_128_MD5

TLS_DH_anon_WITH_DES_CBC_SHA

TLS_DH_anon_EXPORT_WITH_RC4_40_MD5

TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA

TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA

11 2 0.0053 (0.0017) S>C Handshake

ServerHello

Version 3.1

session_id[0]=


cipherSuite TLS_DH_anon_WITH_3DES_EDE_CBC_SHA

compressionMethod NULL

11 3 0.0053 (0.0000) S>C Handshake

ServerKeyExchange

Short read: 0 bytes available (expecting 2)

11 4 0.0065 (0.0012) S>C Handshake

ServerHelloDone

11 5 0.0976 (0.0910) C>S Handshake

ClientKeyExchange

DiffieHellmanClientPublicValue[128]=

8a 23 78 02 8a a5 fc 03 f4 9b 7c 33 05 22 36 91

85 9d 17 e4 bf bf 0a 3e be 45 25 47 07 e0 9c a2

e5 d6 bf 78 95 f1 84 ca cb cc e4 3e f3 d8 d4 9a

3a 01 71 5c 29 0c 0b f9 69 8d 3e a6 f4 08 f0 36

18 fd a7 b9 3e 30 4e a4 a6 19 d9 d3 64 1c 3c 78

d3 c3 fa 83 07 58 f2 be d2 32 80 c0 32 4e 49 4c

bf 73 1a f2 d8 fd f2 16 c7 31 da 48 58 50 bb 99

3f a4 8c 31 6e 5f ed e8 0d d8 91 cf 8f eb fa d8

11 6 0.0976 (0.0000) C>S ChangeCipherSpec

11 7 0.0976 (0.0000) C>S Handshake

11 8 0.0997 (0.0021) S>C ChangeCipherSpec

11 9 0.1000 (0.0002) S>C Handshake

11 10 0.3580 (0.2580) C>S application_data

11 11 0.3580 (0.0000) C>S application_data

11 12 0.3586 (0.0005) S>C application_data

11 13 2.5039 (2.1453) C>S application_data

11 14 2.5039 (0.0000) C>S application_data

11 15 2.5053 (0.0013) S>C application_data

8 20 31.4483 (3.3621) C>S application_data

8 21 31.4483 (0.0000) C>S application_data

8 22 31.4507 (0.0024) S>C application_data

8 23 31.4508 (0.0000) S>C application_data

8 24 32.0824 (0.6316) C>S application_data

8 25 32.0824 (0.0000) C>S application_data

8 26 32.2550 (0.1726) S>C application_data

8 27 32.2550 (0.0000) S>C application_data

8 28 33.1710 (0.9159) C>S application_data

8 29 33.1710 (0.0000) C>S application_data

8 30 33.1745 (0.0035) S>C application_data

8 31 33.1754 (0.0009) C>S application_data

...

Anonymous Cipher Suite
- Categories: Configuration

The failure of a client or server to reject a certificate that is not trusted may present as potential SSL problem. Recall earlier that I describe the process of selecting the most secure Cipher Suite common between both parties.

In a scenario where one of the parties has only the anonymous Cipher Suite enabled and the other party also has it enabled - even if it is one of many - the anonymous cipher suite will be selected and the connection will not be rejected.

Tip 6: see Tip 5 above - in fact, the example ssldump output above is from troubleshooting just such a scenario

Trusted CA's

- Categories: Trust, Configuration

Unless the issuer of a certificate is found in the Truststore of a client or server involved in the establishment of an SSL connection, the certificate validation will fail.

Tip 7: Determine the Truststore/s in use and whether or not the issuer of the presented certificate exists within the configured Truststore

Tip 8: Utilize keytool in order to dump the contents of the Truststores (or keystores for the presented certificates)

Alias name: ttelesecglobalrootclass3ca
Creation date: Feb 10, 2009
Entry type: trustedCertEntry

Owner: CN=T-TeleSec GlobalRoot Class 3, OU=T-Systems Trust Center, O=T-Systems Enterprise Services GmbH, C=DE
Issuer: CN=T-TeleSec GlobalRoot Class 3, OU=T-Systems Trust Center, O=T-Systems Enterprise Services GmbH, C=DE
Serial number: 1
Valid from: Wed Oct 01 03:29:56 PDT 2008 until: Sat Oct 01 16:59:59 PDT 2033
Certificate fingerprints:
MD5: CA:FB:40:A8:4E:39:92:8A:1D:FE:8E:2F:C4:27:EA:EF
SHA1: 55:A6:72:3E:CB:F2:EC:CD:C3:23:74:70:19:9D:2A:BE:11:E3:81:D1
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: B5 03 F7 76 3B 61 82 6A 12 AA 18 53 EB 03 21 94 ...v;a.j...S..!.
0010: BF FE CE CA ....
]
]


Certificate Expiration
- Categories: Certificate Validation, Configuration

Tip 9: see Tip 8 above - In the example keytool output above you can see the dates for which the particular certificate is valid.

Valid from: Wed Oct 01 03:29:56 PDT 2008 until: Sat Oct 01 16:59:59 PDT 2033

Random Number Generation (RNG) Issue
- Categories: Configuration

Performance issues may be encountered due to low or zero entropy on a server. This entropy results in longer than expected blocking in acquiring the random number seeding from /dev/random. There are a couple potential workarounds.

1.use /dev/urandom - NOTE: this may result in degenerated encryption strength and must be investigated by your system/security administrators
2.patches may be available for your particular Linux flavor or Solaris

Tip 10: Observe through SSL debug output whether or not the handshake is timing out as this is an indicator that perhaps we are blocking on the RNG

Tip 11: Ensure that all related patches have been installed on your machine.

Available Tools and Facts

Like any other specialization, troubleshooting and debugging security - and SSL in particular - presents unique challenges and to address these unique challenges we need to be prepared by having appropriate tools and facts at our disposal.

In order to be productive in this area, we need to know certain things about the environment, management consoles, etc.


Debug output
Each middleware platform provides the ability to configure the server to run with SSL debug logging turned on. This configuration enables the viewer of the logs to see pertinent information regarding the configuration and runtime behavior of the handshaking and application data message exchanges in real time.

Tip 12: Determine what the configuration mechanism is for turning on SSL debug information on your platform. On WebLogic Server the following System Properties are used to configure SSL debug information and can be used on the command line or within start up scripts:

-Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true -Dweblogic.security.SSL.verbose=true

Tip 13: When you have access to starting the server with these System properties do so immediately - the information that it creates will be valuable - if not to you then to someone else that is pulled in to help debug - at which time you will be request to do so anyway and go through the whole thing again.

SSLDump

In cases where we don't have access to the server to restart with SSL debug logging or when we would like to supplement that output with additional information SSLDump is hugely valuable.

ssldump is an SSLv3/TLS network protocol analyzer. It identifies TCP connections on the chosen network interface and attempts to interpret them as SSLv3/TLS traffic. When it identifies SSLv3/TLS traffic, it decodes the records and displays them in a textual form to stdout. If provided with the appropriate keying material, it will also decrypt the connections and display the application data traffic.

Tip 14: Download a copy of ssldump from http://www.rtfm.com/ssldump/. You will need to build it on your platform and you may actually need to resolve a couple compilation errors - but it is well worth it.

Examples:

To listen to traffic on port 443:
ssldump -i eth0 port 443

To listen to traffic to the server target on port 443:
ssldump -i eth0 port 443 and host target

To decrypt traffic to the host target server.pem and the password foobar:
ssldump -i eth0 -Ad -k ~/server.pem -p foobar host target

Generic SSL Client

Once you have ssldump built and running on your machine, you can use any SSL client to target the server that you are trying to troubleshoot. Often a browser will suffice - however you may need to build a client more appropriate for your usecase.

Tip 15: Utilize ssldump, SSL debug logging and your SSL client to observe the messages exchanged and the runtime behavior that manifests as a result of your current configuration.

Platform Specific Knowledge

Become intimately familiar with where the appropriate keystores, truststores, configuration files and management consoles are located.

Tip 16: Maintain a checklist of this information and keep it handy so that you don't have to rediscover it every time you encounter SSL issues.

Books

SSL and TLS: Designing and Building Secure Systems, Addison-Wesley, 2001 ISBN 0-201-61598-3

http://www.rtfm.com/sslbook/

Use WLST to configure AQ JMS in Weblogic

For AQ JMS Weblogic integration, our documents were based on Weblogic admin console. While the admin console is more user friendly because of its intuitive and easy GUI interface, it is less desirable for some situations such as following comparing with WLST.

1) The configuration is huge with hundreds of connection factories and destinations
2) The same configuration need to be done repeatedly many times
3) The User wants to keep an easy to read record of what have been configured

The difficulty of directly using the WLST is to understand and navigate the complex structure of WebLogic management. In this blog, we demonstrate a jython script to help to overcome this difficulty.

The script is structured into several jython functions so it can be easily customized by other people. Here is the list of functions

def createDataSource(host, port, sid, username='', password='', \
dsName=DEFAULT_DATASOURCE_NAME, \
dsJndiName=DEFAULT_DATASOURCE_JNDI, xa=DEFAULT_DATASOURCE_XA, \
serverTargets=DEFAULT_DATASOURCE_SERVER_TARGETS, \
clusterTargets=DEFAULT_DATASOURCE_CLUSTER_TARGETS):

This function creates a Weblogic datasource for AQJMS usage

host: the host name of the Oracle database
port: the port number of the Oracle database
sid: the SID of the Oracle database
username: the Oracle database username
password: the Oracle database password
dsName: the name of the datasource, the default is 'aqds'
dsJndiName: the JNDI name of the datasource, the default is 'jdbc/aqds'
xa: whether to use the XA driver or non-XA driver
serverTargets: the server targets of this datasource in a comma separated string
the default is 'AdminServer'
clusterTargets: the cluster targets of this datasource in a comma separated string
the default is empty string

This function will ask for Oracle database username and password dynamically
if they are not passed by parameters

def createAQModule(aqcfs, aqdests, \
moduleName=DEFAULT_AQ_MODULE_NAME, \
fsName=DEFAULT_AQ_FS_NAME, \
dsJndiName=DEFAULT_DATASOURCE_JNDI, \
serverTargets=DEFAULT_AQ_MODULE_SERVER_TARGETS,\
clusterTargets=DEFAULT_AQ_MODULE_CLUSTER_TARGETS):

This function creates an AQ module in Weblogic Server
aqcfs: A sequence of connection factories data of this AQ foreign server
aqdests: A sequence of destinations data of this AQ foreign server
moduleName: the name of this AQ module, the default is 'aq_module'
fsName: the name of the AQ foreign server, the default is 'aq_fs'
dsJndiName: the JNDI name of the Weblogic datasource of this AQ foreign server,
the default is 'jdbc/aqds'
serverTargets: the server targets of this AQ module in a comma separated string
the default is ''AdminServer'
clusterTargets: the cluster targets of this AQ module in a comma separated string
the default is empty string

def setupAQJMSFromFile(filepath):

This function parses the configuration file and setup AQ module and datasource in
Weblogic with the data in the configuration file. The configuration file is in key=value
format.

The list of mandatory keys are listed below
HOST: the host of AQ server
PORT: the port of AQ server
SID: the database sid of AQ server
USERNAME: the database username of AQ server
PASSWORD: the database password of AQ server

The list of optional keys are listed below
CONN_FACT: the data to create a Weblogic foreign server connection factory
pointing to an AQ JMS connection factory. The value of this property is as
following:
{WLS_CF_NAME:,CF_JNDI:,CF_TYPE:}
where the CF_TYPE has to be one of 'ConnectionFactory',
'QueueConnectionFactory', 'TopicConnectionFactory', 'XAConnectionFactory'
'XAQueueConnectionFactory' and 'XATopicConnectionFactory'
DESTINATION: the data to create a Webloic foreign server destination pointing to an
AQ destination. The value of this property is as following:
{WLS_DEST_NAME:,AQ_DEST_NAME:,DEST_JNDI:
,DEST_TYPE:}
where the DEST_TYPE has to be either 'QUEUE' or 'TOPIC'
DATASOURCE_NAME: the name of the datasource, default to 'aqds'
DATASOURCE_JNDI: the JNDI name of the datasource, default to 'jdbc/aqds'
DATASOURCE_XA: whether the datasource uses XA driver, default to 1
DATASOURCE_SERVER_TARGETS: the server targets of this datasource in a
comma separated string. default to 'AdminServer'
DATASOURCE_CLUSTER_TARGETS: the cluster targets of this datasource in a
comma separated string. default to empty string
AQ_MODULE_NAME: the Weblogic module for AQJMS, default to 'aq_module'
AQ_FS_NAME: the Weblogic foreign server name for AQJMS, default to 'aq_fs'
AQ_MODULE_SERVER_TARGETS: the server targets of this aq module in a comma
separated string. default to 'AdminServer'
AQ_MODULE_CLUSTER_TARGETS: the cluster targets of this aq module in a
comma separated string. default to empty string


When the script is run, it will ask the user for a configuration file and call 'setupAQJMSFromFile' with this file. Most configuration parameters in this script have default value except the most essential ones such as database host, port, sid, connection factory info and destination info. The script also does a best effort validation of these data.


Here are some examples of using this script.

1) Running the script standalone

java weblogic.WLST aq_wlst.py

with the AQ configuration file as following:
HOST=localhost
PORT=5521
SID=db11
USERNAME=jmsuser
PASSWORD=jmsuser
CONN_FACT=WLS_CF_NAME:aqcf,CF_JNDI:aqcf,CF_TYPE:XAQueueConnectionFactory}
DESTINATION=WLS_DEST_NAME:aq_queue,AQ_DEST_NAME:MY_QUEUE,DEST_JNDI:aq_queue,DEST_TYPE:QUEUE}
DATASOURCE_SERVER_TARGETS=ms1,ms2
DATASOURCE_CLUSTER_TARGETS=Cluster-0
AQ_MODULE_SERVER_TARGETS=ms1,ms2
AQ_MODULE_CLUSTER_TARGETS=Cluster-0

After it runs, the datasource, AQ module, AQ foreign server, connection factories and destinations will be configured and targeted. The datasource name, datasource JNDI name, AQ module name, AQ foreign server name are not provided in the configuration file, so the script uses the default value as above.

2) Running the script to execute multiple AQ configuration files in one transaction

The script is structured into several functions so it is easy to customize or extend. For example, here is an example extension. We can modify the main function of the script to get all the files with the extension conf in the current directory and execute them in one transaction instead of asking for configuration file from the user.


import glob
connect()
failed=1
try:
edit()
startEdit()
for cf in glob.glob("*.conf"):
setupAQJMSFromFile(cf)
activate(200000, block='true')
failed=0
finally:
if failed:
undo('false','y')
disconnect('true')

The script and a sample configuration file are attached with this blog.

aq_wlst.py

myconfig

October 18, 2009

Developing custom MBeans to manage J2EE Applications (Part I)

This is the first part in a series of blogs, that demonstrate how to add management capability to your own application using JMX MBeans.

In this first blog entry, we will learn:

  • How to implement a custom MBean to manage configuration associated with an application.
  • How to package the resulting code and configuration as part of the application's ear file.
  • How to register our MBean upon application startup, and unregistered them upon application stop (or undeployment).
  • How to use generic JMX clients such as JConsole to browse and edit our application's MBean.

The complete code sample and associated build files are available as a zip file. The code has been tested against WebLogic Server 10.3.1 and JDK6.

Implementing the MBean

We chose to implement an MBean that can be used to manage properties associated with an application. To keep things interesting those properties will be persisted, and originally packaged with the deployed application's ear file.

Among the different types of MBeans, we choose to implement an MXBean. MXBeans have the main advantage to expose model specific types as Open Types. This means that clients interacting with MXBeans do not need to include any jar files containing application custom classes. For our simple example, this doesn't really come into play, but this also doesn't complicate our task either. So MXBean it is.

If you want to know more about MXBean, a tutorial is available here

MBean interface

The MBean interface declares the JMX management interface exposed by the MBean. Java methods translate to JMX attributes ( Java bean getter/setter pattern ) or to JMX operations ( the remaining methods ). More info on the mapping from Java methods to JMX attributes and operations is available in the JMX specification or in this tutorial. The same mapping rules apply to both Standard MBeans and MXBeans.


package blog.wls.jmx.appmbean;

import java.util.Map;
import java.io.IOException;

public interface PropertyConfigMXBean {

    public String setProperty(String key, String value) throws IOException;\
    public String getProperty(String key);
    public Map getProperties();
}

The above interface declares one attribute: Properties, and two operations: setProperty and getProperty. Those are the attribute and operations exposed by our MBean to JMX clients.

MBean implemetation

Our MBean implementation relies on the JDK's Properties class to manage and persist the underlying properties. The complete code for the MBean implementation is included below, and specific sections are discussed thereafter.


package blog.wls.jmx.appmbean;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.File;

import java.net.URL;

import java.util.Map;
import java.util.Properties;

import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.management.MBeanRegistration;

public class PropertyConfig implements PropertyConfigMXBean, MBeanRegistration {

    private String relativePath_ = null; 
    private Properties props_ = null;
    private File resource_ = null;

    public PropertyConfig(String relativePath) throws Exception {
        props_ = new Properties();
        relativePath_ = relativePath;
    }

    public String setProperty(String key,String value) throws IOException {
        String oldValue = null;

        if (value == null) {
            oldValue = String.class.cast(props_.remove(key));
        } else {
            oldValue = String.class.cast(props_.setProperty(key, value));      
        }

        save();
        return oldValue;
    }

    public String getProperty(String key) {
        return props_.getProperty(key);
    }

    public Map getProperties() {
        return (Map) props_;
    }

    private void load() throws IOException {        
        InputStream is = new FileInputStream(resource_);

        try {
            props_.load(is);
        }
        finally {
            is.close();
        }
    } 

    private void save() throws IOException {  
        OutputStream os = new FileOutputStream(resource_);

        try {
            props_.store(os, null);
        }
        finally {
            os.close();
        }
    }

    public ObjectName preRegister(MBeanServer server, ObjectName name)
        throws Exception {
        // MBean must be registered from an application thread
        // to have access to the application ClassLoader
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        URL resourceUrl = cl.getResource(relativePath_);
        resource_ = new File(resourceUrl.toURI());
        load();     
        return name;
    }

    public void postRegister(Boolean registrationDone) { }
    public void preDeregister() throws Exception {}
    public void postDeregister() {}     
}

Our MBean implements the PropertyConfigMXBean interface, as well as the MBeanRegistration interface.

We use the MBeanRegistration.preRegister method to load the persisted properties upon MBean registration. The original properties are included as part of the deployed ear, and accessed using the application's Context ClassLoader:

        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        URL resourceUrl = cl.getResource(relativePath_);
        resource_ = new File(resourceUrl.toURI());
        load();     

The interesting part in the above code is the use of the application's Context ClassLoader to extract the path under which the deployed ear's property file is located. For this to work we must ensure that the preRegister method is triggered by an application Thread, so that we can access the proper ClassLoader. More on this later.

The load and save methods are quite straight forward. The load method use the Properties class to read the properties from the application's property file into the MBean state. The save method use the Properties class to persist the properties from the MBean state back into the application's property file.

Any new property added using the setProperty method is immediately persisted by calling the save method within the setProperty method. On a more complex project, one could expose save as a JMX operation, and build an associated Configuration Session MBean. This is beyond this blog's topic thought.

MBean life-cycle implementation

To register our MBean upon application start, and unregister it upon application stop (or undeploy), we use WebLogic's ApplicationLifecycleListener class. Note: We could have used the J2EE ServletContextListener class in place, but we chose the ApplicationLifecycleListener as it also works for applications that do not include web modules. The complete code is included below:

package blog.wls.jmx.appmbean;

import weblogic.application.ApplicationLifecycleListener; 
import weblogic.application.ApplicationLifecycleEvent; 
import weblogic.application.ApplicationException; 

import javax.management.ObjectName; 
import javax.management.MBeanServer; 

import javax.naming.InitialContext; 

public class ApplicationMBeanLifeCycleListener extends ApplicationLifecycleListener {

    public void postStart(ApplicationLifecycleEvent evt) throws ApplicationException { 
        try { 
            InitialContext ctx = new InitialContext(); 
            MBeanServer mbs  = 
                MBeanServer.class.cast( ctx.lookup("java:comp/jmx/runtime") ); 
            PropertyConfig mbean = new PropertyConfig("config/properties.data");
            ObjectName oname = new ObjectName(
                "blog.wls.jmx.appmbean:type=PropertyConfig,name=myAppProperties"); 
            mbs.registerMBean(mbean, oname); 
        } 
        catch (Exception e) {
            // Deal with exception
            e.printStackTrace();
        } 
    } 

    public void preStop(ApplicationLifecycleEvent evt) throws ApplicationException { 
        try { 
            InitialContext ctx = new InitialContext(); 
            MBeanServer mbs  = 
                MBeanServer.class.cast( ctx.lookup("java:comp/jmx/runtime") ); 
            ObjectName oname = new ObjectName(
                "blog.wls.jmx.appmbean:type=PropertyConfig,name=myAppProperties");
            if ( mbs.isRegistered(oname) ) { 
                 mbs.unregisterMBean(oname); 
            }
        } 
        catch (Exception e) {
            // Deal with exception
            e.printStackTrace();
        } 
    }        
}

In the above code we register our MBean in WebLogic's Runtime MBeanServer: ctx.lookup("java:comp/jmx/runtime"). This MBeanServer is present on all WebLogic's processes, and can be used to register application's custom MBeans.

We construct an instance of our MBean: new PropertyConfig("config/properties.data") passing the relative path to the application's property file. The path is relative to the root of the ear file, as we will see in the next section. Remember from the previous section we used the application Context ClassLoader to load the application property file using the relative path provided above: cl.getResource(relativePath_);. This is made possible by the postStart method that is executed by an application Thread.

We register our MBean under the "blog.wls.jmx.appmbean:type=PropertyConfig,name=myAppProperties" ObjectName. We followed the JMX best practices when picking that name. We recommend you do the same as this greatly helps:

  • Avoid naming collisions.
  • Organize MBeans in MBean browsers. For instance JConsole's MBean tree.
  • Provide a consistent MBean naming structure when MBeans from many different sources are registered in a single MBeanServer.

Another point worth mentioning is the importance of unregistering the MBean when the application is stopped. If the MBean is left registered, the application ClassLoader(s) will be pinned and resource will be leaked.

Packaging the ear file

Now we need to package our code and property file so that both can be loaded by the application Context ClassLoader.

The MBean and life cycle code is packaged in a jar as follow:

$ jar tvf sample-mbean-app-mbeans.jar
     0 Tue Oct 07 16:35:28 PDT 2009 META-INF/
   121 Tue Oct 07 16:35:26 PDT 2009 META-INF/MANIFEST.MF
     0 Tue Oct 07 16:35:26 PDT 2009 blog/
     0 Tue Oct 07 16:35:26 PDT 2009 blog/wls/
     0 Tue Oct 07 16:35:26 PDT 2009 blog/wls/jmx/
     0 Tue Oct 07 16:35:26 PDT 2009 blog/wls/jmx/appmbean/
  1337 Tue Oct 07 16:35:26 PDT 2009 
            blog/wls/jmx/appmbean/ApplicationMBeanLifeCycleListener.class
  2417 Tue Oct 07 16:35:26 PDT 2009 blog/wls/jmx/appmbean/PropertyConfig.class
   408 Tue Oct 07 16:35:26 PDT 2009 
           blog/wls/jmx/appmbean/PropertyConfigMXBean.class
$ 

The jar's manifest includes the following entry: Class-Path: ../... This will ensure that the application's property file can be accessed from the application's ClassLoader. More on this later.

The ear file is packaged as follow:

$ jar tvf sample-mbean-app.ear
     0 Tue Oct 07 16:35:28 PDT 2009 META-INF/
   102 Tue Oct 07 16:35:26 PDT 2009 META-INF/MANIFEST.MF
   602 Tue Oct 0714:50:34 PDT 2009 META-INF/application.xml
   719 Tue Oct 07 16:35:26 PDT 2009 app.war
     0 Tue Oct 07 16:35:28 PDT 2009 APP-INF/
     0 Tue Oct 07 16:35:28 PDT 2009 APP-INF/lib/
  3328 Tue Oct 07 16:35:26 PDT 2009 APP-INF/lib/sample-mbean-app-mbeans.jar
     0 Tue Oct 07 16:35:28 PDT 2009 config/
   105 Tue Oct 07 30 12:46:28 PDT 2009 config/properties.data
   422 Tue Oct 07 13:10:52 PDT 2009 META-INF/weblogic-application.xml
$

The MBean jar file sample-mbean-app-mbeans.jar is added under the APP-INF/lib directory. Any jars located under that directory will be added to the application's ClassLoader.

The application's property file is added under config/properties.data, and can be accessed by the application's Context ClassLoader using that path. This works because we added the Class-Path: ../.. entry in sample-mbean-app-mbeans.jar's manifest, and we placed that jar under the APP-INF/lib directory.

META-INF/application.xml declares a single war module app.war, that is empty, and whose only purpose is to make our application valid by including at least one J2EE module.

META-INF/weblogic-application.xml declares our ApplicationLifecycleListener:

<?xml version="1.0" encoding="UTF-8"?>
<weblogic-application xmlns="http://www.bea.com/ns/weblogic/90"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
  xsi:schemaLocation="http://www.bea.com/ns/weblogic/90 
  http://www.bea.com/ns/weblogic/90/weblogic-application.xsd">
  <listener>
    <listener-class> 
        blog.wls.jmx.appmbean.ApplicationMBeanLifeCycleListener
    </listener-class>
  </listener>
</weblogic-application>

Download the code and build file

The above code and associated build files are available as a zip file. Once you have downloaded the zip file, extract its content, and cd to the app_mbean_part1 directory.

In order to build the code and create the application's ear file, you first need to define the WL_HOME environment property to point to your WebLogic binary install. Make sure $WL_HOME/lib/weblogic.jar is valid. Then just execute:

ant

If everything goes well the sample-mbean-app.ear file should now be located under the build directory.

Deploying the ear

The sample-mbean-app.ear file can be deployed as follow:

java -classpath $WL_HOME/lib/weblogic.jar weblogic.Deployer -noexit -name sample-mbean-app -source build/sample-mbean-app.ear -targets jrfServer_admin -adminurl t3://140.87.10.42:7086 -user weblogic -password gumby1234 -deploy

Make sure to substitute:

  • Your WebLogic server host and port in place of 140.87.10.42:7086. The port value for your WebLogic server is available from the <WLS_INSTANCE_HOME>/config/config.xml file:
     <server>
        <name>jrfServer_admin</name>
        <listen-port>7086</listen-port>
        <listen-address>140.87.10.42</listen-address>
      </server>
    
    Make sure you look under the correct server if several servers are defined as part of your config.xml. For instance in the above case we are connecting to the server identified as "jrfServer_admin ".

  • Your WebLogic server name in place of jrfServer_admin
  • Your WebLogic server administrator's login and password in place of weblogic and gumby1234

The following command can be used to undeploy the sample-mbean-app application:

java -classpath $WL_HOME/lib/weblogic.jar weblogic.Deployer -noexit -name sample-mbean-app -targets jrfServer_admin -adminurl t3://140.87.10.42:7086 -user weblogic -password gumby1234 -undeploy

Using JConsole to browse and edit our application MBean

Using JConsole to connect to WebLogic's MBeanServers was the subject of an earlier blog entry. I recommand reading through that blog if you want to know more. Here I will just fast forward to the command line used:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:$WL_HOME/lib/wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -debug

Once JConsole is started select the Remote process connection, and fill out the target MBeanServer URI, Username and Password fields:

service:jmx:iiop://140.87.10.42:7086/jndi/weblogic.management.mbeanservers.runtime

Make sure you replace the host and port values with the ones you used in your deployment command line. The username and password values are also the ones you used as part of your deployment command line.

You should now be connected and able to see our application MBean under the blog.wls.jmx.appmbean tree entry:

app_mbean_part1_jconsole1.JPG

You can experience with creating new properties using the setProperty operation, and browsing them with either the getProperty operation or Properties attribute. Notice that the Properties attribute is exposed to JMX clients as a TabularData:

app_mbean_part1_jconsole2.JPG

The conversion between the Map instance returned by our public Map getProperties() method and the TabularData exposed to JMX clients is performed by the MXBean implementation that is part of the JDK. More info on MXBeans and how they convert native types to Open Types is available here.

You might wonder where is the property file persisted. The answer is it depends on the deployment staging mode, and whether the application is deployed as an ear file or an exploded directory. In most cases the file will be located in a temporary directory under the target server. For instance:

servers/jrfServer_admin/tmp/_WL_user/sample-mbean-app/dgiyyk/config/properties.data

In this case, re-deploying the ear (provided the ear was modified) will overwrite the existing property file with the one contained in the ear file. Any new property added since the application was last deployed will be lost. It is also not possible to share the application's properties among several WebLogic servers, as each server gets its own local copy of the property file.

We can work around this by either:

  • Removing the property file from the ear file, and access it from a well known external location using regular file I/O.
  • Deploying the application as an exploded ear accessible from all targeted WebLogic servers.

    java -classpath $WL_HOME/lib/weblogic.jar weblogic.Deployer -noexit -name sample-mbean-app -source build/exploded-ear -targets jrfServer_admin -adminurl t3://140.87.10.42:7086 -user weblogic -password gumby1234 -deploy

    Where the exploded-ear directory contains: jar xvf sample-mbean-app.ear.

    In this case the property file will be modified under its original location: build/exploded-ear/config/properties.data.

Note: If the property file is shared among several MBeans, then some form of synchronization will need to be provided across those MBeans when reading/writing the property file. We won't get into this in this blog.

Aside from JConsole we can also use a Java client to browse and edit our MBean. This is demonstrated in an earlier blog entry. Another possibility is to use Oracle's Enterprise's Manager.

What's next?

Even thought our MBean is functional, it cannot be considered production ready. It is lacking descriptions for the MBean itself, its attributes, operations and operation parameters. The operation parameters are also lacking meaningful names as demonstrated below:

app_mbean_part1_jconsole3.JPG

Our goal next time around will be to fix the above issues.

October 12, 2009

Are My Servers Running?

Sometimes, it is necessary to know whether one or more servers within a WebLogic Server domain are not running, and perform appropriate actions. For example, one may want to execute custom scripts if one or more servers within the domain are not in the RUNNING state. The WebLogic Diagnostic Framework (WLDF), which is part of the WebLogic Server makes it easy to send notifications when such conditions are detected. When configured rules match runtime conditions, notifications associated with the rules are sent. WLDF can send notifications in various forms, such as emails, JMX notifications, JMS message, SNMP traps and also cause a Diagnostic Image to be created, which consists of important state of the entire server.

In this blog, I will describe how a watch-notification can be configured to send a JMX notification when one of the servers in the domain is not running. I will also show a simple JMX notification listener program which can process WLDF JMX notifications. In this case, it will simply print the contents of the notification. However, it can be easily adapted to do other things such as running a WLST script.

For simplicity, we will have only two servers in the domain, myserver (admin-server) and MS1 (managed-server). We will configure WLDF on the admin-server to send a JMX notification when the state of the managed-server MS1 (actually, any server within the domain) is not RUNNING.

Configuring WLDF watch/notifications

We will use WebLogic Server Administration Console to configure WLDF. It can also be done with WLST. Console provides a nice feature to Create Repeatable Configuration Scripts, which can be handy for building such scripts. Configuration steps for this case consist of:

  1. Create a Diagnostic Module.
  2. Enable periodic metrics collection in the Diagnostic module.
  3. Create a JMX notification type.
  4. Create a watch to detect inactive servers
  5. Associate the created JMX notification with the configured watch
  6. Target the Diagnostic module to the admin server

We will go over these steps and see how to configure WLDF using Console for this task. Log into Console and acquire Edit Lock so we can add new configuration to WebLogic Server.

Create a Diagnostic Module.

From the left navigation tree, open the Diagnostics node and click on Diagnostic Modules. Console will show a table of existing Diagnostic modules within the domain. Click the New button to create a new diagnostic module. Call it myWLDF. Click Save to create the module. At this point, we have an empty myWLDF diagnostic module. We will proceed to configure it.

Enable periodic metrics collection in the Diagnostic module.

Click on the myWLDF module link in the table of Diagnostics modules. Click on Collected Metrics sub-tab under Configuration tab. Check the Enabled checkbox and set the Sampling Period to 10000 (10 seconds). Click Save.

Create a JMX notification type.

Configuring a watch/notification has two aspects. The first aspect is a watch rule which specifies the condition that WLDF will check. The second aspect is the set of notifications that will be sent when the rule condition is met. Console provides configuration assistants to make the configuration task easier. To create a notification type:

  • Click Watches and Notifications sub-tab under Configuration tab.
  • On the Notifications sub-tab, click New in the Notifications table.
  • Select "JMX Notification" for the notification type from the dropdown and click Next.
  • Give a name to the notification type (myJMX)
  • check the Enable Notification checkbox and click OK to create the notification type.

Create a watch to detect inactive servers

Now, we will create the watch rule. We will create a watch rule based on runtime mbean data. Specifically, we will use the State attribute on the ServerLifeCycleRuntimeMBean mbeans. For each server within the domain, there is a ServerLifeCycleRuntimeMBean mbean in the domain runtime mbean server. The State attribute reflects the current state of the server. We will configure a rule which will fire when at least one server in the domain is not in the RUNNING state.

  • Click on the Watches sub-tab. Click New in the Watches table.
  • Set Watch Name to WatchServerStatus. Select Collected Metrics for Watch Type, check Enabled checkbox and click Next.

Now we come to the interesting part, which is configuration of the watch rule expression itself. You can hand-edit the watch expression by clicking the Edit button and entering the watch expression yourself. However, that will require some familiarity with the watch
expression syntax. For now, we will use the rule-expression builder.

  • Click on Add Expression button.
  • Select DomainRuntime for the MBean Server Location and click Next.
  • From the drop-down list for MBean Type, select weblogic.management.runtime.ServerLifeCycleRuntimeMBean and click Next.
  • Leave the Instance list empty and click Next.
  • Select State attribute from the list of available attributes.
  • Select "!=" (Not Equals) as comparison operator.
  • Set RUNNING as the value for the comparison and click Finish.
  • Click Finish to complete the rule-expression.

Note that the ServerLifeCycleRuntimeMBean is only registered in the Domain runtime mbean server. So, we selected DomainRuntime for the MBean Server Location. Also, since we want to receive notification for any server which is not running, we left the instance list empty and not restrict it to specific instances.

The rule-expression builder would create a rule expression:

(${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != 'RUNNING')

Associate notification and configure alarm type

  • Click on the WatchServerStatus watch from the Watches table.
  • Click on the Notifications tab
  • Choose the myJMX notification from the chooser and click Save
  • Click on the Alarms tab and select "Don't use an alarm". Click Save.

With this, we associated the created JMX notification with the watch. With "Don't use an alarm", we disabled the watch-alarms for now, which will cause a notification to be sent in every cycle when the rule evaluates to true. Later, you may want to use other options such as "Automatic Alarm Reset" so notifications will be suppressed for a configured interval to avoid a flood of notifications.

Finishing the configuration

Do not forget to target the created Diagnostic module to the admin-server (myserver). It will take effect only when the module is targeted to the server. Another thing to note is that, WLDF allows at most one module to be targeted to a server. So, if you have another diagnostic module targeted to the server, you will need to un-target it first. Ofcourse, in that case, you may add the above configuration to the pre-existing module instead of creating a new one.

Lastly, activate the changes so it will take effect.

Ok, so we have configured WLDF watch/notifications. What does it mean? Once above configuration is activated, WLDF will periodically evaluate configured watch rules The period is specified by the Sampling Period. If a watch rule evaluates to true, its associated notifications
are sent. As per the configuration above, WLDF will periodically (every 10 seconds) check if the State attribute of any of the ServerLifeCycleRuntimeMBean mbeans is anything other than RUNNING. If so, it will send a JMX notification.

Receiving Notifications

WLDF sends a JMX notification on a specific WLDF runtime mbean, whose ObjectName is of the form:

com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=$SERVER,
Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,
WLDFWatchNotificationRuntime=WatchNotification

where $SERVER is the name of the WebLogic Server instance. For our case (myserver), it is:

com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=myserver,
Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,
WLDFWatchNotificationRuntime=WatchNotification

By registering for JMX notifications on this mbean, a client program can listen to generated notifications.

JMXWatchNotificationListener.java is a simple notification listener for WLDF JMX notifications. It simply prints the contents of received notification, but can be easily adapted to perform other actions.

Run the program and kill the managed server MS1. The client program displays notifications as shown below, when the managed server MS1 exits the RUNNING state.

$ java JMXWatchNotificationListener -server myserver -host localhost -port 7001 -user weblogic -password welcome1

URL=service:jmx:t3://localhost:7001/jndi/weblogic.management.mbeanservers.runtime
Adding notification handler for: com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=myserver,
Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,
WLDFWatchNotificationRuntime=WatchNotification
===============================================
Notification name: myjmx called. Count= 1.
Watch severity: Notice
Watch time: Oct 12, 2009 4:45:48 PM EDT
Watch ServerName: myserver
Watch RuleType: Harvester
Watch Rule: (${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != 'RUNNING'
)
Watch Name: WatchServerStatus
Watch DomainName: mydomain
Watch AlarmType: None
Watch AlarmResetPeriod: 60000
Watch Message: WatchName: WatchServerStatus WatchSeverityLevel: Notice
Watch Data: com.bea:Name=MS1,Type=ServerLifeCycleRuntime//State = SHUTDOWN com.bea:Name=myserver,Type=ServerLifeCycleRun
time//State = RUNNING
===============================================

As mentioned above, other forms of notifications can be configured as well. For example, the article "Monitoring Performance Using the WebLogic Diagnostics Framework" shows how to configure SNMP traps with the watch rules.


October 5, 2009

Managing WebLogic servers with JConsole

In my previous blog I described how to implement a custom WebLogic JMX client. Another way to interact with WebLogic's MBeans, is to use a generic JMX client such as JConsole. JConsole is shipped as part of the JDK, and doesn't require any specific download or installation. This blog demonstrates how simple it is to browse WebLogic MBeans using JConsole.

In order to connect to a WebLogic MBeanServer, JConsole needs to be started as follow:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar: $WL_HOME/server/lib/wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -debug

WL_HOME points to the WebLogic binary install.

The above uses WebLogic's thin client classes. In some cases when you access server side classes, or you want to take full advantage of the t3 protocol, you are required to use WebLogic's thick client classes. To do so, you first need to build wlfullclient.jar as follow:

cd $WL_HOME/server/lib
java -jar wljarbuilder.jar

You can then start JConsole as follow:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar: $WL_HOME/server/lib/wlfulclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -debug

wlfulclient.jar is not dependent on external jars, and can be copied to client machines. When using the thin client ( wljmxclient.jar ) you cannot just copy wljmxclient.jar to a client's machine as this jar references other jars as part of its manifest Class-Path entry. At the time of this writing, you also need to copy wlclient.jar

-J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote identifies the package implementing WebLogic's client side connectors code. For instance the 't3' or 'iiop' client connectors.

Finally the -debug flag is extremely handy when a connection failure occurs. In this case the corresponding stack trace is dumped on the console.

Once JConsole is started select the Remote process connection, and fill out the target WebLogic's MBeanServer JMX URI, Username and Password fields. See figure below:

JConsole not yet connected to WebLogic's Runtime MBeanServer

The JMX service URI:

service:jmx:iiop://140.87.10.42:12565/jndi/weblogic.management.mbeanservers.runtime

identifies the following:

  • The protocol used to communicate with the remote WebLogic process. "iiop" in the above example. WebLogic supports "t3", "iiop" and "rmi". We won't touch further on this topic here.
  • The WebLogic process to connect to (host & port ). 140.87.10.42:12565 in the above example.

    The port value is available from the <WLS_INSTANCE_HOME>/config/config.xml file:

     <server>
        <name>myServer</name>
        <listen-port>12565</listen-port>
        <listen-address>140.87.10.42</listen-address>
      </server>
    
    Make sure you look under the correct server if several servers are defined as part of your config.xml. For instance in the above case we are connecting to the server identified as "myServer".

  • The MBeanServer to connect to: weblogic.management.mbeanservers.runtime. This identifies WebLogic's "Runtime" MBeanServer. This is the MBeanServer available from any WebLogic process, and that contains both WebLogic and user MBeans. WebLogic also offers two other MBeanServers that are only available from the Domain "AdminServer" process:
    • The "Domain Runtime" MBeanServer. It aggregates the MBeans registered on the domain's "Runtime" MBeanServers. So as long as a managed server is up, its MBeans can be accessed through the "Domain Runtime" MBeanServer. To connect to that MBeanServer, just use the following JMX URI:

      service:jmx:iiop://140.87.10.42:7002/jndi/weblogic.management.mbeanservers.domainruntime

      "7002" is the iiop port for my "AdminServer" install. Replace with your AdminServer's port as previously explained. Remember; only the AdminServer process runs the "Domain Runtime" MBeanServer.

    • The "Edit" MBeanServer. It contains the WebLogic Config MBeans that are used to configure the Domain. To connect to that MBeanServer, just use the following JMX URI:

      service:jmx:iiop://140.87.10.42:12565:7002/jndi/weblogic.management.mbeanservers.edit

      "7002" is the iiop port for my "AdminServer" install. Replace with your AdminServer's port as previously explained. Remember; only the AdminServer process runs the "Domain Runtime" MBeanServer.

    Note: One can also use "t3" or "rmi" as protocol in place of "iiop"

Finally provide the login and password values for the user associated with the connection. The WebLogic admin user is often used, but other users can be used as well. Depending on the authenticated user and his/her associated roles, operations on MBeans will be granted or denied. We won't dive further in this topic here. Click connect and voila!

JConsole connected to WebLogic's Runtime MBeanServer

If JConsole fails to connect, look at the stack trace dumped on the console. Make sure you specified the -debug flag when starting JConsole, or you will not see any meaningful information on the console. Common cause of connection failures include: wrong host/port; wrong login/password; jmx.remote.protocol.provider.pkgs not properly specified and incorrect ClassPath. Also note that the above instructions apply to WebLogic server 10.3.1 and above.

JConsole's MBean tree preserves the key order used when building MBeans ObjectNames. The only exception is for the 'type' property which will be included first if present and the 'j2eeType' property which will be included second if present. In general I also use the 'name' property to identify different MBean instances while WebLogic MBeans use 'Type' and 'Name' properties for the same purpose. To ensure that JConsole's MBean tree is build with 'type' or 'Type' as first node and 'name' or 'Name' as its child I use the following command line:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar: $WL_HOME/server/lib/wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -J-Dcom.sun.tools.jconsole.mbeans.keyPropertyList=type,Type,j2eeType,name,Name -debug

When connecting to WebLogic's "Domain Runtime" MBeanServer, one can also order by MBean 'Location' as follow:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar: $WL_HOME/server/lib/wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -J-Dcom.sun.tools.jconsole.mbeans.keyPropertyList=Location,type,Type,j2eeType,name,Name -debug

Aside from browsing and editing MBeans, JConsole can be used to monitor WebLogic processes resource usage: Memory, Threads, Class loading. Consult JConsole's documentation for more information.

October 3, 2009

Using FastSwap to speed up development

A fast turnaround time is something that most developers appreciate and since JDK 5 it is possible to redefine a class without dropping the class loader. However, the support provided by the JDK is limited and does not allow the developer to reshape the class, i.e., fields and methods cannot be changed/altered. FastSwap is a feature within WLS that, when enabled, is designed to overcome these limitations. For example, it will allow the developer to add new methods to an already loaded class and see those changes take effect immediately.

Enabling FastSwap

In order to take advantage of FastSwap, WLS must be running in development mode and FastSwap must have been turned on for that particular deployment. For applications deployed as an EAR, adding the element to the weblogic-application.xml file enables FastSwap and for applications deployed as a WAR, the element should be inserted into the weblogic.xml file instead.

  <fast-swap>
    <enabled>true</enabled>
  </fast-swap>

FastSwap will only monitor changes of classes in exploded directories so if an exploded EAR contains an archived WAR, any changes to that WAR will not be detected. Therefore, the WAR also needs to be unpacked and then FastSwap will monitor all the classes found under the web application's WEB-INF/classes directory. Note that any jars residing in WEB-INF/lib will not be monitored.

Developing with FastSwap

Once FastSwap has been enabled for a particular deployment, it will pick up any change made to a monitored class at runtime, allowing the change to take place immediately.

Using a web application as an example, the typical development flow would be:


  • Deploy the exploded web application

  • Access the application through a web browser

  • Modify the code, compile the changes and have the corresponding class files saved to the correct location under the exploded application.

  • Access the application through a web browser once again, and your changes will now have been applied and should be visible.

Note that FastSwap will not redefine any classes as soon as they have changed on the file system. The reason is because a change to one class file can affect other classes (e.g. when a method signature is changed) so all classes must be redefined at the same time. Therefore, the developer will need to tell FastSwap when it should start its operation. For web applications, this is accomplished by going through a servlet but for other applications the developer has to use the JMX interface for FastSwap. However, there is an Ant task accessing this JMX interface that can used to trigger FastSwap, as shown in the below example.

<project name='MyProject' default='all'>
  <taskdef name='fast-swap' classname='com.bea.wls.redef.ant.FastSwapTask'/>
  <target name='all'>
    <fast-swap
      adminUrl='t3://localhost:7001'
      user='weblogic'
      password='weblogic'
      server='myserver'
      application='SimpleApp'
    />
  </target>
</project>

Where the parameters are:


  • adminUrl: connection url

  • user: user name

  • password: user password

  • server: managed server name

  • application: deployed application name

Limitations & Further Reading

Even though FastSwap is an improvement over the default support found in the JDK there are still some limitations to what it can accomplish. For a list of these limitations as well as more information regarding FastSwap, please visit the following link:

http://download.oracle.com/docs/cd/E12840_01/wls/docs103/deployment/deployunits.html#wp1053872

September 22, 2009

Sample WebLogic JMX Client

Our goal is to implement a simple WebLogic JMX client that can be used as a starting point to write any WebLogic JMX client. We will look at the client's code. The classpath used to compile and run the code. The different WebLogic MBeanServers and associated JMX URI's the JMX Client can connect to.

Let's directly dive in the client's code below:

package blog.wls.jmx.client;

import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXServiceURL;
import javax.management.remote.JMXConnectorFactory;
import java.util.Hashtable;
import java.util.Set;

public class JMXClient {

public static void main(String[] args) throws Exception {

        JMXConnector jmxCon = null;
 
        try {
            JMXServiceURL serviceUrl = 
                new JMXServiceURL(
         "service:jmx:iiop://127.0.0.1:7001/jndi/weblogic.management.mbeanservers.runtime");

            System.out.println("Connecting to: " + serviceUrl);
 
            Hashtable env = new Hashtable();
            env.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, 
                         "weblogic.management.remote");
            env.put(javax.naming.Context.SECURITY_PRINCIPAL, "weblogic");
            env.put(javax.naming.Context.SECURITY_CREDENTIALS, "welcome1");

            jmxCon = JMXConnectorFactory.newJMXConnector(serviceUrl, env);
            jmxCon.connect();
            MBeanServerConnection con = jmxCon.getMBeanServerConnection();
 
            Set<ObjectName> mbeans = con.queryNames(null, null);
            for (ObjectName mbeanName : mbeans) {
                System.out.println(mbeanName);
            }
        }
        finally {
            if (jmxCon != null)
                jmxCon.close();
        }
    }
}

Let's take a quick look at the above code.

JMXServiceURL serviceUrl = 
                new JMXServiceURL(
      "service:jmx:iiop://127.0.0.1:7001/jndi/weblogic.management.mbeanservers.runtime");

The JMX service URI identifies the following:

  • The protocol used to communicate with the remote WebLogic process. "iiop" in the above example. WebLogic supports "t3", "iiop" and "rmi". We won't touch further on this topic here.
  • The WebLogic process to connect to (host & port ). "127.0.0.1:7001" in the above example.

    The port value is available from the <WLS_INSTANCE_HOME>/config/config.xml file:

     <server>
        <name>myServer</name>
        <listen-port>7001</listen-port>
        <listen-address>myhost</listen-address>
      </server>
    
    Make sure you look under the correct server if several servers are defined as part of your config.xml. For instance in the above case we are connecting to the server identified as "myServer".

    The same information is also available from the server log at <WLS_INSTANCE_HOME>/servers/myServer/logs/myServer.log:

    <Sep 22, 2009 12:05:23 PM PDT> <Notice> <Server> <localhost><myServer>
    <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> 
    <<WLS Kernel>> <> <> <1253585123268> <BEA-002613> <Channel "Default" is now
    listening on myhost:7001 for protocols iiop, t3, ldap, snmp, http.> 
    
  • The MBeanServer to connect to: " weblogic.management.mbeanservers.runtime". This identifies WebLogic's "Runtime" MBeanServer. This is the MBeanServer available from any WebLogic process, and that contains both WebLogic and user MBeans.
            Hashtable env = new Hashtable();
            env.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES,
                    "weblogic.management.remote");
            env.put(javax.naming.Context.SECURITY_PRINCIPAL, "weblogic");
            env.put(javax.naming.Context.SECURITY_CREDENTIALS, "welcome1");

The above code specifies properties associated with the connection. "weblogic.management.remote" identifies the package implementing WebLogic's client side connector code. For instance the 't3' or 'iiop' client connector.

"weblogic" and "welcome1" are the login/password associated with the user originating the JMX connection.

In the rest of the code we establish the JMX connection, and then retrieve the ObjectNames for all the MBeans registered in the "Runtime" MBeanServer. The ObjectNames are then displayed on the console. We also make sure we release the connection in a finally block to avoid leaking any resource.

The code doesn't contain any WebLogic specific classes, and can be compiled as follow:

javac -d . blog/wls/jmx/client/JMXClient.java

Executing the code requires to add WebLogic client side classes to the execution ClassPath. We need to include the WebLogic JMX connector client side classes. Remember we specified those in the environment properties passed to the JMXConnectorFactory. Below is the command I used:

java -classpath .:$WL_HOME/server/lib/wljmxclient.jar blog/wls/jmx/client/JMXClient

WL_HOME points to the WebLogic binary install.

The above uses WebLogic's thin client classes. In some cases when you access server side classes, or you want to take full advantage of the t3 protocol, you are required to use WebLogic's thick client classes. To do so, you first need to build wlfullclient.jar as follow:

cd $WL_HOME/server/lib
java -jar wljarbuilder.jar

You can then execute the JMX Client as follow:

java -classpath .:$WL_HOME/server/lib/wlfullclient.jar  blog/wls/jmx/client/JMXClient

You can also bundle wlfulclient.jar with your client code. When using the thin client you cannot just bundle wljmxclient.jar with you client code, as this jar references other jars as part of its manifest Class-Path entry. At the time of this writing, you also need to bundle wlclient.jar

At this point we are now able to connect to WebLogic's "Runtime" MBeanServer. Each WebLogic process contains a "Runtime" MBeanServer in which local WebLogic MBeans and user-defined application MBeans are registered. WebLogic also offers two other MBeanServers that are only available from the Domain "AdminServer" process:

  • The "Domain Runtime" MBeanServer. It aggregates the MBeans registered on the domain's "Runtime" MBeanServers. So as long as a managed server is up, its MBeans can be accessed through the "Domain Runtime" MBeanServer. To connect to that MBeanServer, just use the following JMX URI:

    service:jmx:iiop://127.0.0.1:7002/jndi/
    weblogic.management.mbeanservers.domainruntime
    

    "7002" is the iiop port for my "AdminServer" install. Replace with your AdminServer's port as previously explained. Remember; only the AdminServer process runs the "Domain Runtime" MBeanServer.

  • The "Edit" MBeanServer. It contains the WebLogic Config MBeans that are used to configure the Domain. To connect to that MBeanServer, just use the following JMX URI:

    service:jmx:iiop://127.0.0.1:7002/jndi/weblogic.management.mbeanservers.edit 
    

    "7002" is the iiop port for my "AdminServer" install. Replace with your AdminServer's port as previously explained. Remember; only the AdminServer process runs the "Domain Runtime" MBeanServer.

Note: One can also use "t3" or "rmi" as protocol in place of "iiop"

The rest of the client's code remains unchanged. Only the JMX URI needs to be changed to connect to different MBeanServers.