WebLogic Server Performance

There was a pretty interesting post to one of our internal mailing lists regarding an issue that a customer was seeing when trying to scale up their application.   The application architecture was a fairly common one in which the WebLogic Server was accepting batch type requests via a JMS queue.  The requests were then processed by an MDB.  The system is backed by a really substantial RAC instance and the team was hoping to see the system processing lots and lots of requests concurrently.   The test driver that they created was generating on the order of 100 concurrent requests, yet when they looked at the server they found that at most there were 16 threads being created.    

Initially the team assumed that they were somehow being constrained by the number of jdbc connections and tried increasing the number of connections in the pool.  They found that this did not get them any increased concurrency.  Next they increased the max beans in pool for the MDB, but still this did not help either.   

What is happening in this case is that the WebLogic Server MDB container is trying to protect against externally generated asynchronous work overwhelming the server.  It helps to consider that such batch requests are frequently long running and may consume considerable resources.   As the default for max beans in pool is 1000 it is possible to quickly overwhelm a server with a large number of concurrent requests and monopolize the servers resources.

In response to many such issues the decision was made by the EJB container team to limit the number JMS consumers to 16.  This has the effect of constraining the number of concurrent requests that are dispatch to 16.  The mechanism that is provided to remove this artificial limit is to configure a separate work manager and associated explicitly with that MDB.  In doing so you are telling the container that you are aware of the issue and are taking control of the situation.

The Tuning Message-Driven Beans chapter of the Performance & Tuning book describes MDB concurrency in detail and explains all about this issue and describes lots of options.   To get a better understanding of work managers and how they can be used to balance the work load in a server there is an excellent white paper entitled Workload Management in WebLogic Server.

The following example should get you started and can be done completely in the context of the weblogic-ejb-jar.xml.  

<?xml version="1.0"?>
<weblogic-ejb-jar xmlns=http://www.bea.com/ns/weblogic/10.0 
                  xmlns:j2ee=http://java.sun.com/xml/ns/j2ee
                  xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
                  xsi:schemaLocation="http://www.bea.com/ns/weblogic/10.0
                                                     http://www.bea.com/ns/weblogic/950/weblogic-ejb-jar.xsd">

  ...

  <weblogic-enterprise-bean>
     <ejb-name>SomeMDB</ejb-name>
     <message-driven-descriptor>
       <destination-jndi-name>jms/SomeQueue</destination-jndi-name>
     </message-driven-descriptor>
     <jndi-name>SomeMDB</jndi-name>
     <dispatch-policy>mdb_workmanager</dispatch-policy>
   </weblogic-enterprise-bean>

  ...

  <work-manager>
    <name>mdb_workmanager</name>
    <max-threads-constraint>
       <name>maxthreads_32</name>
       <count>32</count>
    </max-threads-constraint>
  </work-manager>

</weblogic-ejb-jar> 

With this work manager configuration  in place the server will always make sure that if there are no more than 32 threads dedicated to processing requests to this MDB.  

Over two years ago I had the pleasure to make the acquaintance of making the Samuel Kounev.  I had recently been given the opportunity to define the role of Performance Lead for the WebLogic Server Development team and I was looking beyond the usual sources for information on performance modeling of JEE.  It was then that I stumbled across Samuel’s work in a search of the web.   I exchanged emails and then later learned connected to Samuel through some of my work with the SPEC benchmarks with which he was also involved.   BEA had previously supported some of his research by providing access to technology and I was interested in continuing to foster what I believed to be some truly novel approaches to performance modeling.

One year and one acquisition later we had worked out the details by which Oracle could provide support for this research.   Fabian Brosig, a diploma student, took up the torch and what followed was a fruitful year of collaboration. The result of this work was a degree for Fabian, some really good feedback into the WLS diagnostics capabilities and really interesting publication for the OTN.

Besides providing insight into the issue of performance modeling it also serves as a practical demonstration of how to use the WebLogic Diagnostics framework. 

Thanks again to Samuel and Fabian for all of their diligence and to the WLDF team for their support.

A little know feature of the WebLogic Server implementation is something called the XML Registry.  In many applications it is still common to receive an XML document as a post to a servlet and then parse that document and translate the contents into calls to various local services.  This approach was a common early step in the movement towards web services and there are are still plenty of applications out there with similar logic.  

In the WebLogic Server the XMLRegistry interposes a server specific implementation of the various SAX and DOM parsers and the services used to resolve the entities that are encountered during parsing.   As part of optimizing the server for optimal benchmark performance we spent a bunch of cycles during the 10.3 release to improve the efficiency of this code and to eliminate a number of areas of contention.  We also noticed that the default parsers were repeatedly downloading the dtd for the documents that were being parsed.   By configuring the XML registry we were able to dramatically reduce the number of outbound requests to download the dtds during our testing.  

To see what is going on lets look at a diagram of a somewhat typical application.  In this case we have two applications which are exchanging order and delivery documents each of which is validated with a DTD.  It can be seen that the way the application was written each request is parsed in the servlet. While it is parsed the underlying XML parser retrieves the DTD that is needed to validate it and thus blocked.  This ties up a thread on both the sending and receiving application servers.

By enabling the XMLRegistry the request to retrieve the required entity is trapped in the XML Registry and resolved against a local cache.  This avoids repeated calls between servers and eliminates the latency in the initial processing and also in the session that is making the initial post.   This reduces the number of active threads on both servers that are involved in the processing.

This works because it in most cases the contents of the DTD are very, very unlikely to change.  Any change would be well coordinated between the communicating parties.  The downside to such an approach is that if the DTD or XSD were to change there could be any number of failed requests.  Handling that would require special coding in the applications   In the 10.3 release additional functionality was added to address even this unlikely occurrence (http://e-docs.bea.com/wls/docs103/issues/known_resolved.html).

The “handle-entity-invalidation” feature that was added to the XML Registry tells the interposed parser handler to trap any parsing failures and first verify if they are the result of changes to the XML entity that is being used to validate the document.  When it encounters these errors it will download a fresh copy of the entity and compare it to the entity in the cache.  If it is changed it will reinitiate the parse.  If that parse is successful the application will continue on as if nothing happened and be none the wiser.   Subsequent failure are reported in the standard way.

Enabling this set of features is a great way to get better performance out of existing applications without having to do any code rework. The example code below shows how to configure this capability in the WLS config.xml  It can also be configured through WLST scripting.  The only trick is to make sure that the system-id and public-id match the values that are in the documents that you are receiving from the counterparty. 

Another little tweak that might help performance some even if you don’t enable entity caching, is replacing the default document builder factory.  We found that the implementation below was working better than the default implementation of  “weblogic.apache.xerces.jaxp.DocumentBuilderFactoryImpl” specified in releases prior to 10.1.  In 10.3 this was updated and so you won’t need to make this change to get the performance benefit.

<xml-registry>
<name>dtdCache</name>
 <document-builder-factory>
   com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
</document-builder-factory>
<xml-entity-spec-registry-entry>
   <name>deliveryResponse</name>
   <public-id>DELIVERY</public-id>
   <system-id>
      http://acme:7011/Emulator/dtd/delivery.dtd
   </system-id>
   <entity-uri>
     http://acme:7011/Emulator/dtd/delivery.dtd
   </entity-uri>
   <when-to-cache>cache-on-reference</when-to-cache>
   <cache-timeout-interval>60000</cache-timeout-interval>
   <handle-entity-invalidation>true</handle-entity-invalidation>
</xml-entity-spec-registry-entry>
</xml-registry>

While not strictly related to performance I did come across an interesting tidbit today that I think is worth sharing.   I have been working with JPA based application that has a persistence.xml that is set up to provide the person deploying the application with some flexibility when defining the data sources.  

<persistence xmlns="http://java.sun.com/xml/ns/persistence" version="1.0">
  <persistence-unit name="Order">
    <jta-data-source>jdbc/OrderDS</jta-data-source>
    <class>foo.bar.orders.Customer</class>
    <class>foo.bar.orders.Order</class>
    <class>foo.bar.orders.OrderLine</class>
  </persistence-unit>
  <persistence-unit name="Mfg">
    <jta-data-source>jdbc/MfgDS</jta-data-source>
    <class>foo.bar.mfg.Assembly</class>
    <class>foo.bar.mfg.Component</class>
    <class>foo.bar.mfg.Inventory</class>
    <class>foo.bar.mfg.WorkOrder</class>
  </persistence-unit>
</persistence>

The first reaction when confronted with this descriptor might be to deploy multiple connection pools one for each persistence unit.  This may even be appropriate in some cases because if could be that you are actually going to different back end systems. 

If you happen to know that you are actually hitting the same back end database you may think that your only choice is to modify the persistence.xml to point them to a common data source.  Fortunately, WLS provides the flexibility of providing multiple jndi names for each data source.  

<jdbc-data-source xmlns="http://www.bea.com/ns/weblogic/90" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <name>specjds-jdbc</name>
  <jdbc-driver-params>
    <url>@DATABASE_URI@</url>
    <driver-name>@DATABASE_DRIVER@</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>@DATABASE_USER@</value>
      </property>
      <property>
        <name>password</name>
        <value>@DATABASE_PASSWORD@</value>
      </property>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <initial-capacity>32</initial-capacity>
    <max-capacity>64</max-capacity>
    <capacity-increment>4</capacity-increment>
    <statement-cache-size>128</statement-cache-size>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbc/OrderDS</jndi-name>
    <jndi-name>jdbc/MfgDS</jndi-name>
  </jdbc-data-source-params>
</jdbc-data-source>

This is a handy way of mapping from JPA onto your database configuration without having to perform any manipulation of the descriptors that are embedded inside the application archive.

I have been wanting to start a blog about issues related to WebLogic Server Performance for some time now, but have always found it hard to get off the ground.  To make this easy I thought that I should start off posting about a tool that I use regularly while trying to identify performance issues in the WLS code base.  

We in the WebLogic Server development team spend a fair amount of time trying to identify parts of the product where we should focus our attention.   The most common way of doing this is to collect a profile of the system while it is running.   There are a number of really good java profiling tools out there that can be used to analyze a particular scenario from end to end while the system is not under load.  JProfiler is one that I really like and that has some really useful capabilities.  The problem with most of these tools is that under load they tend to impact the performance of the application itself and thus have limited usefulness in these scenarios.

The JRockit Runtime Analyzer is a tool that can be used even on a very heavily loaded system with very little impact.  The reason is that the creation of a JRA profile takes advantage of information that is already being maintained by the VM to identify parts of the code to focus on for optimization.   We have actually measured the impact of collecting profiles to be less than 1% in most cases. Because of this we collect JRA profiles during many of our benchmarking runs.

The easiest and least intrusive way that I have found to collect these profiles is to add options to the java command used to start the server instance.   This is particularly useful when running automated benchmarks.  It should be noted that it is also really easy to trigger a JRA profile using the JRockit Mission Control or the newer plugin for Eclipse.  The simple command below will generate a JRA recording 3 minutes after the JVM is started and record for a one minute period.   It also specifies a location into which to place thee file and a stack trace depth of 64.

-XXjra:delay=180,recordingtime=60,filename=./snapshots/jrarecording.xml,tracedepth=64

The default for stack trace depth is only 16, so it is important that you increase that value so that you can actually get down into your own application code.  

What is going on behind the scenes is that the JRockit VM is periodically stopping a certain number of threads and sampling the register that tells it what piece of code is executing.  It also samples the full stack at a somewhat less frequent interval and that is where the stack trace depth come into play. 

For modern applications that are running in an application server container there is often a fair amount of code that sits in front of your application code.   Trimming the stack at 16 will often mean that you are just getting to our own code when the stack is trimmed and thus you may miss the good part.

The resulting profile can be loaded using JRockit Mission Control(jrmc) or the newer JRA plugin to Eclipse.

The screenshot below is of a jrmc session shows the information in the “Methods” tab.   One useful approach that I often use is to filter on “*ExecuteThread.run”.  These are the methods that the WebLogic Server thread scheduling mechanism uses to do application work.  

One interesting thing to note is that there are two of the same method.  The one that has the little lightning bolt has been optimized by the JVM.  This indicates that the method was optimized and replaced during the recording period.  To the JVM these are actually two separate methods.   Note also that these numbers reflect the number of times that the method itself was sampled.  When that element is selected the windows on the right indicate the call tree that both precedes and succeeds that particular method.   The number that is at the top of the successors window indicates the sum total of samples for this method and all of the methods that succeed it in the hierarchy.  For each branch in the tree the percentage of successor samples is tallied so that you can see a break down of where the actual CPU is going.

Working your way down the tree as I have done on the right hand side indicates how much time is spent doing which type of work.   In this case there is a mix of regular web traffic and web services traffic.   From this profile it is easy to see the distribution during this run.   

<click on this image to get a readable version in a new window>

The key to using a tool like this is that you are doing sampling, so it really only makes sense to do this in cases where you can get a big enough statistical sample set.  Trying to use this to analyze one particular scenario that you drive from a web browser is not appropriate.    For such cases you will want to use a different tool that actually tracks every call.   When you are trying to analyze a real production deployment or you want to get some insight into what is going on when your system is under heavy stress,  this is the approach to use.

Well…. I could probably go on for ever about this scenario, but that is about enough for now.  Let me know if you get a chance to try this out and if you find it useful.  I’ll try to provide some interesting examples of what I learn using this approach in subsequent postings.  In the meantime there is lots of good documentation on how to use JRA and I’ll be glad to answer any questions that you come up with…