WebLogic Server Performance

I have been wanting to start a blog about issues related to WebLogic Server Performance for some time now, but have always found it hard to get off the ground.  To make this easy I thought that I should start off posting about a tool that I use regularly while trying to identify performance issues in the WLS code base.  

We in the WebLogic Server development team spend a fair amount of time trying to identify parts of the product where we should focus our attention.   The most common way of doing this is to collect a profile of the system while it is running.   There are a number of really good java profiling tools out there that can be used to analyze a particular scenario from end to end while the system is not under load.  JProfiler is one that I really like and that has some really useful capabilities.  The problem with most of these tools is that under load they tend to impact the performance of the application itself and thus have limited usefulness in these scenarios.

The JRockit Runtime Analyzer is a tool that can be used even on a very heavily loaded system with very little impact.  The reason is that the creation of a JRA profile takes advantage of information that is already being maintained by the VM to identify parts of the code to focus on for optimization.   We have actually measured the impact of collecting profiles to be less than 1% in most cases. Because of this we collect JRA profiles during many of our benchmarking runs.

The easiest and least intrusive way that I have found to collect these profiles is to add options to the java command used to start the server instance.   This is particularly useful when running automated benchmarks.  It should be noted that it is also really easy to trigger a JRA profile using the JRockit Mission Control or the newer plugin for Eclipse.  The simple command below will generate a JRA recording 3 minutes after the JVM is started and record for a one minute period.   It also specifies a location into which to place thee file and a stack trace depth of 64.

-XXjra:delay=180,recordingtime=60,filename=./snapshots/jrarecording.xml,tracedepth=64

The default for stack trace depth is only 16, so it is important that you increase that value so that you can actually get down into your own application code.  

What is going on behind the scenes is that the JRockit VM is periodically stopping a certain number of threads and sampling the register that tells it what piece of code is executing.  It also samples the full stack at a somewhat less frequent interval and that is where the stack trace depth come into play. 

For modern applications that are running in an application server container there is often a fair amount of code that sits in front of your application code.   Trimming the stack at 16 will often mean that you are just getting to our own code when the stack is trimmed and thus you may miss the good part.

The resulting profile can be loaded using JRockit Mission Control(jrmc) or the newer JRA plugin to Eclipse.

The screenshot below is of a jrmc session shows the information in the “Methods” tab.   One useful approach that I often use is to filter on “*ExecuteThread.run”.  These are the methods that the WebLogic Server thread scheduling mechanism uses to do application work.  

One interesting thing to note is that there are two of the same method.  The one that has the little lightning bolt has been optimized by the JVM.  This indicates that the method was optimized and replaced during the recording period.  To the JVM these are actually two separate methods.   Note also that these numbers reflect the number of times that the method itself was sampled.  When that element is selected the windows on the right indicate the call tree that both precedes and succeeds that particular method.   The number that is at the top of the successors window indicates the sum total of samples for this method and all of the methods that succeed it in the hierarchy.  For each branch in the tree the percentage of successor samples is tallied so that you can see a break down of where the actual CPU is going.

Working your way down the tree as I have done on the right hand side indicates how much time is spent doing which type of work.   In this case there is a mix of regular web traffic and web services traffic.   From this profile it is easy to see the distribution during this run.   

<click on this image to get a readable version in a new window>

The key to using a tool like this is that you are doing sampling, so it really only makes sense to do this in cases where you can get a big enough statistical sample set.  Trying to use this to analyze one particular scenario that you drive from a web browser is not appropriate.    For such cases you will want to use a different tool that actually tracks every call.   When you are trying to analyze a real production deployment or you want to get some insight into what is going on when your system is under heavy stress,  this is the approach to use.

Well…. I could probably go on for ever about this scenario, but that is about enough for now.  Let me know if you get a chance to try this out and if you find it useful.  I’ll try to provide some interesting examples of what I learn using this approach in subsequent postings.  In the meantime there is lots of good documentation on how to use JRA and I’ll be glad to answer any questions that you come up with…