Wednesday Jan 30, 2013

OSB Performance Tuning - RouterRuntimeCache

Many customers start out with smaller projects for an initial release.  Typically, these applications require 20-30 Proxy services.  But as time goes on and later phases of the project rollout, the number of proxy services can increase drastically.  The RouterRuntimeCache is a cache implemented by OSB to improve performance by eliminating or reducing the amount of time spent on compiling the proxy pipeline. 

By default, OSB will not compile a pipeline until a request message for a given service is received.  Once it has been compiled, the pipeline is cached in memory for re-use.  You have probably noticed in testing that the first request to a service takes longer to respond than subsequent requests, and this is a big part of the reason.  Since free heap space is often at a premium, this cache can not be infinite in size so this cache has a built in limit.  When the cache is full, the least recently used entry is released and the pipeline that is currently being requested is placed into cache in its place.  The next time a request comes in for the service who's pipeline was released, that pipeline has to be re-compiled and placed in cache, again forcing out the least recently used pipeline.  Once a pipeline is placed in cache it is never removed from cache unless forced out by a full cache scenario as above, or if the service is updated, forcing it to be recompiled.

The default size limit of the RouterRuntimeCache is 100 entries (or pipelines).  It is limited by the number of services in the cache, not the memory used by the cache so the amount of memory used by a full cache will vary greatly based on the complexity of the services, the extent and complexity of inline xquery, etc.  If your project grows beyond 100 proxy services, system performance can degrade significantly if the cache size is not increased to hold all frequently used services. 

Unfortunately, the way to tune this cache is not exposed through the OSB console.  As of 11g PS5, the only way to set this parameter is via a system property specified on the Java command-line.  The property name is   For example,

“java … … weblogic.Server …”. 

In this example, OSB will cache 500 proxies, instead of the default 100.  Because increasing the RouterRuntimeCache.size value will require more space in the heap to hold the additional proxies, be aware that you may need to reevaluate your JVM memory settings to allow OSB to continue to perform optimally.

Tuesday Dec 04, 2012

Using BPEL Performance Statistics to Diagnose Performance Bottlenecks

Tuning performance of Oracle SOA 11G applications could be challenging. Because SOA is a platform for you to build composite applications that connect many applications and "services", when the overall performance is slow, the bottlenecks could be anywhere in the system: the applications/services that SOA connects to, the infrastructure database, or the SOA server itself.How to quickly identify the bottleneck becomes crucial in tuning the overall performance.

Fortunately, the BPEL engine in Oracle SOA 11G (and 10G, for that matter) collects BPEL Engine Performance Statistics, which show the latencies of low level BPEL engine activities. The BPEL engine performance statistics can make it a bit easier for you to identify the performance bottleneck.

Although the BPEL engine performance statistics are always available, the access to and interpretation of them are somewhat obscure in the early and current (PS5) 11G versions.

This blog attempts to offer instructions that help you to enable, retrieve and interpret the performance statistics, before the future versions provides a more pleasant user experience.

Overview of BPEL Engine Performance Statistics 

SOA BPEL has a feature of collecting some performance statistics and store them in memory.

One MBean attribute, StatLastN, configures the size of the memory buffer to store the statistics. This memory buffer is a "moving window", in a way that old statistics will be flushed out by the new if the amount of data exceeds the buffer size. Since the buffer size is limited by StatLastN, impacts of statistics collection on performance is minimal. By default StatLastN=-1, which means no collection of performance data.

Once the statistics are collected in the memory buffer, they can be retrieved via another MBean[Server Name],name=BPELEngine,type=BPELEngine.>

My friend in Oracle SOA development wrote this simple 'bpelstat' web app that looks up and retrieves the performance data from the MBean and displays it in a human readable form. It does not have beautiful UI but it is fairly useful.

Although in Oracle SOA onwards the same statistics can be viewed via a more elegant UI under "request break down" at EM -> SOA Infrastructure -> Service Engines -> BPEL -> Statistics, some unsophisticated minds like mine may still prefer the simplicity of the 'bpelstat' JSP. One thing that simple JSP does do well is that you can save the page and send it to someone to further analyze

Follows are the instructions of how to install and invoke the BPEL statistic JSP. My friend in SOA Development will soon blog about interpreting the statistics. Stay tuned.

Step1: Enable BPEL Engine Statistics for Each SOA Servers via Enterprise Manager

First st you need to set the StatLastN to some number as a way to enable the collection of BPEL Engine Performance Statistics

  • EM Console -> soa-infra(Server Name) -> SOA Infrastructure -> SOA Administration -> BPEL Properties
  • Click on "More BPEL Configuration Properties"
  • Click on attribute "StatLastN", set its value to some integer number. Typically you want to set it 1000 or more.

Step 2: Download and Deploy bpelstat.war File to Admin Server,

Note: the WAR file contains a JSP that does NOT have any security restriction. You do NOT want to keep in your production server for a long time as it is a security hazard. Deactivate the war once you are done.
  • Download the bpelstat.war to your local PC
  • At WebLogic Console, Go to Deployments -> Install
  • Click on the "upload your file(s)"
  • Click the "Browse" button to upload the deployment to Admin Server
  • Accept the uploaded file as the path, click next
  • Check the default option "Install this deployment as an application"
  • Check "AdminServer" as the target server
  • Finish the rest of the deployment with default settings

  • Console -> Deployments
  • Check the box next to "bpelstat" application
  • Click on the "Start" button. It will change the state of the app from "prepared" to "active"

Step 3: Invoke the BPEL Statistic Tool

  • The BPELStat tool merely call the MBean of BPEL server and collects and display the in-memory performance statics. You usually want to do that after some peak loads.
  • Go to http://<admin-server-host>:<admin-server-port>/bpelstat
  • Enter the correct admin hostname, port, username and password
  • Enter the SOA Server Name from which you want to collect the performance statistics. For example, SOA_MS1, etc.
  • Click Submit
  • Keep doing the same for all SOA servers.

Step 3: Interpret the BPEL Engine Statistics

You will see a few categories of BPEL Statistics from the JSP Page.

First it starts with the overall latency of BPEL processes, grouped by synchronous and asynchronous processes. Then it provides the further break down of the measurements through the life time of a BPEL request, which is called the "request break down".

1. Overall latency of BPEL processes

The top of the page shows that the elapse time of executing the synchronous process TestSyncBPELProcess from the composite TestComposite averages at about 1543.21ms, while the elapse time of executing the asynchronous process TestAsyncBPELProcess from the composite TestComposite2 averages at about 1765.43ms. The maximum and minimum latency were also shown.

Synchronous process statistics
    <stats key="default/TestComposite!2.0.2-ScopedJMSOSB*soa_bfba2527-a9ba-41a7-95c5-87e49c32f4ff/TestSyncBPELProcess" min="1234" max="4567" average="1543.21" count="1000">

Asynchronous process statistics
    <stats key="default/TestComposite2!2.0.2-ScopedJMSOSB*soa_bfba2527-a9ba-41a7-95c5-87e49c32f4ff/TestAsyncBPELProcess" min="2234" max="3234" average="1765.43" count="1000">

2. Request break down

Under the overall latency categorized by synchronous and asynchronous processes is the "Request breakdown". Organized by statistic keys, the Request breakdown gives finer grain performance statistics through the life time of the BPEL requests.It uses indention to show the hierarchy of the statistics.

Request breakdown
    <stats key="eng-composite-request" min="0" max="0" average="0.0" count="0">
        <stats key="eng-single-request" min="22" max="606" average="258.43" count="277">
            <stats key="populate-context" min="0" max="0" average="0.0" count="248">

Please note that in SOA, the statistics under Request breakdown is aggregated together cross all the BPEL processes based on statistic keys. It does not differentiate between BPEL processes. If two BPEL processes happen to have the statistic that share same statistic key, the statistics from two BPEL processes will be aggregated together. Keep this in mind when we go through more details below.

2.1 BPEL process activity latencies

A very useful measurement in the Request Breakdown is the performance statistics of the BPEL activities you put in your BPEL processes: Assign, Invoke, Receive, etc. The names of the measurement in the JSP page directly come from the names to assign to each BPEL activity. These measurements are under the statistic key "actual-perform"

Example 1: 
Follows is the measurement for BPEL activity "AssignInvokeCreditProvider_Input", which looks like the Assign activity in a BPEL process that assign an input variable before passing it to the invocation:

                               <stats key="AssignInvokeCreditProvider_Input" min="1" max="8" average="1.9" count="153">
                                    <stats key="sensor-send-activity-data" min="0" max="1" average="0.0" count="306">
                                    <stats key="sensor-send-variable-data" min="0" max="0" average="0.0" count="153">
                                    <stats key="monitor-send-activity-data" min="0" max="0" average="0.0" count="306">

Note: because as previously mentioned that the statistics cross all BPEL processes are aggregated together based on statistic keys, if two BPEL processes happen to name their Invoke activity the same name, they will show up at one measurement (i.e. statistic key).

Example 2:
Follows is the measurement of BPEL activity called "InvokeCreditProvider". You can not only see that by average it takes 3.31ms to finish this call (pretty fast) but also you can see from the further break down that most of this 3.31 ms was spent on the "invoke-service". 

                                <stats key="InvokeCreditProvider" min="1" max="13" average="3.31" count="153">
                                    <stats key="initiate-correlation-set-again" min="0" max="0" average="0.0" count="153">
                                    <stats key="invoke-service" min="1" max="13" average="3.08" count="153">
                                        <stats key="prep-call" min="0" max="1" average="0.04" count="153">
                                    <stats key="initiate-correlation-set" min="0" max="0" average="0.0" count="153">
                                    <stats key="sensor-send-activity-data" min="0" max="0" average="0.0" count="306">
                                    <stats key="sensor-send-variable-data" min="0" max="0" average="0.0" count="153">
                                    <stats key="monitor-send-activity-data" min="0" max="0" average="0.0" count="306">
                                    <stats key="update-audit-trail" min="0" max="2" average="0.03" count="153">

2.2 BPEL engine activity latency

Another type of measurements under Request breakdown are the latencies of underlying system level engine activities. These activities are not directly tied to a particular BPEL process or process activity, but they are critical factors in the overall engine performance. These activities include the latency of saving asynchronous requests to database, and latency of process dehydration.

My friend Malkit Bhasin is working on providing more information on interpreting the statistics on engine activities on his blog ( I will update this blog once the information becomes available.

Update on 2012-10-02: My friend Malkit Bhasin has published the detail interpretation of the BPEL service engine statistics at his blog

Retrieve Performance Data from SOA Infrastructure Database

Here I would like offer examples of some basic SQL queries you can run against the infrastructure database of Oracle SOA Suite 11G to acquire the performance statistics for a given period of time. The final version of the script will prompt for the start and end time of the period of your interest.[Read More]

Thursday May 24, 2012

How to Set JVM Parameters in Oracle SOA 11G

You know you need to tune the JVM and you already know what parameters to set: -Xmx, -verbose:gc ... Plus, there are plenty of WebLogic or Java documentation talking about them. Now you just need to ... set them. But, such a simple task is sometimes confusing.

In your Oracle SOA 11G running within WebLogic, HOW and WHERE to set these parameters? Which files:, or the server-starts in config.xml? And where in the files to set them, as the scripts seem to set and modify the parameter values a lot ...

If you have this question, please see my blog  How to Set JVM Parameters in Oracle SOA 11G

Thursday Feb 09, 2012

Analyzing Thread Dumps in Middleware - Part 4

This posting is the fourth and final section in the series Analyzing Thread Dumps in Middleware

In this post, we will see a new version of TDA (Thread Dump Analysis) tool developed by the author of this blog (Sabha Parameswaran in collaboration with his colleague, Eric Gross, also from Oracle A-Team) and its capabilities as well as some real world samples of thread dump analysis before concluding the series.

Analyzing Thread Dumps - Series:

Part 4: TDA A-Team and real world samples

Part 3: TDA tools

Part 2: How to capture and analyze Thread Dumps, navigate lock chains

Part 1: Basics of Thread states and thread locking

[Read More

Wednesday Feb 08, 2012

Analyzing Thread Dumps in Middleware - Part 3

This posting is the third section in the series Analyzing Thread Dumps in Middleware.

In this post, we will discuss some of the tools that can help with thread dump analysis as well as their limitations. In the next and final section in this series, we will look a new tool developed by the author of this blog in collaboration with his colleague in Oracle to fill the gaps mentioned with some of existing TDA tools in general as well as look at some real world thread dump analysis.

Analyzing Thread Dumps - Series:

Part 3: TDA tools

Part 2: How to capture and analyze Thread Dumps, navigate lock chains

Part 1: Basics of Thread states and thread locking

[Read More

Analyzing Thread Dumps in Middleware - Part 2

This posting is the second section in the series Analyzing Thread Dumps in Middleware

This section details with when and how to capture and analyze thread dumps with special focus on WebLogic Application Server related thread dumps as well as some tips and pointers and what to look for and optimize. The next post will go into tools that can help automate thread dumps analysis, some real world examples and limitations of thread dumps.

Analyzing Thread Dumps - Series:

Part 2: How to capture and analyze Thread Dumps, navigate lock chains

Part 1: Basics of Thread states and thread locking

[Read More

Analyzing Thread Dumps in Middleware - Part 1

How to analyze Thread dumps, for improving Middleware Performance (at App Server or Application level) as well as for general troubleshooting? 

This is part 1 of a series of posts on how to analyze thread dumps. In this post, we will go over basics of thread states and thread locking. In the following posts, we will drill deeper into capturing and analyzing Thread Dumps with special look into WebLogic Application Server specific thread dumps. 

Please see my blog for more details.

Wednesday Feb 01, 2012

Skipping an unnecessary step

Say you have a simple BPEL process that gets messages from JMS queue through a JMSAdapter, transforms the message, and then calls a down stream service via SOAP.

For such a simple BPEL process, are there any optimization that can be done for performance?

The answer is yes. And the low hanging fruits can be picked from layer between the inbound JMSAdapter and BPEL process.

Please read my blog Skipping an unnecessary step at BlogSpot to find out how to optimize the performance for this scenario.

[Read More]

This is the blog for the Oracle FMW Architects team fondly known as the A-Team. The A-Team is the central, technical, outbound team as part of the FMW Development organization working with Oracle's largest and most important customers. We support Oracle Sales, Consulting and Support when deep technical and architectural help is needed from Oracle Development.
Primarily this blog is tailored for SOA issues (BPEL, OSB, BPM, Adapters, CEP, B2B, JCAP)that are encountered by our team. Expect real solutions to customer problems, encountered during customer engagements.
We will highlight best practices, workarounds, architectural discussions, and discuss topics that are relevant in the SOA technical space today.


« March 2015