Threading in 10.1.3.4
By Antony Reynolds on Feb 11, 2010
Seems a long time since I wrote anything, between getting sick with Mono (Glandular fever to fellow Brits), Christmas, New Year, and another large project I have been slow to write up anything.
Seems I have been working with a number of customers recently who have problems with or are concerned about BPEL Threading. So in this entry I will give an overview of how threads are allocated by the BPEL engine.
Threads in BPEL are divided into thread pools. This a feature of the JVM that is also used in most Java systems. Typically a thread pool is dedicated to a particular task such as processing invocation messages or running JMS adapters.
The first question to answer when talking about threading in BPEL is about the source of the activity in the BPEL process. Different thread pools are used depending on the source of the activity. Some of these pools are BPEL thread pools, others are application server pools used by BPEL.
If the interface to your BPEL process is a synchronous request/reply interface then you will “steal” the thread from the source of the call. A synchronous request/reply interface is characterized in WSDL by an operation with an <input> and an <output> message. In your BPEL process this translates to a <receive> and a <reply> for the partner link.
Common sources of threads for request/reply activites include
- Application server servlet thread pool for HTTP web service invocations.
- Application server servlet thread pool for invocations from JSP pages and servlets through the Java API.
- EJB thread pool for remote invocations using Java API.
- Adapter thread pool for invocations from adapters.
Activities that Use Additional Threads
Normally a request/reply interaction will all execute in a single thread. However if there are any activities that would cause the BPEL process to be suspended or the client transaction to be suspended then another thread would be used.
If the BPEL process must pause its execution to wait for a long running activity then the original request reply thread will block waiting for the reply to be reached. This allows the user transaction to be preserved. When the reply activity is reached then the invoking thread is notified and it can return the result to the client. Any activities that occur after the invoker thread is suspended will execute in an engine thread as they are not part of the original invoke.
One-Way and Asynchronous Invocations
If the interface to your BPEL process is a one-way or an asynchronous request/reply interface then your BPEL process will receive its request through a message placed on a queue in the context of the caller thread. The same sources for request/reply activities also provide the one way interactions. A one-way interaction is characterized in WSDL by an operation with only an <input> message. In your BPEL process this translates to a <receive> without a <reply> for the partner link. An asynchronous request/reply interface is characterized in WSDL by two port types each with an operation with only an <input> message. The port types provide two different roles. In your BPEL process this translates to a <receive> and an <invoke> for the partner link.
The actual message is stored in the invocation tables in the BPEL schema and a short notification message referring to the message in the table is placed on an in-memory JMS queue. The invoker threads from the invoker thread pool are waiting to receive notifications from this queue. The number of threads a given domain has waiting to read these asynchronous messages is controlled by the dspInvokeThreads property.
If the invoker thread hits an activity that would cause the BPEL process to pause then it will stop working on that process instance and return to waiting on the queue for a new notification message.
Activities that Pause the BPEL Process
The following activities cause a BPEL process instance to be suspended and its state stored in the dehydration store, they also cause the current thread to stop working on this process instance:
- <receive> – any receives other than the one that initiated the process will potentially cause the BPEL engine to wait for a message to arrive.
- <wait> – any waits greater than a small value will be implemented through the quartz timers and the BPEL process will wait for the alarm to trigger before continuing.
- <pick> – waits for one of a number of potential messages to arrive or a timeout to expire. The net result is again that the BPEL process will wait for either the first message to arrive or the timeout to occur.
- <invoke> with nonBlockingInvoke – nonBlockingInvoke allows the thread to find other activities to perform without waiting for the invoke to complete. This is commonly used in a flow statement to allows concurrent execution of synchronous request/reply messages. Under the covers this creates an internal receive to wait for the result of the non blocking invoke to complete. Of course a receive is one of our activities that causes thread execution to change. The actual invoke will execute in a separate thread and when it completes it will notify the BPEL process that it has completed through the internal receive. Hence it is useful to think of the non blocking invoke as an invoke followed by a receive.
Resuming a Paused BPEL Process
A paused BPEL process instance will be woken either by a message arriving or a timer expiring. Messages for paused processes arrive in the same way as any other message, as explained in the request/reply section. In this case the delivering thread places the message in the dispatch table rather than the invoke table and posts a notification to a queue. A different thread pool to the invocation pool is used to consume these messages. The engine threads are listening to this queue and one of them will receive the message and resume executing the process. The number of threads in the pool is controlled by the dspEngineThreads parameter.
The engine thread will continue to execute the process until it reaches another activity that causes the process to be suspended at which point the process state will again be stored in the dehydration store and the thread will return to waiting on the queue.
Additional Thread Pools
In addition to the servlet thread pools, each adapter has its own thread pool which can be tuned in the adapter configuration files. Typically the adapter thread pools sit blocked and waiting for incoming messages, either waking up and polling for database changes or new files, or blocking and waiting on a queue. There is another BPEL thread pool called the system thread pool controlled by dspSystemThreads. This thread pool us used to perform system tasks, and generally doesn’t have too much to do.
Domains, Clusters and Database Connections
The discussion above explains how different threads are used by the BPEL engine. It is worth thinking for a moment about what this means the environment in which the BPEL processes execute.
Each domain has its own dspInvoke and dspEngine thread pools. So if you have multiple domains you could end up with a lot of threads in the thread pool. More threads does not mean that the system will run faster. Generally if the threads in the pool are active it means that they are consuming CPU. Threads executing a BPEL process instance will always consume CPU unless they are blocked in request/reply interaction with another system. This means that if you allow too many threads you may see a lot of context switching occurring between threads. This can result in lower system throughput than could be achieved with a smaller number of threads. Tuning the thread pools is best done empirically based on the system utilization expected in production.
The total number of threads used will be the sum of the dspInvokeThreads, the dspEngineThreads and the dspSystemThreads across all the domains in the BPEL server. All these threads exist in a single JVM, the container running the BPEL process manager.
In a BPEL cluster the same number of threads is used on each node of the cluster, as configured in the domain.xml file for each domain. It is worth remembering that adapters that are active on all nodes in the cluster will be using resources on every machine and this should be factored in to number of resources available in the target systems.
Each thread executing a BPEL process instance, whether a dspInvokeThread, a dspEngineThread, a servlet thread or whatever pool it may come from will require at least a database connection to the dehydration store. This requires that the database connection pool in the application server must at least as big as the sum of the thread pools across all the BPEL domain. In addition it must allow for request/.reply interactions from EJBs and servlets. by default this connection pool is unlimited in OC4J. Even if the database connection pool is unlimited on the application server the database itself does not supported an unlimited number of sessions. The database sessions parameter must be sized at least big enough for all the threads on all the domains on all the nodes to have a connection to the dehydration store. Even for a small cluster this can be hundreds or even thousands of potential sessions.
So the short summary is that request/reply messages execute on the thread in which the message arrived, asynchronous messages are delivered using the thread on which they arrived but are processed with one of thwo thread pools. New process instances start executing using the dspInvokeThreads pool, existing processes are resumed using threads from the dspEngineThreads pool. Of course there are exceptions to what we have spoken about, but basically although properties may change the behavior of threading it all boild down to either being treated as a request/reply message using the arriving thread, or an async message using either the invoke thread pool or the engine thread pool as appropriate.
Hopefully this clarified how threads are used in BPEL 10.1.3.4.