Work Manager Leash for Slow JSPs in WebLogic Server
By james.bayer on Jan 13, 2010
If part of your application goes bad, you do not want issues to cascade to the rest of your application or the rest of the server. Based on my recent discovery of Michael Nygard’s architecture writings, this post will be somewhat applicable to his Bulkhead pattern (see page 39) of his excellent presentation. I’ve been told that his ReleaseIt! book is phenomenal by peers I respect, so it’s already on it’s way to my bookshelf. WebLogic has a way to put a leash on a JSP with only a few minor deployment descriptor edits – read on for an example on how to not let one resource intensive JSP bring you down.
Actual Customer Problem
One of my customers has an interesting problem with a long-running JSP. They use a 3rd party java library exposed via a JSP that brings back large amounts of content from a backend file system. This can take tens of minutes to process each request. Within the same application (WAR file) they also have many short-lived requests hitting normal JSPs like a traditional web application. In some cases they were having stability problems when too many threads were busy (STUCK threads) on the long-running JSP, taking up lots of server memory and starving threads from the fast JSPs. In some cases, they would have to restart the WLS instance to recover stability. The reason - by default the application components and even the rest of WebLogic Server are all using threads from the same default Work Manager – which is used by WebLogic Server’s self-tuning thread pool.
Potential Solution Using Custom Work Manager and Constraint
In 9.x and above of WebLogic, you have the ability to use a WorkManager with different types of constraints to control the shared threads in WebLogic Server. Here is a detailed article on Work Managers on OTN.
Work Managers can be used to put some constraints on a troublesome JSP such as the max number of threads and the max number of requests that are being serviced and queued (called a capacity constraint) simply by editing the web.xml and weblogic.xml deployment descriptors. Requests over the capacity constraint will result in a 503 error code. Using web.xml a special page can be shown in this case that provides a helpful message as to why this condition exists instead of the default 503 page and the application can stay up while limited access to the JSPs that hog server resources.
2 JSPs - normal.jsp and veryslow.jsp - the veryslow.jsp has a 30 second java.lang.Thread.sleep()
A simple Filter that applies to both JSPs that simply logs to standard out – This is included just to show that Filters are not affected by Work Managers – you don’t have to worry about a Russian Nested Doll issue to target a work manager successfully at a JSP, even if it has Filters.
503error.html page (for the informational message in the overload condition). I included this to show you that this can be a custom pretty page and does not have to be the ugly Internal Server Error page that you see by default.
weblogic.xml with a custom Work Manager defined in with constraints:
web.xml – this simply shows how you can target a custom Work Manager directly at a JSP/Servlet – It is probably more common to target a Work Manager at an entire web application – but we have a very targeted use case in this example. In EJB modules, work managers are targeted per EJB or MDB. The important part is looking at the init-param of the veryslow.jsp. The 503 error code page is also defined here.
Here’s an illustration with a max thread constraint of 5 and a capacity constraint of 7 showing what will happen when 10 requests hit veryslow.jsp at the same time.
Here is a JMeter report using 10 threads instantaneously hitting normal.jsp which uses the default Work Manager from WLS. Notice that all finish executing right away which is exactly as we expect because there are no relevant constraints in the default Work Manager.
Here is a JMeter report showing 10 requests instantaneously hitting veryslow.jsp which uses the custom Work Manager with the constraints discussed before. In this case we see 3 rejected requests (the last 3 in), 5 that complete in 30 seconds, and 2 that complete in 60 seconds. This is exactly what we expect given our constraints since 2 threads will wait not only for their 30 second sleep, but also for the 30 second sleep for the thread that they are waiting on.
When you go to the browser and hit veryslow.jsp while the veryslow test is being run, you get the 503error.html page defined in web.xml to come up.
Here’s the access.log from the test – notice the 503 codes:
Here’s the Standard Out log – notice that both JSP’s use the Filter and it is not affected by using different work managers for JSPs.
I’ve attached the code for this example that includes both the war file and my Oracle Enterprise Pack for Eclipse project.
As you can see, it’s pretty easy to put constraints on individual JSPs, Servlets, EJB’s, MDB’s, and entire web applications using only a few deployment descriptor edits. That’s a very powerful feature when you need it to keep hogs from bogging down other parts of your application or your WLS instance.