GlassFish Tuning - HTTP Thread Pool

In the white paper, Optimize GlassFish Performance in a Production Environment, my colleague Kim Lichong discusses eleven tips for improving the performance of GlassFish app server. This paper is a great starting point for anyone interested in tuning the performance of GlassFish application server.

Once the basic tunings are applied, the next step is to monitor the performance of the server instance and modify the various configuration parameters for additional performance gains. The key to this dynamic tuning is to understand what to monitor as well as what the different numbers spit out by the monitoring framework mean. In this and the next few blogs, I will look at some of the monitorable values and how to tune it for improved performance. In this entry, I discuss the monitoring of HTTP thread pools.

In case you are not familiar with GlassFish monitoring, it is recommended that you read Masoud Kalali's blog on GF V2 monitoring. Additonally, Monitoring Components and Services chapter of the Administration Guide provides details on how to enable/disable monitoring for the various components and a brief description of the various monitorable values.

Performance Tip #6 in the white paper talks about setting the request processing threads to either equal to or double the number of CPUs in the system. If  the application is able to fully utilize the CPUs and achieve maximum throughput with these values, then no further tuning is required. If on the other hand, there are CPU resources available, the application may be able to achieve higher throughput or be able to support larger number of concurrent users if you increase the number of request processing thread pools.

Once a thread starts processing a HTTP request, that thread is used to execute all the application logic until the response is committed. Any I/O interaction that is part of the application (examples: invocation of remote EJBs, database interactions, communicating with slow clients, file system interactions) can cause the thread to be in a I/O wait state, thereby making the CPU resources available for other threads to run. For such applications, configuring the thread-pool with too few threads can cause requests to be queued for processing while CPU resources are available. By monitoring the request processing thread pool as well as CPU utilization, a determination can be made if the pool size needs to be increased. Before proceeding with this tuning, lock contention and/or resource contention in other parts of the system (eg. all threads waiting on a slow disk I/O) which can cause poor CPU utilization should be identified and eliminated.

The request processing pool statistics are available under http-service.http-listener-X (or whatever the listener name is). In the default case, the values for HTTP and HTTPS request processing pools can be found under server.http-service.server.http-listener-1 and server.http-service.server.http-listener-2, respectively. The pool utilization can be understood by monitoring the currentthreadsbusy-count attribute which shows the status of the server at the time of statistics collection. The other attribute of interest, maxthreads-count is a static value based on the configuration setting.

The following table shows the different values to be monitored and the associated tuning hints.

 Attribute  Description  Comments
Maximum number of threads allowed in the thread pool
Size this value based on the CPU utilization and currentthreadsbusy-count.  Increase the value if all the threads are being used on a consistent basis and CPU resources are still available. Setting the pool size to an excessive high value can have detrimental performance effects due to increased context switches, cache misses etc.
Number of request processing threads currently in use in the listener thread pool serving requests.
If this value is consistently equal to the maxthreads-count, it signifies that there is enough load on the system to keep the request processing pool fully utilized. A value lower than maxthreads-count means that the server has the capacity to handle additional load.

The currentthreadsbusy-count value is a good indicator of the amount of work being done in the server. I sometimes use this for debugging abnormal benchmark results. Monitoring this value during a benchmark run is a good sanity check to verify that there is enough load on the system. This may help identify benchmark client and network issues.

 The following example is for GlassFish V2.

To enable monitoring of http-service.

# ./asadmin set server.monitoring-service.module-monitoring-levels.http-service=LOW

To monitor the three values listed above use asadmin get -m command. Use --interval argument to specify how often to collect the values.

#./asadmin get -m --iterations 10 --interval 5  server.http-service.server.http-listener-1.currentthreadcount-count server.http-service.server.http-listener-1.currentthreadsbusy-count

server.http-service.server.http-listener-1.currentthreadcount-count = 8
server.http-service.server.http-listener-1.currentthreadsbusy-count = 8

server.http-service.server.http-listener-1.currentthreadcount-count = 8
server.http-service.server.http-listener-1.currentthreadsbusy-count = 8 


It is clear that all the request processing threads are being utilized. How about the CPU utilization?

# vmstat 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 s1 s2   in   sy   cs us sy id
 0 0 0 26089640 13978972 152 633 0 0 0 0 0  0  0 16  0 2863 8516 2834 47  3 50
 0 0 0 26030828 13960068 0 12 0 0  0  0  0  0  0  0  0 2731 6467 2744 43  3 54

There are plenty of CPU resources available (refer to the last column, id in the vmstat output). We should be able to process more requests simultaneously. Try increasing the number of processing threads.

# ./asadmin set server.http-service.request-processing.thread-count=16
server.http-service.request-processing.thread-count = 16

Even though you can set the thread-cout value dynamically, it looks like you need to restart the instance for it to take effect (I could not get the dynamic resize to work correctly, more investigation is required).

After changing the values, the new data looks like this -

#./asadmin get -m --iterations 10 --interval 5 server.http-service.server.http-listener-1.currentthreadcount-count server.http-service.server.http-listener-1.currentthreadsbusy-count

server.http-service.server.http-listener-1.currentthreadcount-count = 16
server.http-service.server.http-listener-1.currentthreadsbusy-count = 16

server.http-service.server.http-listener-1.currentthreadcount-count = 16
server.http-service.server.http-listener-1.currentthreadsbusy-count = 16


As before, all the threads are fully utilized. How about CPU usage?

# vmstat 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 s1 s2   in   sy   cs us sy id
 9 0 0 26030504 14024240 0 109 0 0 0  0  0  0  0  0  0 2787 8436 2402 96  3  1
 12 0 0 26029700 13988508 0 27 0 0 0  0  0  0  0  0  0 2474 7870 2345 95  3  2

Good, we have managed to push the server to its maximum capacity. Is this the most optimal value for this? You may need to run iterative tests with different thread-count values to be absolutely sure. However, for my tests (on a 4-CPU server) this seems to be a reasonable value, so I plan to forgo the expense of the additional tests. One important note - don't set this to an excessively high value since this may cause performance degradation due to increased context switches, cache misses etc. If there is a need to set this to very high values (eg: > 64 for a 4 CPU machine), it means that there are performance issues in other parts of your system that needs to get sorted out.

In the next blog, I'll discuss how to monitor HTTP connection queue and keep alive values.


Post a Comment:
Comments are closed for this entry.



« July 2016