Thursday Nov 04, 2010

Reverse proxy and custom content

Reverse proxy content overwrite from sed filters
Recently one of the Web Server customer asked about how to overwrite the content of 404 response code coming from origin-server. Customer tried the following :
Service fn="proxy-retrieve"
Error fn="send-error" code="404" path="/tmp/myfile.txt"

However it didn't work. Why? Because when Service saf proxy-retrieve receives any response from origin server, it send the response back to client. So before send-error Error SAF gets executed, response was already complete. So it didn't work the way customer expected. It works if response code is 504 (Gateway timeout) because for 504 proxy-retrieve SAF doesn't send anything to the client.

Web Server supports powerful sed scripts which came to rescue in this case. What user need to do is to insert a sed filter which does the magic. Fortunately sed filter is also very simple.

Following sed script does that (replaceall.sed):
1,1 {
r /tmp/someotherfile.txt
2,$ d

$ cat somefile.txt | sed -f replaceall.sed

Above example will replace content of somefile.txt with someotherfile.txt

In Web Server, I tested this and it worked for me.
<If code eq "404">
Output fn="insert-filter" filter="sed-response" sed="1,1 {"
                                                sed="r /tmp/one.txt"
                                                sed="2,\\$ d"

Above snippet will replace the content of 404 output send by origin server with content of /tmp/one.txt

Monday Aug 31, 2009

Auto tuning of file descriptors in Sun Web Server

Web Server 7.0 uses a algorithm to divide file descriptors to various needs on unix systems.

Here is the list of items which require file descriptors :

1. Web Applications. Web Server leaves 80% file descriptors for web applications.
2. Daemon session threads connections. For each daemon session thread Web
Server 7.0 expects a average of 4 file descriptors. One client socket
connection, requested file, an included file and a backend connection.
3. jdbc pools
4. access log counts for all virtual servers
5. Listener counts
6. file descriptors for file cache
7. keep alive file descriptors.
8. thread pool queue size

In above list (6), (7) and (8) are auto tuned that means if user specify them in server.xml, it uses those, if user doesn't specify then it divides the remaining or available file descriptors to (6), (7) and (8). Available file descriptors = Total descriptors - item (1) to (5)

If available file descriptors are more than 1024 then Web Server uses 1:16:16 ratio for item (6),(7),(8). If available file descriptors are less than 1024 then it uses 1:16:8 ratio. File cache is given the least importance and keep alive is given the highest importance. It also round of the numbers to the power of 2.

Note that Web Server doesn't uses above algorithm for Windows systems. On Windows it uses 64K descriptors for keep alive, 16K descriptors for thread pool queue size.

Thursday Jul 09, 2009

Web Server top "wstop" video

Web Server top "wstop" video Some times back I posted a blog to show how to write a "top" like utility for Sun Web Server using dtrace. Here is the link to you tube video which shows a small demo of wstop.

Wednesday May 20, 2009

Using large pages support in Sun Web Server

  Solaris has a multiple page size support which has been explained here:

I was running specweb ecommerce php benchmark and noticed that 4-8% of gains can be obtained on CMT systems by using 256M pages. When process address space getting bigger and bigger (specially for 64 bit), number of tlb entries grow significantly. I observed that in ecommerce specweb php runs, around 7% of CPU time was spent to handle tlb misses (trapstat output).  When I used 256M pages on CMT systems, I found out that CPU time spent because of tlb misses reduced to 2%. One of the interesting thing I observed that Sun Web Server tends to slow down with time. In my setting, I had set the file cache heap size to 2.5GB. I was wondering why WebServer is slowing down with time. What was really happening, that when Web Server starts, it initally only allocate 256 MB so used process address space was smaller and hence less number of tlb misses.  After some time as file cache starts expanding, more and more tlb misses starts happening. So server sounds sluggish after 20 minutes or so.
So if you are running Sun Web Server on CMT systems, you should try 256M page support.

To enable mpss support in Sun Web Server (64 bit) put the following in

LD_PRELOAD_64=/usr/lib/sparcv9/; export LD_PRELOAD_64     

For 32 bit Web Server :
LD_PRELOAD=/usr/lib/; export LD_PRELOAD

"pagesizes -a" can be used to determine the max page sizes supported on your system. For systems which have only 4M pages, following can be used :
LD_PRELOAD_64=/usr/lib/64/; export LD_PRELOAD_64     

pmap -sx gives the following information :
# pmap -sx 1938
1938:   webservd -d /opt/SUNWwbsvr/https-nsapiphp/config -r /opt/SUNWwbsvr -t
         Address     Kbytes        RSS       Anon     Locked Pgsz Mode   Mapped File
0000000100B00000       6080       6080          -          -    - rwx--    [ heap ]
00000001010F0000     244800     244800     244224          -  64K rwx--    [ heap ]
0000000110000000    3932160    3932160    3932160          - 256M rwx--    [ heap ]
0000000200000000     262144     262144     262144          - 256M rwx--    [ heap ]
FFFFFFFF68000000       4096       4096       4096          -  64K rwx--    [ anon ]

Note that few page sizes are mapped with 256M pages while others are mapped
64K pages.

mpss setting has been used in specweb jsp publications.

Tuesday Aug 12, 2008

Caching static content in reverse proxy in Sun Java System Web Server 7.0

Sun Java System Web Server 6.1 has a function named "check-passsthrough" which will look the resource in reverse proxy instance's docroot first. I was wondering how to do so in 7.0. I didn't find any documentation for that. The advantage of serving static content from reverse proxy docroot is that it is very efficient. Sun Web Server has a excellent caching capabilities which often backend servers don't have.
Basically user need to do the following test :
\* If file exists on local document root then serve the file else go to back-end instance.
Here is how we can do it in WS 7.0.
1) Define docroot variable for each virtual server :
2) Modify the obj.conf to look for local resources before redirecting :
<If not -f "$docroot/$path">
     NameTrans fn="map" from="/" name="reverse-proxy-/" to="http:/"
Web Server 7.0, file cache is enabled by default. So all static content will be cached by default. set-file-cache-prop wadm command can be used to tune the file cache content.

Tuesday Aug 05, 2008

Hacking Sun Java System Web Server pblocks using dtrace.

Hacking Sun Java System Web Server pblocks using dtrace. In my previous blog, I showed how to use NSAPI plugin and dtrace to write several monitoring tools. In this blog I will show that it is even possible to do few things with dtrace without having any NSAPI plugin installed into your web server. Yes that is true, dtrace can be used with stock web server without any configuration changes. Here is my watchpblocks.d dtrace script output (when I send a "/" request to my Web Server):
# dtrace -qs watchpblocks.d  11463
Req->vars : ntrans-base="/var/www" path="/var/www/" required-rights="list" content-length="1912"
Req->reqpb : clf-request="GET / HTTP/1.1" method="GET" protocol="HTTP/1.1" uri="/"
Req->headers : user-agent="curl/7.16.1 (sparc-sun-solaris2.8) libcurl/7.16.1 
                  OpenSSL/0.9.8d zlib/1.2.3 libidn/0.5.19" host="" 
                  accept="\*/\*" content-type="text/html" status="200 OK" 
                  transfer-encoding="chunked" content-length="2003"
11463 is the child process id of my test Web Server. Here is the output of (similar to my previous version
# perl -d 5 11463
12:49:55  Requests: 3     (    0/sec) Bytes: 5736(  1147/sec)
Requests: GETs: 3      POSTs: 0      HEADs: 0      TRACE: 0
Responses: 1xx:0      2xx:3      3xx:0      4xx:0      5xx:0

Requests  Reqests/sec  Bytes Sent  URI
3         0            5736         /
So how does it work. If you look at the dtrace script, you will find :
So at the end of each request webserver calls flex_log method to log the request, at that time above dtrace probe is fired. As with most NSAPI functions Request and Session structure pointers are passed as an argument. dtrace script parses the structure and try to decode the pblocks. For this technique to work users need to have accesslog enabled which is almost always enabled.

The interesting aspect is that we don't need to do any configuration changes.

Caution : Since dtrace doesn't allow "for" loop or if/else logic so the pblock hash decoding is a complete hack. It may not work on all scenarios. Also on busy systems, lots of dtrace probes might be missed using this method. Since we are copying data from kernel land to user land several times.

Previous dtrace version, was much more lightweight as far as dtrace work is concerned. If future version of dtrace provides if/else and loop constuct inside dtrace script then the script could be improved and make it more reliable.

Also, right now these scripts will only work for 32 bit web servers. Here are the scripts :

Monday Jun 16, 2008

Sun Java System Web Server and dtrace.

Sun Java System Web Server and dtrace dtrace is often used to trace kernel and applications. It would be interesing to monitor Web Server using dtrace. Sun Web Server does not contain built in dtrace probes but it is possible to write a small NSAPI plug-in which can be used to monitor Web Server.
In this blog, I will show how to write a small NSAPI plug-in and load the plug-in in to the Web Server and insert a log level hook (AddLog). This NSAPI plug-in will contain a dtrace probe. When plug-in is loaded in to the Web Server, Web Server will register the dtrace probes with kernel. During execution of HTTP request, Web Server will invoke AddLog (at the end of the request) which will fire the dtrace probe. Dtrace probe will provide information about the request and response. Using this probe, it is possible to write "top" like utility for Web Server.
Here is my NSAPI plug-in's code wbdtrace.c
\* Copyright (c) 2005, 2008 Sun Microsystems, Inc. All Rights Reserved.
\* Use is subject to license terms.
#include "nsapi.h"
#include <sys/sdt.h>

int NSAPI_PUBLIC dtrace_log_request(pblock \*pb, Session \*sn, Request \*rq)
DTRACE_PROBE4(sjsws, log__request,
(rq && rq->reqpb) ? pblock_pblock2str((pblock\*) rq->reqpb, NULL) : NULL,
(rq && rq->headers) ? pblock_pblock2str((pblock\*) rq->headers, NULL) : NULL,
(rq && rq->srvhdrs) ? pblock_pblock2str((pblock\*) rq->srvhdrs, NULL) : NULL,
(rq && rq->vars) ? pblock_pblock2str((pblock\*) rq->vars, NULL) : NULL,
Here is the dtrace provider's code sjsws.d
\* Copyright (c) 2005, 2008 Sun Microsystems, Inc. All Rights Reserved.
\* Use is subject to license terms.
provider sjsws {
probe log__request(string, string, string, string);
We now need to compile this plug-in and load into the Web Server. Here is how I compiled the plug-in :
$ cc -c -KPIC -mt -DXP_UNIX -I/opt2/installed/ws70u1/include wbdtrace.c
$ dtrace -G -s sjsws.d wbdtrace.o
$ cc -KPIC -mt -G -L/opt2/installed/ws70u1/lib -lCrun -lCstd -ldl -lposix4 -lns-httpd40 -o wbdtrace.o sjsws.o
Now I copied the plug-in to Web Server's plug-in directory :
$ cp /opt/ws70u1/plug-ins/
Now I loaded the plug-in into the Web Server and inserted a AddLog statement into obj.conf file. Here are the configuration changes which are required for this plug-in to work.
magnus.conf :
Init fn="load-modules" shlib="" shlib_flags="(global|now)" funcs="dtrace_log_request"

obj.conf :
<Object name="default">
AddLog fn="dtrace_log_request"

After making the configuration changes I restarted the Web Server :
$ bin/stopserv; bin/startserv
Web Server is now started and plug-in is loaded. This dtrace NSAPI plug-in will register the dtrace probe (sjsws provider) with kernel. Now let us see how Web Server dtrace probe looks like using "dtrace -l":
# dtrace -l | grep sjsws
53309 sjsws6523 dtrace_log_request log-request
53310 sjsws6524 dtrace_log_request log-request
Note that both primordial (53310) and child process (53309) have registered the probe with Solaris kernel. But since only child process will serve the request so only child process will fire the dtrace probes. Now let us write a small dtrace script (viewreq.d) which just print a message whenever our probe is fired.

@[probefunc] = count();
printf("\\narg0 = %x arg0= %s \\n arg1=%s \\nargs2=%s arg3 = %s\\n",

trace("End of dtrace\\n");

Now I ran this script and sent a HTTP request "GET /" from browser.
# dtrace -s viewreq.d
dtrace: script 'viewreq.d' matched 3 probes
CPU ID FUNCTION:NAME 1 53310 dtrace_log_request:log-request arg0 = 7553c8 arg0= clf-request="GET / HTTP/1.1" ... \^C 0 2 :END End of dtrace
Now whenever message is logged, our dtrace probe is fired. Our dtrace script just printed the various pblocks.

Based on the above dtrace NSAPI plug-in, it is possible to write monitoring code in scripting languages e.g perl. Here are some of the scripts :
These are perl scripts which encapsulates the dtrace script. Most of the text processing is written in perl. Dtrace probe provide the pblock content from where request and response information is obtained.

Here is the output of these scripts.

View various types of HTTP requests count with timestamp

# perl
Timestamp GETs POSTs HEADs TRACEs 12:41:13 0 0 0 0 12:41:18 5469 0 0 0 12:41:23 10000 0 0 0

View various types of HTTP requests count with client browsers

# perl
Firefox   19   0     0     0
MSIE      0    0     0     0
Wget      0    0     0     0
Other     303  0     0     0

View HTTP requests' URIs and their response codes

# perl
Response Code   Uri                            File Name
200             /index.html                    /index.html
304             /index.html                    /index.html
403             /img/gradation_header_L.gif    /img/gradation_header_L.gif
403             /img/gradation_header_R.gif    /img/gradation_header_R.gif

View HTTP requests' URIs for a particular response code

# perl -c 200
Response Code   Uri                            File Name
200             /index.html                    /index.html

A top like utility for Sun Web Server (similar to apachetop)

# perl -d 10
20:10:40  Requests: 4223  (  422/sec) Bytes: 12930794(1293079/sec)
Requests: GETs: 4223   POSTs: 0      HEADs: 0      TRACE: 0
Responses: 1xx:0      2xx:4025   3xx:68     4xx:130    5xx:0

Requests  Reqests/sec  Bytes Sent  URI
601       60           7234838      /index.html
545       54           198925       /img/gradation_header_L.gif
498       49           3988482      /img/side_bg.gif
396       39           930204       /img/sun_logo.gif
359       35           56004        /img/gradation_header-btm.gif
342       34           2394         /cgi-bin/
250       25           10750        /img/a.gif
240       24           96720        /img/gradation_header_R.gif
206       20           52324        /img/footer_R.gif
191       19           58828        /img/gradation_header.gif
130       13           37960        /notfound.html
115       11           15410        /img/content_hline.gif
113       11           29041        /img/footer_L.gif
109       10           207754       /img/sjsws_title_text.gif
68        6            0            /yahoo
60        6            11160        /img/footer.gif
Here is the compiled binary for the plugin :
Solaris sparc
Solaris x86

Wednesday Jun 11, 2008

Reverse proxy and redirecting request.

Web Server reverse proxy and redirect. If reverse proxy instance wants to redirect every request using the following
syntax :
NameTrans fn="redirect" url-prefix="http://host:port/xyz"

It works fine in normal instances but it creates problem in reverse proxy
functionality. Reverse proxy have a thread named route_offline_thread which
uses "OPTIONS" HTTP method to find the health of the backend instance. The
result of the above is that OPTIONS request is also redirected and
route_offline_thread doesn't work properly. There is a very simple workaround
for this situation. Just do not redirect the OPTIONS request in RPP. Use the
following instead :

<If not $internal and not method = "OPTIONS">
    NameTrans fn="redirect" url-prefix="http://host:port/xyz"

Reverse proxy instance health check (route_offline_thread)

How Sun Web Server reverse proxy manages backend instances lifecycle.[Read More]

Monday Apr 28, 2008

Connection pooling in Sun Java System Web Server

Web Server Architecture : Connection pooling
Connection pooling is one of the cool features of Sun Web Server. This feature has been there from a long time. When a request comes to Web Server, acceptor thread accept the requests and puts into the connection queue. One of the daemon session threads (also called worker threads) pulls the request from the connection queue and starts processing the request. After the request is served, if the client has requested the connection to be alive then connection will go to the keep alive pollmanager. Keep alive threads poll all keep alive connections, whenever a new request comes to the this connection then keep alive threads again puts the connection to the connection queue. If there is no further request to the same connection for some period (keep alive timeout) then keep alive threads close the connection.

The reason why Sun Web Server perform so well under high load conditions e.g in specweb is because of the above Conneciton pooling architecture. In Apache Web Server, prefork and worker mpm are most common mpms. One of the problem with these mpm is that connection is bound to a thread.  So what this mean is that number of concurrent connection is typically the number of processing threads. If load increases beyond some point then connections will start timing out very soon. Event mpm tries to address this problem.

Tunables :
Here are some of the configuration parameters which affect the connection queue
(a) thread-pool --> queue-size
(b) http-listener->listen-queue-size.
(c) keepalive->timeout
(d) keepalive->pollinterval

As I wrote before, acceptor threads accept the connection and puts into the connection queue. The question is what really happen to new connections when OS is busy and OS has not yet scheduled the acceptor thread. OS kernel maintains the TCP connections on behalf of Web Server process. listen-queue-size is the number of connections kernel will accept before application accept the connection. If there is more than listen-queue-size connections before Web Server calls accept, then new connection will be rejected. This is not at all a common situation but could happen on very busy systems. Here is a small experiment to demostrate this situation:

Step 1 : I sent a stop signal to my child Web Server process so that acceptor thread
         won't be able to accept any new connection.
Step 2 : I sent a 200 simultaneous request (using apache benchmark tool ab)
$ ab -c 200 -n 200 http://hostname/index.html
Step 3 :  I ran the netstat -an command to see the connections
Here is the connection line looks.  5312      0 49437      0 ESTABLISHED

As expected, kernel will reject the new connections. Here is the ab output :
Benchmarking (be patient)
apr_poll: The timeout specified has expired (70007)

What happens if we disable the connection pool :
If we disable the thread-pool. In such cases, there is no connection queue, no daemon session threads. Acceptor threads themselves process the request. Here is the typical call stack will look like :
-----------------  lwp# 12 / thread# 12  --------------------
 fd5c1134 pollsys  (fc8efc80, 1, fc8efc00, 0)
 fd55d028 poll     (fc8efc80, 1, 1388, 10624c00, 0, 0) + 7c
 fe3d862c pt_poll_now (fc8efcec, 1, fc8efd1c, fc8efc80, 20, ffffffff) + 4c
 fe3d9ed0 pt_Accept (247408, 1bd990, ffffffff, 15, 0, ffffffff) + cc
 ff17db5c PRFileDesc\*ListenSocket::accept(PRNetAddr&,const unsigned) (1bd708, 1bd990, ffffffff, 2060, 245a88, 1bd988) + c
 ff16e67c int DaemonSession::GetConnection(unsigned) (2b7648, ffffffff, ff2ff800, ffffe800, ff26f000, ff2abc00) + 64
 ff16edb0 void DaemonSession::run() (6c0008, 2b7648, 2b7668, 2b76f0, 2000, ffffffff) + 150
 fef76d48 ThreadMain (2b7648, 2a6eb0, 0, fef89c00, ff16ec60, ff2abc64) + 1c
 fe3de024 _pt_root (2a6eb0, fef76d2c, 400, fe3f6ba4, 0, fe3f891c) + d4
 fd5c0368 _lwp_start (0, 0, 0, 0, 0, 0)

Few Call stacks :
Let us see the pstack information to see how it looks like. In my test configuration, I setup the min/max threads to 2 and disabled j2ee plugin. Then I dumped the stack using :
$ pstack <child_webservd_pid> | c++filt > pstack.txt

Let us see what are various threads doing :
1. Acceptor threads call stack :

Here is the call stack of acceptor thread.
-----------------  lwp# 13 / thread# 13  --------------------
 fd5c1134 pollsys  (fc8bfc88, 1, fc8bfc08, 0)
 fd55d028 poll     (fc8bfc88, 1, 1388, 10624c00, 0, 0) + 7c
 fe3d862c pt_poll_now (fc8bfcf4, 1, fc8bfd24, fc8bfc88, 20, ffffffff) + 4c
 fe3d9ed0 pt_Accept (245368, 20ce50, ffffffff, 15, 1, ffffffff) + cc
 ff17db5c PRFileDesc\*ListenSocket::accept(PRNetAddr&,const unsigned) (1bd708, 20ce50, ffffffff, 3, 2ab988, 2ab988) + c
 ff1782a4 void Acceptor::run() (12e488, 245908, 20ce48, 6, 3e8, 45) + 184
 fef76d48 ThreadMain (12e488, 11d828, 0, fef89c00, ff178120, ff2ac444) + 1c
 fe3de024 _pt_root (11d828, fef76d2c, 400, fe3f6ba4, 0, fe3f891c) + d4
 fd5c0368 _lwp_start (0, 0, 0, 0, 0, 0)

2. Idle Daemon session thread's call stack :
Here is the stack trace of an idle deamon session thread :
-----------------  lwp# 16 / thread# 16  --------------------
 fd5c0408 lwp_park (0, 0, 0)
 fd5ba49c cond_wait_queue (50a10, 2cfec8, 0, 0, 0, 0) + 28
 fd5baa1c cond_wait (50a10, 2cfec8, 0, 1c, 0, fcd52d00) + 10
 fd5baa58 pthread_cond_wait (50a10, 2cfec8, 1, fe3f8518, 5fc, 400) + 8
 fe3d79e8 PR_WaitCondVar (50a08, ffffffff, 2a7700, 0, 2ab988, 0) + 64
 ff17797c Connection\*ConnectionQueue::GetReady(unsigned) (8bcc8, ffffffff, ffffffff, 8bcc8, 5fc, 2ab968) + c4
 ff16e630 int DaemonSession::GetConnection(unsigned) (2b7648, ffffffff, ff2ff800, 0, ff26f000, ff2abc00) + 18
 ff16edb0 void DaemonSession::run() (746008, 2b7648, 2b7668, 2b76f0, 2000, ffffffff) + 150
 fef76d48 ThreadMain (2b7648, 2a7700, 0, fef89c00, ff16ec60, ff2abc64) + 1c
 fe3de024 _pt_root (2a7700, fef76d2c, 400, fe3f6ba4, 1, fe3f891c) + d4
 fd5c0368 _lwp_start (0, 0, 0, 0, 0, 0)

As connection queue is empty so this daemon session is waiting for request to arrive in connection queue.

3. Processing Daemon session thread's call stack :
Here is the another daemon session thread (which is processing the "/cgi-bin/" request) :
 fd5c1248 read     (1d, 758088, 2000)
 ff167588 int ChildReader::read(void\*,int) (3c9c4, 758088, 2000, 0, b71b00, 1) + 1c
 ff0f2e30 INTnetbuf_next (758028, 1, 2000, 2001, 60, 758028) + 2c
 ff13a364 int cgi_scan_headers(Session\*,Request\*,void\*) (11ee70, 11eee8, 758028, 27c, ff29bbd8, 0) + 84
 ff13abd4 int cgi_parse_output(void\*,Session\*,Request\*) (758028, 11ee70, 11eee8, ff2a1c50, ff29bbd8, 6e) + 1c
 ff13b8cc cgi_send (2c6dc8, 11ee70, 11eee8, 3c948, 1400, ff2a25cc) + 514
 ff10df68 func_exec_str (2354f8, 2c6dc8, 0, fc348, 11eee8, 11ee70) + 1c0
 ff10edc0 INTfunc_exec_directive (3d948, 2c6dc8, 11ee70, 11eee8, 280a28, 1) + 2a0
 ff113b60 INTservact_service (0, 11ee70, 2c6dc8, 0, 11eee8, 27b470) + 374
 ff11472c INTservact_handle_processed (0, 11eee8, 11eee8, 11ee70, 0, 2d6328) + 8c
 ff172064 void HttpRequest::UnacceleratedRespond() (11edc8, 3, 1, ff26f400, 0, 20) + e34
 ff170a24 int HttpRequest::HandleRequest(netbuf\*,unsigned) (11edc8, 754008, 11edf8, 2000, 754068, 11edd0) + 7a8
 ff16f10c void DaemonSession::run() (ffffffff, 280a08, 280a28, 280ab0, 1, ffffffff) + 4ac
 fef76d48 ThreadMain (280a08, 266728, 0, fef89c00, ff16ec60, ff2abc64) + 1c
 fe3de024 _pt_root (266728, fef76d2c, 400, fe3f6ba4, 1, fe3f891c) + d4
 fd5c0368 _lwp_start (0, 0, 0, 0, 0, 0)

Note that the above thread is waiting for data from child cgi process (

4. Keep alive thread's call stack :
Let us see the call stack of keep alive thread :
-----------------  lwp# 9 / thread# 9  --------------------
 fd5c0408 lwp_park (0, 0, 0)
 fd5ba49c cond_wait_queue (2c6e98, 7092c8, 0, 0, 0, 0) + 28
 fd5baa1c cond_wait (2c6e98, 7092c8, 0, 1c, 0, fcd51100) + 10
 fd5baa58 pthread_cond_wait (2c6e98, 7092c8, 1, fe3f8518, 5fc, 400) + 8
 fe3d79e8 PR_WaitCondVar (2c6e90, ffffffff, 13e790, 0, 0, 0) + 64
 ff176024 void PollArray::GetPollArray(int\*,void\*\*) (25e4c8, fcc3fec0, fcc3fec4, 3, 0, ff2ac000) + 5c
 ff176984 void KAPollThread::run() (15c5a8, 5, 4, ff2ac000, ff2ff8cc, fcc3fec4) + 6c
 fef76d48 ThreadMain (15c5a8, 13e790, 0, fef89c00, ff176918, ff2ac3b8) + 1c
 fe3de024 _pt_root (13e790, fef76d2c, 400, fe3f6ba4, 1, fe3f891c) + d4
 fd5c0368 _lwp_start (0, 0, 0, 0, 0, 0)

Saturday Nov 03, 2007

Sun Java System Web Server 7.0 monitoring - Part 1.

     Web Server 7.0 has some interesting ways to monitor the server. In previous versions of the Web Server, we typically create a uri e.g /.perf to obtain perfdump output.  When we wanted to get the perfdump output, we send a /.perf request. The problem with this approach is if Web Server is hanging because of some faulty application or in situations where Web Server is very busy processing the request then perfdump request also hung and we need to wait for /.perf request to get processed. In complicated situations, sometimes we were never able to obtain perfdump output. Web Server 7.0 comes to rescue in such situations.
    Let us do simple simple experiments to simulate a hanging Web Server.  Let us first change the min/max daemon session threads and set those to 2 so that Web Server can only process 2 requests at a time. Here is thread-pool element of my server.xml ,

After changing the server.xml, let us restart the Web Server. Now let us create a small cgi script which takes more than 5 minutes to execute. This will help us simulate a hanging web server.

print "Content-Type: text/html\\r\\n";
print "\\r\\n";
print "Hello\\r\\n";

Now let use send two requests in parallel from two browser windows or by some command line tools e.g curl
$ curl --dump-header - -o - http://localhost:8080/cgi-bin/ &
$ curl --dump-header - -o - http://localhost:8080/cgi-bin/ &

The above two requests will be served after 5 minutes. Now let us try to see
if we can obtain the perfdump.
$ curl --dump-header - -o - http://localhost:8080/.perf
[3]+  Stopped                 curl --dump-header - -o - http://localhost:8080/.perf
$ bg
[3]+ curl --dump-header - -o - http://localhost:8080/.perf &

Our perfdump (/.perf) request hung. Now let us start the admin server and try to get perfdump of instance from admin server.
$ cd admin-server
$ bin/startserv
$ ./bin/wadm get-perfdump --config=myhostname --node=localhost
Process  Status    Client         Age  VS            Method  URI               Function

4952     response  110  myhostname  GET     /cgi-bin/  send-cgi
4952     response  109  myhostname  GET     /cgi-bin/  send-cgi

Voila! it worked. It showed me that there are two running and they are running since 110 seconds.

So how does it work. Why did /.perf request hung and why did get-perfdump wadm command work. Here is the explanation.

Since we created 2 daemon session threads and then send 2 requests.  All daemon session threads got busy processing requests. When we send the /.perf request. Since there was no daemon session threads available so The request went into the connection queue and waiting in connection queue to be served.

This information is also available in perfdump. Here is the snippet.
Current/Peak/Limit Queue Length            1/1/1226
Total Connections Queued                   17

Note that there is 1 connection in the connection queue and that should be our "/.perf" request.

So it is now clear that if all daemon session threads are busy then web server won't be able to respond to monitoring requests based on uri. That was a big bottleneck in Web Server 6.1 and before. In situations when Web Server hung, we were not able to collect the perfdump output and it was not easy to find out what requests Web Server was processing at hanging stage.

Now the next question is how does get-perfdump wadm command works. Answer to this question lies into the monitoring architecture. There is a communiation channel from admin server to Web Server instance. All monitoring data is passed to admin server by that channel.  Also Web Server worker process has a dedicated thread for serving monitoring requests. Even though all daemon session threads are busy, this thread can generate the performance data and pass it to admin server. This is exactly what happened when we send the get-perfdump wadm command.

There are several interesting things to note in the above exercise. Admin server was not running at the time when Web Server hung. We simply started the admin server and ran get-perfdump command. During admin server startup the communication channel between admin server and instance is set up. This communication channel setup is independent of startup order of admin server or Web Server instance. They could be started in any order. Even when Web Server was busy processing the requests, this channel was successfully initialized.
There are other monitoring means e.g stats-xml, wadm CLI commands and administration GUI. They all use the same channel to access monitoring data from instance.


Basant Kukreja


Top Tags
« August 2016