Using perfdump to Troubleshoot Hangs in Web Server 7.0

perfdump, for the uninitiated, is a tool for retrieving information about web server performance. It's been built into Sun and Netscape's web server software since Netscape Enterprise Server 3.0.

We're now at version 7.0, and perfdump has undergone a couple notable enhancements.

In the past, the only way to access perfdump was to send an HTTP request to the server. For example, you could access /.perf in your web browser and get output like the following:

webservd pid: 17278

Sun Java System Web Server 7.0-Technology-Preview-3 B09/13/2006 15:00 (SunOS DOMESTIC)

Server started Mon Sep 18 21:16:47 2006
Process 17278 started Mon Sep 18 21:16:48 2006

ConnectionQueue:
-----------------------------------------
Current/Peak/Limit Queue Length            0/3/1352
Total Connections Queued                   4723
Average Queue Length (1, 5, 15 minutes)    0.00, 0.00, 0.00
Average Queueing Delay                     0.14 milliseconds

...

Because perfdump could only be accessed over HTTP, it wasn't much good for figuring out what went wrong when your server stopped responding to requests. This is no longer a problem in Sun Java System Web Server 7.0, because you can now get the same perfdump data using wadm, Web Server's new command line admin tool:

wadm> get-perfdump --config webtier.sfbay.sun.com --node localhost

Sun Java System Web Server 7.0-Technology-Preview-3 B09/13/2006 15:00 (SunOS DOMESTIC)

Server started Mon Sep 18 21:16:47 2006
Process 17278 started Mon Sep 18 21:16:48 2006

ConnectionQueue:
-----------------------------------------
Current/Peak/Limit Queue Length            0/3/1352
Total Connections Queued                   4726
Average Queue Length (1, 5, 15 minutes)    0.00, 0.00, 0.00
Average Queueing Delay                     0.14 milliseconds

...

The perfdump output has been enhanced, too, and now includes the client IP and request URI for every request it's processing, as well as how long it's spent processing that request:

Sessions:
--------------------------------------------------------------------------------------------------------
Process  Status      Client          Age  VS                     Method  URI                Function

17278    response    192.18.176.71   2    webtier.sfbay.sun.com  GET     /webtier/Wiki.jsp  service-j2ee
17278    response    192.18.177.198  1    webtier.sfbay.sun.com  GET     /.perf             service-dump
17278    keep-alive  192.18.176.71
17278    keep-alive  192.18.177.198
17278    keep-alive  10.5.83.28

...all of which is very handy for identifying problematic plugins and web apps!

Update: Sun Java System Web Server 7.0 has now been officially released, and you can download it for free from http://www.sun.com/download/.

Comments:

Hi Chris, This is of course great news that with version 7 it is possible to do .perf dump via command line tool, but frankly speaking I see very little benefit except for the more informative output. Today I can always do something like this from the server prompt even with earlier versions:
(echo "GET /.perf";sleep 1 ) | telnet www.mysite.com 80
Of course, if the server does not respond or the process has crashed there is no way I can see what is going on. Can you address the question that I have asked here... When the process is dead, is there any way to see the last .perf dump result anywhere cached on the disk? I would not mind to have the access to last 10-100 smaples just to be able to see the trend. It is really critical to be able to say at which state the server was when it is crashed under heavy load. If the dump is available the parameter that was maxed-up can be adjusted! Best regards, Alexander aka ttalex

Posted by ttalex on October 19, 2006 at 07:33 PM PDT #

perfdump data is collected when you request it. It's not cached anywhere. It doesn't exist in any file on disk.

In any event, perfdump data is frankly not useful for diagnosing server crashes. Fortunately, there already exists an excellent mechanism for diagnosing server crashes: the core file. The core file contains cycle-accurate information about exactly what the server was doing; interfaces like perfdump will never be able to do that.

As its name implies, perfdump is useful for diagnosing performance problems. Unfortunately, until 7.0, perfdump couldn't be used to diagnose server hangs, a common type of performance problem. As of 7.0, you can now use perfdump to diagnose server hangs.

Posted by Chris Elving on October 20, 2006 at 11:52 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

elving

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today