Hacking Sun Java System Web Server pblocks using dtrace.

Hacking Sun Java System Web Server pblocks using dtrace. In my previous blog, I showed how to use NSAPI plugin and dtrace to write several monitoring tools. In this blog I will show that it is even possible to do few things with dtrace without having any NSAPI plugin installed into your web server. Yes that is true, dtrace can be used with stock web server without any configuration changes. Here is my watchpblocks.d dtrace script output (when I send a "/" request to my Web Server):
# dtrace -qs watchpblocks.d  11463
Req->vars : ntrans-base="/var/www" path="/var/www/" required-rights="list" content-length="1912"
Req->reqpb : clf-request="GET / HTTP/1.1" method="GET" protocol="HTTP/1.1" uri="/"
Req->headers : user-agent="curl/7.16.1 (sparc-sun-solaris2.8) libcurl/7.16.1 
                  OpenSSL/0.9.8d zlib/1.2.3 libidn/0.5.19" host="chilidev4.red.iplanet.com" 
                  accept="\*/\*" content-type="text/html" status="200 OK" 
                  transfer-encoding="chunked" content-length="2003"
11463 is the child process id of my test Web Server. Here is the output of wstop2.pl (similar to my previous version wstop.pl)
# perl wstop2.pl -d 5 11463
12:49:55  Requests: 3     (    0/sec) Bytes: 5736(  1147/sec)
Requests: GETs: 3      POSTs: 0      HEADs: 0      TRACE: 0
Responses: 1xx:0      2xx:3      3xx:0      4xx:0      5xx:0

Requests  Reqests/sec  Bytes Sent  URI
3         0            5736         /
\^C
So how does it work. If you look at the dtrace script, you will find :
pid$1::flex_log:entry
{
...
}
So at the end of each request webserver calls flex_log method to log the request, at that time above dtrace probe is fired. As with most NSAPI functions Request and Session structure pointers are passed as an argument. dtrace script parses the structure and try to decode the pblocks. For this technique to work users need to have accesslog enabled which is almost always enabled.

The interesting aspect is that we don't need to do any configuration changes.

Caution : Since dtrace doesn't allow "for" loop or if/else logic so the pblock hash decoding is a complete hack. It may not work on all scenarios. Also on busy systems, lots of dtrace probes might be missed using this method. Since we are copying data from kernel land to user land several times.

Previous dtrace version, was much more lightweight as far as dtrace work is concerned. If future version of dtrace provides if/else and loop constuct inside dtrace script then the script could be improved and make it more reliable.

Also, right now these scripts will only work for 32 bit web servers. Here are the scripts :
watchpblocks.d
wstop2.pl
wbdtrace.pm
Comments:

Cool, this is useful!

Posted by Manish Kapur on August 06, 2008 at 09:28 AM PDT #

Hi, Basant,

I tried and got this error,

in web 6.1,
> dtrace -qs watchpblocks.d 15598
dtrace: failed to compile script watchpblocks.d: line 13: probe description pid15598::flex_log:entry does not match any probes

in web7,
> dtrace -qs watchpblocks.d 15968
dtrace: failed to compile script watchpblocks.d: line 13: probe description pid15968::flex_log:entry does not match any probes

Both got
AddLog fn="flex-log" name="access"
AddLog fn="flex-log"

Did I miss any ?

Posted by Walter Lee on January 05, 2010 at 03:47 PM PST #

I tried on 6.1 too and it worked for me.

# dtrace -qs watchpblocks.d 21691
Req->vars : path=".../ws61/SUNWwbsvr/docs/index.html"
Req->reqpb : method="GET"
Req->headers : content-length="452" content-type="text/html"
\^C

Are you sure, your process id is correct? It has to be child (worker) process id (webservd).

Posted by Basant Kukreja on January 06, 2010 at 02:48 AM PST #

Excellent set of scripts!

Is there any way to get access to the names of the NSAPI modules? For example, if there is a proxy plugin getting loaded, it might be nice to see the order in which it loaded before the request gets served.

Thanks!

Posted by Brian on December 30, 2010 at 11:24 PM PST #

Post a Comment:
Comments are closed for this entry.
About

Basant Kukreja

Search

Top Tags
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today