X

Solaris serviceability and nifty tools

Grepping dtrace logs

I have been working on a tough bug for some non-trivial time. The bug is a
combination of race condition and data consistency issues. To debug this I am
using multi-threaded apache process and dtrace heavily. The logs produced by
dtrace are huge and contain mostly dumps of internal data structures.

The excerpt from such log looks e.g. like this:

  1  50244       pk11_return_session:return       128   8155.698
1 50223 pk11_RSA_verify:entry 128 8155.7069
1 50224 pk11_get_session:entry 128 8155.7199
1 50234 pk11_get_session:return 128 8155.7266
PK11_SESSION:
pid = 1802
session handle = 0x00273998
rsa_pub_key handle -> 0x00184e70
rsa_priv_key handle -> 0x00274428
rsa_pub = 0x00186248
rsa_priv = 0x001865f8
1 50224 pk11_get_session:entry 128 8155.7199
1 50244 pk11_return_session:return 128 8155.698

Side note: This is post-processed log (probe together with their bodies are
timestamp sorted, time stamps are converted
to miliseconds - see Coloring dtrace output entry for details and scripts).

Increasingly, I was asking myself questions which resembled this one: "when function
foo() was called with data which contained value bar ?"

This quickly lead to a script which does the tedious job for me.
dtrace-grep.pl accepts 2 or 3 parameters. First 2 are probe pattern and
data pattern, respectively. Third, which is optional, is input file (if not supplied,
stdin will be used). Example of use on the above pasted file looks like this:

~/bin$ ./dtrace-grep.pl pk11_get_session 0x00186248 /tmp/test
1 50234 pk11_get_session:return 128 8155.7266
PK11_SESSION:
pid = 1802
session handle = 0x00273998
rsa_pub_key handle -> 0x00184e70
rsa_priv_key handle -> 0x00274428
rsa_pub = 0x00186248
rsa_priv = 0x001865f8
~/bin$

Now I have to get back to work, to do some more pattern matching.
Oh, and the script is here.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.