dtrace and process name table

Kernel experts and dtrace experts probably know about the following, but not all of us fit into this category!

The pid provider is quite useful when you decide to narrow your dtrace investigations to a single process. But what if you want to look at a short lived process or muliple processes from a single executable, for example, the dozens of instances of mozilla on a Sun Ray server? Dtrace allows you to filter probes on execname. For example, if I want to look at the syscall entries of an executable and surrounding user call stack, I could do something like this:

#!/usr/sbin/dtrace -s

syscall:::entry
/execname == "nautilus-throbber" /
{
  ustack(5);
}

I called the script whodo.d, chmoded it 755 so I can run it:
# ./whodo.d
Now I browse a directory or two in nautilus and...nothing. Absolutely nothing prints out when I should be getting stack traces from every syscall. Well, that's unfortunate. When I ps -efl | grep nautilus-throbber it appears:
$ ps -efl | grep -i nautilus-throbb
 0 S bn128650 21415     1   0  40 20        ?   8500        ? 14:43:31 ?    0:01 /usr/lib/nautilus-throbber --oaf-ac
When I pgrep it...
# pgrep nautilus-throbber   
#                                {nothing?? What the???}
# pgrep nautilus-throbbe
#                                {still nothing}
# pgrep nautilus-throbb 
21415                            {aha!  A clue?}
I man pgrep and find out that: Both utilities match their pattern argument against either the pr_fname or pr_psargs fields of the /proc/nnnnn/psinfo files. The lengths of these strings are limited according to definitions in sys/procfs.h. Patterns which can match strings longer than the current limits may fail to match the intended set of processes.

The -f flag selects whether pgrep matches pr_fname or pr_psargs. The default is pr_fname which is limited to 16 characters by PRFNSZ in /usr/include/sys/procfs.h. Is this a case where Solaris's strict backwards compatibility is hurting us? I login to a 2.6 kernel linux box and man pgrep:

...
NOTES
       The process name used for matching is limited  to  the  15
       characters  present  in the output of /proc/pid/stat.  Use
       the -f option to match against the complete command  line,
       /proc/pid/cmdline.
...

  STANDARDS
       pkill and pgrep were introduced in Sun's Solaris 7.   This
       implementation is fully compatible.
Well, linux's pgrep isn't strictly compatible since 15 != 16, but wait a minute, nautilus-throbb is only 15 characters so pgrep's limit defined by PRFNSZ is one more than the real limit. I ran the whodo.d script with the 15 character truncated process name:
/execname == "nautilus-throbb" /
and... it didn't work! Now what? I add an "e".
/execname == "nautilus-throbbe" /
and it finally works! It seems that dtrace sees all 16 characters of the execname. Fortunately dtrace allows execname to be matched as a regular expression so when in doubt, add an "\*". I know this isn't rocket science but problems like this can be vexing for newcomers to dtrace so I hope this post is helpful to someone else and a reminder to me.
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

bnitz

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today