Chime and the DTraceToolkit, Part 2

In my previous entry, Chime and the DTraceToolkit, I demonstrated how to adapt a script from the DTraceToolkit for display in Chime (including screenshots). I chose a simple script, bitesize.d, and deferred the investigation of how easily Chime could run more complicated scripts from the toolkit.

Before I pick up that investigation again (this time using procsystime), I want to highlight a point from the previous entry that may have gone unnoticed in the larger discussion. The point is that Chime can display DTrace programs directly from the command line, provided that they satisfy a few simple requirements. You can try this with many of the OneLiners from Brendan Gregg's DTrace Tools page, for example:

# Read bytes by process,
dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

To run with Chime, simply change dtrace to chime:

/opt/OSOL0chime/bin/chime -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

You can use -n to specify a program string and -s to specify a program file, just as you would with dtrace(1M). Similarly, you can specify DTrace options using -xoption=value and -Z. The column headers are taken from identifiers in your D program unless you specify headers with the -h option. You can specify a title for the title bar other than "Display" using the -t option. The previous entry has more detail on how to improve the display (and how to save it with the -w option).

Adapting a Shell Script

Most of the DTrace programs used in the toolkit are wrapped in shell scripts that offer extensive options. The iosnoop script, for example, has the following options:

# USAGE: 	iosnoop [-a|-A|-DeghiNostv] [-d device] [-f filename] 
#			[-m mount_point] [-n name] [-p PID]

Since the DTrace API cannot compile shell scripts, Chime cannot run them as-is. The DTrace program first needs to be removed from its wrapper. What I found is that the various "snoop" scripts (iosnoop, execsnoop, opensnoop, etc.) are not good candidates for adaptation to Chime because they do not use aggregations and instead rely on printf(). In a snoop, every probe firing is appended to the output. In Chime, that would translate into an event-driven display rather than a regular sampling of the aggregate. The Java DTrace API provides a ConsumerListener interface to listen for ProbeData in a DataEvent; the formatted elements of a DTrace printf() statement can thus be obtained from the getRecords() method of a PrintfRecord included in the probe data. The point is that it's certainly possible to add an event-driven display module to Chime, but currently Chime does not support printf(). Often it makes sense to change a script that relies on printf() so that it aggregates its data instead, but I don't think that's the case with any of the snoop scripts.

If the shell script wraps a DTrace program that does use aggregations, you still have to look at each shell option one by one and decide how it will be supported in Chime. As an example, I've done this with procsystime, a script that supports the following options:

# USAGE:	procsystime [-acehoT] [ -p PID | -n name | command ]
#		-p PID          # examine this PID
#		-n name         # examine this process name
#		-a              # print all details
#		-c              # print syscall counts
#		-e              # print elapsed times
#		-o              # print CPU times
#		-T              # print totals

I decided that the -a, -c, -e, and -o options could all be dropped, since there is no reason for Chime not to display all the fields at once. When running procsystime -a on the command line, syscall counts, elapsed times, and CPU times are displayed in separate output blocks stacked vertically, so that only syscall counts are visible without scrolling up to see the aggregated times. Chime will improve on this by placing the three value types side-by-side in a row for each process, with column headers that never scroll out of view.

Similarly, I decided to drop the -T option, since I saw no reason not to display totals every time.

That left the three mutually exclusive options: [ -p PID | -n name | command ]. Without a shell wrapper, the only way to support these in Chime is with macro arguments for the -p and -n options, and target process for the command option. The basic idea is to replace shell script option assignments with macro argument assignments. So, from procsystime:

  \* Command line arguments
 inline int OPT_elapsed  = '$opt_elapsed';
 inline int OPT_cpu      = '$opt_cpu';
 inline int OPT_counts   = '$opt_counts';
 inline int OPT_filter   = '$opt_filter';
 inline int OPT_pid      = '$opt_pid';
 inline int OPT_name     = '$opt_name';
 inline int OPT_totals   = '$opt_totals';
 inline int OPT_command  = '$opt_command';
 inline int PID          = '$pid';
 inline string NAME      = "'$pname'";
 inline string COMMAND   = "'$command'";

we change the assignments as follows in procsystime.d as it will be used by Chime:

inline string NAME = $$1;
inline int PID = $2;
inline int OPT_pid = (PID > 0);
inline int OPT_name = (NAME != "");
inline int OPT_command = ($target > 0);
inline int OPT_filter = (OPT_pid || OPT_name || OPT_command);

The -a, -c, -e, -o, and -T (totals) options are dropped. I chose not to tab-align the assignment operators, since the alignment does not display correctly in Chime's program viewer (in spite of monospace font). I think that inlining the cacheable expressions should be more efficient than assigning them to non-cacheable global variables in the BEGIN clause, but I may be missing a more efficient way to do this.

Differences such as the removal of the END clause along with its printa() statements were already explained in the previous entry; I won't repeat them here. The unwrapped program is now ready for display in Chime:

/opt/OSOL0chime/bin/chime -c 'date' -s procsystime.d -xdefaultargs \\
-h "System Call, Count, Elapsed Time, CPU Time" -t "Process System Call Time"

The above command generates a draft display (I could have omitted the -h and -t options for this first cut). The most important part of the command to note here is the -xdefaultargs option. This sets the "defaultargs" DTrace compile-time option, which tells DTrace to use zero (0) or empty string ("") as the value for unspecified macro args. Since PID, NAME, and COMMAND are mutually exclusive, we need to leave all but one unspecified. Without -xdefaultargs, that would fail with the following error:

failed to compile script /opt/OSOL0chime/displays/new/procsystime.d: line 44: macro argument $$1 is not defined

Specifying Macro Arguments

Chime reads the macro variables in your program and generates a dialog to prompt for the argument values:

You can specify labels for the macro variables using the -M option, potentially sparing people the effort of inspecting your program to decide what kind of values are expected:

/opt/OSOL0chime/bin/chime -c 'date' -s procsystime.d -xdefaultargs \\
-h "System Call, Count, Elapsed Time, CPU Time" -t "Process System Call Time" \\
-M "Process Name, Process ID"

To enforce the mutual exclusivity of these options, surround the macro argument labels after -M with square brackets:

/opt/OSOL0chime/bin/chime -c 'date' -s procsystime.d -xdefaultargs \\
-h "System Call, Count, Elapsed Time, CPU Time" -t "Process System Call Time" \\
-M "[Process Name, Process ID, \\$target]"

In order to group all three options, it is necessary to include the special string "$target" (escaping the dollar-sign), which indicates the order of the Target Process field relative to the named macro arguments. If the $target variable appears in your DTrace program, and you omit "$target" after -M, the Target Process appears first by default. The above command instead places it last (just for demonstration purposes):

Now it is only possible to specify one of the three options, as intended in the original procsystime script. The first field (in the order specified by -M) is selected initially. The remaining unselected fields are disabled.

It is possible to bracket multiple groups of argument labels after -M, resulting in multiple radio button groups. The first label in each group is selected by default. Without the square brackets, Chime will not generate radio buttons, allowing any or all of the arguments to be specified. (Macro argument grouping can also be done in Chime's display creation wizard.)

Without the -xdefaultargs option, Chime will require all options to be specified before enabling the OK button.

If the macro argument labels are not enough to make clear what is expected, a person can still inspect the DTrace program by clicking the View D Program button:

The procsystime Display

After selecting Target Process in the argument prompt and clicking OK (accepting the "date" value already specified with -c on the command line), the following display appears:

This is a rough first cut, with many opportunities for improvement:
  • Add a totals row (to satisfy the original -T option)
  • Link the ranges of the Elapsed Time and CPU Time columns so they are visually comparable
  • Add unit labels to the nanosecond time values
  • Use a different color for the Count column, to differentiate it from the time columns
  • Set the initial sort to Elapsed Time descending
  • Make the window a little wider
As with bitesize.d, we'll also want to accumulate values over time, rather than clearing them after each time interval. These changes are not yet supported from the command line, so we need to save what we've done so far by repeating the command with the -w option:

; /opt/OSOL0chime/bin/chime -c 'date' -s procsystime.d -xdefaultargs \\
-h "System Call, Count, Elapsed Time, CPU Time" -t "Process System Call Time" \\
-M "[Process Name, Process ID, \\$target]" -w
Wrote /opt/OSOL0chime/displays/new/process_system_call_time.xml
To run, enter "/opt/OSOL0chime/bin/chime -C /opt/OSOL0chime/displays/new/process_system_call_time.xml".

This writes the .xml display description that we can edit using Chime's display wizard. Both the previous entry and the New Display Wizard page give examples of how to make these kinds of changes, so I won't go through the steps here. Instead I'll skip to the final display after all the changes are finished:

I also took some text from the comments in the procsystime script to put in the Description pane when the new display is selected:

The description helps people understand what they're looking at and remains visible while the display is running as long as the new display is selected.

To try the procsystime display for yourself, add the following files to /opt/OSOL0chime/displays/new: Then run /opt/OSOL0chime/bin/chime without any options and select New Displays from the Trace Group pulldown. Finally, double-click Process System Call Time ... from the Traces list. Macro argument grouping (for mutually exclusive options) is a new feature in Chime version 1.4.20 (16 Jan 2007), so if you don't already have that version, you can download it here.

Post a Comment:
Comments are closed for this entry.



« April 2014

No bookmarks in folder