Chime and the DTraceToolkit

A while back someone suggested in the "Chime Comments/Questions" thread on the DTrace discussion list

"Maybe we can all chip in money and beer (or pop if he doesn't drink alcohol) to get Brendan to integrate his toolkit with chime. :)"

I've been meaning to look into this myself. Chime is a graphical tool for displaying DTrace aggregations. Its ability to sort by multiple columns, display column totals, and plot aggregation values over time might provide useful advantages over command-line output for many of the scripts in the toolkit. I wanted to see if the idea of integrating the toolkit is feasible.

Generating a Display from the Toolkit

First, I downloaded and installed the DTraceToolkit. Next I scanned Brendan's DTrace Tools page for a simple script and found bitesize.d:

"a simple program to examine the way in which processes use the disks ..."

This sounded like a good one to start with, so I located bitesize.d in the toolkit:

; find . -name bitesize.d

I diff'd the files and found that they are identical, so I ran the one in Bin to get an idea of what Chime ought to display:

; ./Bin/bitesize.d
Tracing... Hit Ctrl-C to end.

     PID  CMD


Nothing. Of course, I needed to generate some I/O in order to get any data. So I ran it again, and this time I wrote some bytes to a file in another window:

; echo "cat dog mouse" > tmp.txt

then I pressed Ctrl-C:

; ./Bin/bitesize.d
Tracing... Hit Ctrl-C to end.

     PID  CMD
       3  fsflush\\0

           value  ------------- Distribution ------------- count
             256 |                                         0
             512 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
            1024 |                                         0

       0  sched\\0

           value  ------------- Distribution ------------- count
            2048 |                                         0
            4096 |@@@@@@@@@@@@@@@@@@@@                     1
            8192 |@@@@@@@@@@@@@@@@@@@@                     1
           16384 |                                         0


Without even looking at the script, I could see that it's a distribution of I/O size in bytes aggregated by process ID and command. Chime has -n and -s options like those in dtrace(1M) for running arbitrary DTrace programs, so I used the -s option to specify the program file:

; /opt/OSOL0chime/bin/chime -s ./Bin/bitesize.d
WARNING: printa on line 79 makes aggregation @Size unavailable to Chime.

An empty Chime window accompanied the console warning. No problem. This issue with printa() is covered on the Chime project page under Adding New Chime Displays (a link in the left margin) in the section Adapting an Existing DTrace Program, item 2:

If you use the printa() action on an aggregation, that aggregation will thereafter be unavailable to Chime. Typically, printa() is found in a profile probe such as tick-1sec. You should delete these tick clauses from your program. Instead of printing your aggregations, you are now relying on Chime to display them.

So I made a copy of bitesize.d with the following changes:

; diff ./Bin/bitesize.d ~/bitesize.d
>  \* 02-Jan-2007        Tom Erickson    Adapted for use in Chime.
< #pragma D option quiet
<  \* Print header
<  \*/
< dtrace:::BEGIN
< {
<       printf("Tracing... Hit Ctrl-C to end.\\n");
< }
< /\*
< }
< /\*
<  \* Print final report
<  \*/
< dtrace:::END
< {
<       printf("\\n%8s  %s\\n", "PID", "CMD");
<       printa("%8d  %S\\n%@d\\n", @Size);

There is no need to print anything in the BEGIN and END clauses, so I deleted them. Chime will display the aggregated distribution on a regular interval. That interval will default to 1 second, but Chime lets you change the interval while the display is running. (It's also possible to set the initial interval by specifying the aggrate option in the display's XML description.)

It would be nice if Chime could run bitesize.d without modifications. If there was a DTrace option to ignore specified actions, Chime could tell DTrace to ignore both printa() and clear(), but for now you need to delete these actions from existing DTrace programs.

So I tried again with the modified bitesize.d:

; /opt/OSOL0chime/bin/chime -s ~/bitesize.d

empty Chime display of bitesize.d

Although the console warning about printa() was gone, I still got the same empty Chime window. Of course the reason is that I need to generate some I/O in order to get any data (just as I do when running the script on the command line). Yes, Chime should print a message to that effect on the console. I'll add that soon. (When a colleague of mine is asked for work status he responds, "It's perfect and getting better every day."). Fortunately Chime has the option to wait a given number of seconds before generating the display, just for cases like this, giving time to generate I/O in another window:

; /opt/OSOL0chime/bin/chime -s ~/bitesize.d -S 10

By default, Chime waits one second before generating the display. Waiting ten seconds lets us run the following in another window, as before:

; echo "cat dog mouse" > tmp.txt

As I write this, I realize that a better solution might be to drop the -S option and instead make Chime wait as long as it needs for non-empty aggregations before generating the display (although it would need some way to know which aggregations to wait for); but for now, a ten second countdown on the console gets the job done. This time a working display appeared:

Generated display of bitesize.d

The generated column names are taken from the DTrace program (except for "bucket" over the I/O size buckets, which lack a name within the program). Also the default title "Display" in the title bar isn't very helpful. Still this is a good start, and we can easily improve on it:

; /opt/OSOL0chime/bin/chime -s ~/bitesize.d -S 10 -h "PID,Command,Bytes,Count" \\
-t "I/O Size Distribution"

Generated display of bitesize.d

This looked good, so I ran it once more with the -w option to write the .xml display description:

; /opt/OSOL0chime/bin/chime -s ~/bitesize.d -S 10 -h "PID,Command,Bytes,Count" \\
-t "I/O Size Distribution" -w
Wrote /opt/OSOL0chime/displays/new/bitesize.d
Wrote /opt/OSOL0chime/displays/new/i_o_size_distribution.xml
To run, enter "/opt/OSOL0chime/bin/chime -C /opt/OSOL0chime/displays/new/i_o_size_distribution.xml".

Chime prints a message telling you how to run the generated display. The -S option to delay a given number of seconds will no longer be necessary, since the display is already generated (it no longer needs data to generate columns and column headers) and will, from now on, run successfully even before any data appears.


Chime also borrows -Z and -x from dtrace(1M) to specify DTrace options for the generated display. For example, if the display was failing due to drops, you could add -xaggsize=4m to increase the size of the aggregation buffer (from Chime's default 256 KB). If you wanted to change the initial interval from one second to five seconds, you would specify -xaggrate=5s.

To make further changes to the display not supported on the command line, you can edit the XML description in Chime's New Display Wizard. For example, bitesize.d does not use a tick-1sec clause to print data on a regular interval, but instead prints the accumulated data once at the very end. This probably indicates that a display of running totals would more closely match the intention of the script than a display of values per time interval. To access the wizard, run Chime without any options and click the toolbar icon to invoke the New Display Wizard icon in the toolbar:

New Display Wizard

Click the "Browse ..." button and select i_o_size_distribution.xml to load the generated display description. Then click "Next" at the bottom of the wizard twice to reach the "Set Cleared Aggregations" step:

New Display Wizard

Click "Clear only selected aggregations" and leave the "@Size" aggregation unchecked. This will result in running totals, since the aggregation values will never get reset to zero. Then click "Next" at the bottom of the wizard three more times to reach the "Provide a Description" step:

New Display Wizard

The above description is taken (mostly unchanged) from the comments in the original bitesize.d file. Enter the description and click "Finish".

To run the display, first select "New Displays" from the "Trace Group" pulldown:

New Display Wizard

Select "I/O Size Distribution" (the display title specified with -t on the command line) to view its description:

New Display Wizard

Double-click "I/O Size Distribution" to run the display:

New Display Wizard

Now the counts grow over time, just like they do when running the script on the command line, except that we can see them grow instead of blindly picking the instant for the final viewable result (the Chime display can also be paused at any time).

One script from the DTraceToolkit is now integrated with Chime. Granted, bitesize.d is probably the simplest script in the toolkit (other than the one-liners, which you can try as-is with Chime's -n option, except for a few that don't use aggregations; for example /opt/OSOL0chime/bin/chime -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'). The other scripts I looked at will not integrate so easily, since they are shell scripts with extensive options and many do not use aggregations. I'm thinking that some will need custom visualization tools to do them justice. I still think that many of them can be adapted in a way that preserves the original intention and increases usefulness, but the process will be more involved (for example, deciding if changing a script to use aggregations defeats the original performance considerations, or deciding which options are supportable by equivalent features in Chime, or which options justify multiple displays, etc.). Still I hope that this bitesize.d example gives a good idea of how to go about integrating an existing script with Chime.

After integrating a few more scripts from the toolkit, the next thing to do will be to create a separate "DTraceToolkit" subdirectory for them under /opt/OSOL0chime/displays, move the appropriate .xml and .d files from /opt/OSOL0chime/displays/new to the new subdirectory, and add a description.xml file to summarize the directory contents. Thereafter the group of displays can be selected from the "Trace Group" pulldown and a description of the group will appear in the "Description" pane. If many scripts from the toolkit are integrated, they can be organized in a subtree under the "DTraceToolkit" directory just as they are in the toolkit itself. Chime automatically generates submenus under the "Load" menu (under "File" in the menu bar) that mirror the directory subtree under /opt/OSOL0chime/displays. The "Trace Group" pulldown provides a flat list of the trace groups most recently loaded (remembered across multiple Chime sessions).

It would be nice if Chime could manage display directories for you, but for now you still need to create the directories manually. That's on my to-do list, along with a text pane to write the directory description (you'll need to copy and modify an existing description.xml file into each new directory for now).

Let me know if there's a particular script that you'd like to see integrated. If you're interested in integrating a script on your own, excellent! You are welcome to post it on the DTrace mailing list (that is the official discussion list for the Chime project).

Post a Comment:
Comments are closed for this entry.



« October 2016

No bookmarks in folder