Chime Automatic Drilldown

I think you'll like this latest update to Chime. Drilldown has been a feature of Chime since its initial release. It lets you see more detail about selected items of interest, for example a breakdown by function of a counter for a particular executable. Drilldown supports some of the dynamic capability of dtrace(1m) by launching a new display to answer a question prompted by the current display. However, even though drilldown is easy to use once enabled (simply right-click on an item of interest), it has been difficult to enable in new displays and downright tedious if you wanted to specify multiple levels of drilldown. The feature has always required the addition of placeholders in your DTrace program and an explicit description of every drilldown option you wanted to support.

This latest update removes some of the barriers to enabling drilldown in new displays, and even supports drilldown automatically for displays generated on the command line using the -n and -s options. For example:

    % chime -ktn 'syscall:::entry { @[execname] = count(); }' -T "System Calls"

To try this (Chime is available for download here), you'll first want to add /opt/OSOL0chime/bin to your PATH, or else add an alias to your shell startup script: 'alias chime=/opt/OSOL0chime/bin/chime' so you can use the command 'chime' as conveniently as you use the command 'dtrace'. The -n option specifies the DTrace program, just as it does with the dtrace command. The -k option says you want the display to include a sparkline of values over time. The -t option says you want display totals, and the -T option sets the title bar to something more informative than the default title "Display". There's nothing new here, except that now you get drilldown automatically. Run the above command to see how this works:

System Calls drilldown

The command produces a simple count of system calls per second for each executable. Imagine a mouse pointer at the top left corner of the popup menu over the highlighted "gnome-panel" item. Right-clicking brings up a long list of drilldown options labeled "gnome-panel by <something>". Selecting "... by CPU" modifies the DTrace program as follows:

program before drilldown by cpu program after drilldown by cpu

The aggregation key has been moved to the predicate and filters out all but the selected value. The new aggregation key, cpu, replaces execname in the square brackets. You can drilldown still further:

System Calls by CPU

A few things to notice here: First, the by CPU option no longer appears, since we already aggregated by cpu in the previous drilldown, and doing so again would not further expose any useful detail. Also, you can break down the sum of multiple items, rather than just a single item. In the example above, both CPU 1 and CPU 2 are selected, so breakdown by function looks like this:

program after drilldown by function System Calls by Function

Next, we could select the "read" function and request a breakdown by User Stack:

System Calls by User Stack

The Caller field shows the top of the stack, and the User Stack field shows a tag to identify the full stack trace. To see those stack traces, right-click one or more selected items and choose View Selected Stack Traces from the popup menu. Changing the row selection automatically updates the stack trace display. Multiple stack traces are arranged in a table three to a row so you can easily compare stack traces side-by-side. (Use the Ctrl key to select multiple rows, or the Shift key to select a range of rows.)

Stack Trace display

Drilldown by stack trace is the end of the line and does not permit further drilldown. (The DTrace stack() and ustack() actions are not allowed in a predicate.) Our DTrace program, after three levels of drilldown, looks like this:

program after drilldown by user stack

It would be a pain to type all of that, so you can see that drilldown adds a lot convenience! At the end we got per-second counts of gnome-panel reads on CPU 1 and CPU 2 broken down by user stack, having specified very little of that on the command line. We could have requested running totals instead using the -a option. (Run chime -h to see the complete set of available options.)

Adding the -w option saves the generated display so you can further improve it in the New Display Wizard, then add it to a Chime category so it can be run by selection from a list. (By default -w adds it to the New Displays category.) Currently you add a new category by manually creating a subdirectory of /opt/OSOL0chime/displays with a description.xml file, but eventually I hope to make that a task performed by the GUI. Creating categories with descriptive text for each display is the best way to organize useful traces and remember what they do, and make running them in the future a simple matter of point and click. (Run chime without options to access the list of categories.)

Automatic drilldown relieves you from having to specify many common drilldown options. Installed Chime displays that already supported drilldown continue to do so normally, but add all the options shown above (plus the by Executable option not shown in the example above because the original DTrace program was already aggregated by execname). You can still specify a drilldown option explicitly, and it will appear at the top of the list ahead of any automatic options (overriding an automatic option whose name or aggregation key matches that of the explicit option.)

Drilldown still requires placeholders in your DTrace program. Automatic drilldown generates those placeholders for you in programs specified on the command line (unless they're already present), but it does not do so for displays listed in the GUI. If you add a display to the GUI and want it to support drilldown, you need to ensure that your program has the expected placeholders, either by inserting them manually (the wizard has right click options to help you do that) or by saving a display generated on the command line with -w. To further remove the barrier of understanding Chime's placeholder syntax, another method, the -U option, was added to insert placeholders for you. For example:

    % chime -Un 'syscall:::entry { @[execname] = count(); }'
    { @[??T??] = count(); }

    ??T0 execname

Redirecting stdout to a file creates a file that is valid to specify with the -s option on the command line or to select in the wizard (it will even compile successfully in the wizard), or to specify with the programFileName property in the display description (selectable Chime displays are described in XML). Once you have a file with the required placeholders, turning on automatic drilldown in the wizard is as simple as clicking a check box in the Add Drilldown Support screen:

Wizard auto drilldown checkbox

Drilldown menu items can still be specified explicitly in this screen, and where they do not override an automatic option, they will appear in addition to the automatic menu items.

The Callouts screen in the Initial Displays category now supports drilldown (along with many other displays in that category that already supported it).

Chime main window

Here's one that's fun to demo: Select rwtop ... from the DTraceToolkit category (another display that supports drilldown). Notice that the drilldown options change depending on whether you start the display with the by User or by Zone button selected. Although by Project is not included in the popup menu, you can specify it manually with the Specify Drilldown ... option by entering curpsinfo->pr_projid in the input dialog. The ability to specify the drilldown variable on the fly should occasionally come in handy.

Post a Comment:
Comments are closed for this entry.



« July 2014

No bookmarks in folder