Monday Jan 10, 2011

Chime 1.5

I recently announced version 1.5 of the Chime Visualization Tool for DTrace. One of my favorite changes in this update is not listed in that announcement: Chime now loads the last selected trace category at startup.

Last used Chime category selected at startup

What this means is that Chime feels more like your own toolbox, since it loads the traces you were last using. You can customize Chime by making your own category and populating it with the traces you use most. Simply create a directory anywhere under /opt/OSOL0chime/displays/ and copy into it just the traces you want. For example:

% mkdir /opt/OSOL0chime/displays/favorites
% cp /opt/OSOL0chime/displays/syscall.xml /opt/OSOL0chime/displays/favorites
% cp /opt/OSOL0chime/displays/system_probes.xml /opt/OSOL0chime/displays/favorites/
% cp /opt/OSOL0chime/displays/toolkit/vopstat.\* /opt/OSOL0chime/displays/favorites/
% cp /opt/OSOL0chime/displays/toolkit/procsystime.\* /opt/OSOL0chime/displays/favorites/

Then copy /opt/OSOL0chime/displays/toolkit/description.xml into that same directory and edit the "name" and "description" properties to match what you'd like to appear in the Category list and the Description text pane, respectively (see picture above). For example:

% cp /opt/OSOL0chime/displays/toolkit/description.xml /opt/OSOL0chime/displays/favorites/
% vi /opt/OSOL0chime/displays/favorites/description.xml

Then you might change
  <void property="name">
  <void property="description">
   <string>Traces adapted from scripts&lt;br&gt;in the DTraceToolkit.


  <void property="name">
  <void property="description">
   <string>Frequently used traces.</string>

When you next run Chime, you would find Favorites in the Category list. After selecting it, assuming you never changed it, Chime would always start up like this:

Custom Chime category loaded at startup

You can also specify a directory with 'chime -d', in case you want commands for several different customizations. It's easy to create your own traces to populate your custom categories. Give it a try!

Tuesday Jun 30, 2009

Chime Automatic Drilldown

I think you'll like this latest update to Chime. Drilldown has been a feature of Chime since its initial release. It lets you see more detail about selected items of interest, for example a breakdown by function of a counter for a particular executable. Drilldown supports some of the dynamic capability of dtrace(1m) by launching a new display to answer a question prompted by the current display. However, even though drilldown is easy to use once enabled (simply right-click on an item of interest), it has been difficult to enable in new displays and downright tedious if you wanted to specify multiple levels of drilldown. The feature has always required the addition of placeholders in your DTrace program and an explicit description of every drilldown option you wanted to support.

This latest update removes some of the barriers to enabling drilldown in new displays, and even supports drilldown automatically for displays generated on the command line using the -n and -s options. For example:

    % chime -ktn 'syscall:::entry { @[execname] = count(); }' -T "System Calls"

To try this (Chime is available for download here), you'll first want to add /opt/OSOL0chime/bin to your PATH, or else add an alias to your shell startup script: 'alias chime=/opt/OSOL0chime/bin/chime' so you can use the command 'chime' as conveniently as you use the command 'dtrace'. The -n option specifies the DTrace program, just as it does with the dtrace command. The -k option says you want the display to include a sparkline of values over time. The -t option says you want display totals, and the -T option sets the title bar to something more informative than the default title "Display". There's nothing new here, except that now you get drilldown automatically. Run the above command to see how this works:

System Calls drilldown

The command produces a simple count of system calls per second for each executable. Imagine a mouse pointer at the top left corner of the popup menu over the highlighted "gnome-panel" item. Right-clicking brings up a long list of drilldown options labeled "gnome-panel by <something>". Selecting "... by CPU" modifies the DTrace program as follows:

program before drilldown by cpu program after drilldown by cpu

The aggregation key has been moved to the predicate and filters out all but the selected value. The new aggregation key, cpu, replaces execname in the square brackets. You can drilldown still further:

System Calls by CPU

A few things to notice here: First, the by CPU option no longer appears, since we already aggregated by cpu in the previous drilldown, and doing so again would not further expose any useful detail. Also, you can break down the sum of multiple items, rather than just a single item. In the example above, both CPU 1 and CPU 2 are selected, so breakdown by function looks like this:

program after drilldown by function System Calls by Function

Next, we could select the "read" function and request a breakdown by User Stack:

System Calls by User Stack

The Caller field shows the top of the stack, and the User Stack field shows a tag to identify the full stack trace. To see those stack traces, right-click one or more selected items and choose View Selected Stack Traces from the popup menu. Changing the row selection automatically updates the stack trace display. Multiple stack traces are arranged in a table three to a row so you can easily compare stack traces side-by-side. (Use the Ctrl key to select multiple rows, or the Shift key to select a range of rows.)

Stack Trace display

Drilldown by stack trace is the end of the line and does not permit further drilldown. (The DTrace stack() and ustack() actions are not allowed in a predicate.) Our DTrace program, after three levels of drilldown, looks like this:

program after drilldown by user stack

It would be a pain to type all of that, so you can see that drilldown adds a lot convenience! At the end we got per-second counts of gnome-panel reads on CPU 1 and CPU 2 broken down by user stack, having specified very little of that on the command line. We could have requested running totals instead using the -a option. (Run chime -h to see the complete set of available options.)

Adding the -w option saves the generated display so you can further improve it in the New Display Wizard, then add it to a Chime category so it can be run by selection from a list. (By default -w adds it to the New Displays category.) Currently you add a new category by manually creating a subdirectory of /opt/OSOL0chime/displays with a description.xml file, but eventually I hope to make that a task performed by the GUI. Creating categories with descriptive text for each display is the best way to organize useful traces and remember what they do, and make running them in the future a simple matter of point and click. (Run chime without options to access the list of categories.)

Automatic drilldown relieves you from having to specify many common drilldown options. Installed Chime displays that already supported drilldown continue to do so normally, but add all the options shown above (plus the by Executable option not shown in the example above because the original DTrace program was already aggregated by execname). You can still specify a drilldown option explicitly, and it will appear at the top of the list ahead of any automatic options (overriding an automatic option whose name or aggregation key matches that of the explicit option.)

Drilldown still requires placeholders in your DTrace program. Automatic drilldown generates those placeholders for you in programs specified on the command line (unless they're already present), but it does not do so for displays listed in the GUI. If you add a display to the GUI and want it to support drilldown, you need to ensure that your program has the expected placeholders, either by inserting them manually (the wizard has right click options to help you do that) or by saving a display generated on the command line with -w. To further remove the barrier of understanding Chime's placeholder syntax, another method, the -U option, was added to insert placeholders for you. For example:

    % chime -Un 'syscall:::entry { @[execname] = count(); }'
    { @[??T??] = count(); }

    ??T0 execname

Redirecting stdout to a file creates a file that is valid to specify with the -s option on the command line or to select in the wizard (it will even compile successfully in the wizard), or to specify with the programFileName property in the display description (selectable Chime displays are described in XML). Once you have a file with the required placeholders, turning on automatic drilldown in the wizard is as simple as clicking a check box in the Add Drilldown Support screen:

Wizard auto drilldown checkbox

Drilldown menu items can still be specified explicitly in this screen, and where they do not override an automatic option, they will appear in addition to the automatic menu items.

The Callouts screen in the Initial Displays category now supports drilldown (along with many other displays in that category that already supported it).

Chime main window

Here's one that's fun to demo: Select rwtop ... from the DTraceToolkit category (another display that supports drilldown). Notice that the drilldown options change depending on whether you start the display with the by User or by Zone button selected. Although by Project is not included in the popup menu, you can specify it manually with the Specify Drilldown ... option by entering curpsinfo->pr_projid in the input dialog. The ability to specify the drilldown variable on the fly should occasionally come in handy.

Friday Dec 19, 2008

Chime Python Category

I just added a Python category to Chime, in addition to the Ruby category added earlier.

list of Python displays in main Chime window

A PHP category is coming soon. These are all scripts from the DTraceToolkit. Chime has gotten pretty flexible in terms of supporting script options, so I was able to merge the py_cpudist.d and py_cputime.d scripts into py_calldist.d and py_calltime.d, which differ only in measuring on-CPU time instead of elapsed time.

dialog prompting for elapsed time or on-CPU time

The prompt defaults to elapsed time. Here's a trace of a Mercurial command:

py_calltime display

It shows time in two ways side-by-side: the entire time from entry to return including time spent in other functions, and time in the current stack frame excluding time spent calling other functions. Here are some other displays showing total bytes allocated alongside a malloc() size distribution, by function and by stack trace.

py_malloc display py_mallocstk display

The tagged stack traces are viewable by accessing a right-click popup menu on one or more selected rows.

selected stack traces side by side in the stack viewer

So have fun tracing your Python apps! Chime is downloadable here.

Tuesday Dec 16, 2008

Chime Data Scrolling

As I was scrolling through output of the dtrace command recently, it struck me that the command line allowed me to do something I could not do with Chime: look back at recent data. It bothered me to see Chime at a disadvantage in terms of visibility, where it actually tries to excel over the command line.

To remedy this, I added controls to provide the equivalent of scrolling in a command terminal. Since Chime already has a scrollbar to view whatever part of the current aggregate snapshot doesn't fit in the window, it didn't make sense for a scrollbar to access previous snapshots as well. Instead I added forward and back buttons next to the pause button in the lower right corner.

System Calls with back and pause buttons enabled and forward button disabled

The forward button is disabled because we're normally looking at the most recent aggregate snapshot. I replaced the text of the pause button with an icon so that the button no longer changes width when switching the text from "Pause" to "Resume" and back. Changing the button width shifts the location of the forward and back buttons by the difference in width, whereas switching same-size icons leaves the buttons in place and also looks cleaner.

Clicking the back button automatically pauses the display and steps it to the previous aggregate snapshot.

System Calls with mouse pointer over back button and back, forward, and pause buttons enabled

Clicking the back button repeatedly moves the spark backward on the sparkline until reaching the oldest snapshot in the recent data cache.

System Calls with mouse pointer over disabled back button; forward and pause buttons enabled

I chose ten as the default limit of the recent data cache, since I didn't want to consume a lot of memory, and the most recent snapshots are usually all you want to see. If you want to be able to scroll back farther, you can set the number of steps in the recent data cache using the Options menu.

Main Chime Window with Set Recent Data Count ... selected in Options menu Set Recent Data Count dialog launched by menu item

While the display is paused, you can step forward and back through recent aggregate snapshots in a temporary copy of the cache. The actual cache continues to add new snapshots at the front and roll old snapshots off the back as data comes in. To resume the display of current data, click the pause/resume button (displaying the resume icon while paused).

Another convenience of the command line is that in the case of data drops, you can easily change option values and re-run the command. That convenience is next on my list of things to add to Chime.

Get the latest version of Chime (1.4.39) to try out the forward and back buttons (link here). The latest version also includes a large number of scripts from the DTraceToolkit, including a new Ruby category.

Tuesday Oct 16, 2007

Chime In Your App

Several people have asked me if there is a way to drop a Chime display into their own application. For example, a Chime plugin for Netbeans might be nice (code, anyone?). It's not hard to drop a Chime display into any Java app. For those who want to try, here's an example: [ view]

To compile and run the example (on Solaris Nevada build 35 or later), first install Chime, then enter the following commands (syscall.xml comes installed with Chime, although you can specify any valid display description):

; javac -cp /opt/OSOL0chime/lib/java/chime.jar:/usr/share/lib/java/dtrace.jar  \\
; java -cp .:/opt/OSOL0chime/lib/java/chime.jar:/usr/share/lib/java/dtrace.jar \\
TestChime /opt/OSOL0chime/displays/syscall.xml

TestChime app

Voila! This test app creates its own frame and drops a Chime display into it. Here is the significant code that generates the display and adds it to panel p:

    DisplayDescription d = AggregationDisplay.getDescriptionFromFile(file);
    try {
    } catch (Exception e) {

    Command c = AggregationDisplay.createCommand(d);
    final AggregationDisplay display = new AggregationDisplay(c, d);
    p.add(display.getDisplay(), BorderLayout.CENTER);

As in the example, you can provide your own controls that are more suitable for your app than the "Pause" button and "Interval in seconds" spinner that appear at the bottom of a Chime display. Here are the AggregationDisplay instance methods to do that:

    setPaused(boolean paused)
    boolean isPaused()
    setIntervalMillis(long millis)
    long getIntervalMillis()
    java.util.List <JMenuItem> menuItems()

The last method supplies the items for Chime's right-click popup menu. Chime interprets a null list element as a menu item separator, so your code needs to handle null list elements. From each JMenuItem, you can call getAction() to supply the action for the control you want to provide and getName() to label that control. Anything beyond that you can figure out from the Chime source code, starting with (which implements all of Chime's command-line options).

Many of Chime's menu items invoke actions that run in a new window. For example, drilldown displays, plots over time, and Chime's DTrace program viewer all run in a new window. Chime also prompts for DTrace macro arguments in a dialog box:

Chime macro arguments prompt

Currently (as of Chime version 1.4.29), if you need these features to run within your application frame, there is no easy way to do it: you will need to work with the Chime source code. If you do, please share your work to help the Chime project!

Tuesday Oct 09, 2007

Chime Sparklines

Continuing in the same vein as my previous blog entry, Chime Non-Repeating Keys, I've added another option for generating Chime displays on the command line.

    [-k add a sparkline to the first plottable data column]

Chime has had the ability to display sparklines since its initial OpenSolaris release, as demonstrated by the System Calls display that comes installed with Chime:

System Calls sparklines

The History column provides a view of the System Calls Per Second data over time. It scales from zero to max for each executable. Because the series uses a different vertical scale for each plot it cannot be compared visually as one might expect. If the same vertical scale was used for all executables, the plots would flatten out such that individual variation would no longer be visible. Instead, relative vertical scale is indicated by the cyan line, which is positioned farther to the right the higher the maximum value of the aggregation key. If you click on the History column header, it will sort the column in descending order by maximum value (cyan indicators farthest to the right will move to the top of the table).

The tooltip displays
  1. how much time has elapsed since the first non-zero data was plotted
  2. the maximum value defining the vertical scale of the plot
The tooltip 21 seconds: min 0 .. max 11,814 appeared when resting the mouse pointer (not shown) over the Xorg sparkline; a similar tooltip will appear as the mouse pointer moves across other sparklines.

To generate such a display directly from the command line:

% /opt/OSOL0chime/bin/chime -kTn 'syscall:::entry { @[execname] = count(); }'

Chime sparklines generated with -k option

The -k option adds the History column displaying the values from the @ aggregation over time. The -T option adds a total row. To change the labels in the title bar and column headers to match the System Calls display, enter the following command instead:

% /opt/OSOL0chime/bin/chime -kTn 'syscall:::entry \\
{ @[execname] = count(); }' -t "System Calls" \\
-h "Executable, History, System Calls Per Second"

To add thousands separators and change total units from "keys" to "executables", you will need to set properties in the wizard. Add -w to the previous command to save the display, then reopen it in the wizard with -W:

% /opt/OSOL0chime/bin/chime -kTn 'syscall:::entry \\
{ @[execname] = count(); }' -t "System Calls" \\
-h "Executable, History, System Calls Per Second" -w
Wrote /opt/OSOL0chime/displays/new/system_calls.xml
To run, enter "/opt/OSOL0chime/bin/chime -C /opt/OSOL0chime/displays/new/system_calls.xml".
% /opt/OSOL0chime/bin/chime -C \\ 
/opt/OSOL0chime/displays/new/system_calls.xml -W

Chime wizard

Click Next until you reach step 4. Specify Columns.

Chime wizard, step 4. Specify Columns

Select the "Executable" column and click the Specify >> button to replace "keys" with "executables". Next, select the "System Calls Per Second" column and click the Specify >> button. Check the Value Per Second check box and select #,##0 from the Format list to request thousands separators. Click Next until you reach step 5. Test Run the Display, then click the Run Display button:

New System Calls Test Run

Close the display and click the Finish button. The next time you run Chime without any options, the finished "System Calls" display will be selectable from the "New Displays" Trace Group in the main window.

Sparklines in Chime rely on a custom renderer that you can get from the Chime source code (see src/org/opensolaris/chime/swingx/ Chime uses JFreeChart to provide larger plots of selected values. For example, in the display just created, select Xorg, Java, and mixer_applet2 by holding down the Ctrl key while left-clicking the desired values in the table.

Multiple row selection

If values are moving targets because the current sort keeps changing the row under the mouse pointer, click the Pause button first (the button label will change to Resume; click the button again when you're ready). Right click anywhere on the selection to bring up the popup menu and select Plot Over Time:

Multiple row selection

Values already accumulated in the selected sparklines are displayed in the line graph that appears. The plot accumulates new data (scrolling to the left) after the initial capture as long as the original table remains open.

Friday Sep 28, 2007

Chime Non-Repeating Keys

Someone recently asked me if there was a way to make DTrace display non-repeating aggregation keys. He wanted something like this:

% dtrace -qn 'syscall:::entry { @[execname, probefunc] = count(); }' -n \\
'tick-1sec { printf("\\n%25s %25s %12s\\n", "Executable", "Function", \\
"Count"); printf("%25s %25s %12s\\n", "----------", "--------", "-----"); \\
printa("%25s %25s %@12d\\n", @); }' -xaggsortkey

               Executable                  Function        Count
               ----------                  --------        -----
                     Xorg             clock_gettime           18
                                        lwp_sigmask            4
                                            pollsys            9
                                               read            8
                                         setcontext            4
                                          setitimer            8
                                             writev            1
                   dtrace                       brk            6
                                              ioctl           77
                                           lwp_park            1
                                               mmap            1
                                           p_online           64
                                           schedctl            1
                                          sigaction            4
                                          sysconfig            3
              firefox-bin                  lwp_park            8
         gnome-netstatus-                     ioctl           11
                                            pollsys            5
                                               read            2
                                              write            2
           gnome-terminal                     ioctl            4
                                            pollsys            4
                                              write            2
         gnome-vfs-daemon                   pollsys            1
                                             stat64            2
                    inetd                  lwp_park            1
                     java                     ioctl           22
                                      lwp_cond_wait            2
                                            pollsys           45
                                              yield            9
            mixer_applet2                     ioctl           27
                                            pollsys           17

I told him I didn't think so. (It might make a nice option for dtrace(1M).) I also told him that Chime can display non-repeating keys. I wanted to demonstrate using his DTrace script, but I realized he would have to sit patiently while I opened the wizard and fiddled with display properties, so I didn't bother. Later it occurred to me that a command line option for non-repeating keys would have made the demonstration simple. Hence -g (for group):

% /opt/OSOL0chime/bin/chime -n 'syscall:::entry { \\
@[execname, probefunc] = count(); }' -g

Chime non-repeating keys

I was pretty happy about that. For one thing, the wizard makes non-repeating keys a little tricky to specify, since you have to list two columns, one for the data and another for a non-repeating view of that data. The -g option does this for you, giving you one less thing to worry about in the wizard later on if you do decide to make further improvements.

In case you want lines to separate your non-repeating groups, you can use -G instead of -g:

% /opt/OSOL0chime/bin/chime -n 'syscall:::entry { \\
@[execname, probefunc] = count(); }' -G

Chime key separator lines

Anything you specify on the command line can be saved using -w. So the more you can specify directly on the command line (thanks to new options like -g and -G), the more -w can do for you. Other things you can specify on the command line include the title and column headers (with -t and -h respectively); see Chime's Command Line page for details. For this example just save the display with non-repeating keys:

% /opt/OSOL0chime/bin/chime -n 'syscall:::entry { \\
@[execname, probefunc] = count(); }' -gw
Wrote /opt/OSOL0chime/displays/new/display.xml
To run, enter "/opt/OSOL0chime/bin/chime -C

Then you can tweak things in the wizard:

/opt/OSOL0chime/bin/chime -C /opt/OSOL0chime/displays/new/display.xml -W

The -C option (to specify a saved Chime display) works in combination with the -W option (for Wizard) on or after version 1.4.28 (29 Sep 2007). This lets you cut and paste from the "To run..." message in the command output above and simply tack on a -W. Before 1.4.29 the wizard came up in its initial state, requiring you to find the display with the Browse ... button.

Chime Wizard

Pretty convenient, so I thought why stop there? It might be nice if a total row could be specified just as easily. (All these things you can do in the wizard, but you can't beat the command line for instant gratification.) The result is the -T option, which generates a total for every column in your display:

% /opt/OSOL0chime/bin/chime -n 'syscall:::entry { \\
@[execname, probefunc] = count(); }' -gT

Chime Totals

If you want nicer labels like "23 executables" rather than the generic "23 keys", you can specify that in the wizard, but for a one-off, it's nice just to have the numbers without any fuss.

Again, you can rerun the command with -w to save the display, then tweak it in the wizard with -W. If you click the Next button in the wizard, you'll find your DTrace program already specified in step 2. If you continue to step 4, you can see the hidden column and the non-repeating view of the execname:

Chime Wizard Columns

The green check marks indicate that all the required column properties are already specified (although you might wish to specify others, such as alignment, thousands separator, etc). For more information about what you can do in the wizard, see Chime's New Display Wizard page.



« April 2014

No bookmarks in folder