DTrace and Visualisation

As I talk to people about DTrace one of the most frequent requests that arises is for scripts and tools. On the scripts front people want cookbook type stuff as it is always easier to learn a new technology such as DTrace by "re-using" and building upon pre-prepared material. This is an excellent request that, hopefully, Sun and the ever growing band of DTrace addicts will fulfill in the near future.

On the tools front a request that is made is for a DTrace GUI. In the early days of DTrace (about a year or so ago...) my usual base reaction to such a request was to reply that, whilst I never say never, I doubt that this is high on the priority list but I'll feed it back ... However, age and thought has softened me tremendously and I now feel that the union of DTrace and Visualisation techniques is a source of terrific innovation and will be of immense benefit to all.

With DTrace we have the infrastructure to ask almost arbitrary questions about the behaviour of our software stacks from the top of an application through to the bottom of a bus nexus driver. However, some of the data that these complex subsystems generate can be rich in nature. This is not a size issue particularly as DTrace has great primitives for data reduction but more referring to the relationships that exist within the data. Sometimes a bit of visual stimulation is required to get our synapses firing and to force our grey cells into seeing patterns and relationships in the data. I'm not knocking DTrace here; on the contrary I'm saying that DTrace is the nearly infinitely rich architecture with which we can feed the visualisation engines. I'm convinced that in the near future we will see some real innovation in the tools around systemic and application observability as the data is so easily at hand.

For example, take something that a colleague ( Simon Ritter ) and and I have been working on in our copious spare time, the area of call graphs. With DTrace we can trivially discover what functions a given function is calling and in it's simplest form the script looks like:
#usr/sbin/dtrace -s

#pragma D option flowindent

    self->in = 1;

/self->in && probefunc != "$1"/

    self->in = 0;
So, a trivial example would be to look at what activity the clock routine generates:
# ./fbt.d clock
dtrace: script './fbt.d' matched 31751 probes
CPU FUNCTION                                 
  0  -> clock                                 
  0    -> set_anoninfo                        
  0    <- set_anoninfo                        
  0    -> set_freemem                         
  0    <- set_freemem                         
  0    -> wakeup_pcgs                         
  0    <- wakeup_pcgs                         
  0    -> clock_tick                          
  0      -> sys_nullsys                       
  0      <- sys_nullsys                       
  0      -> mutex_owned                       
  0      <- mutex_owned                       
  0      -> rctl_test                         
  0      <- rctl_test                         
  0      -> rctl_test_entity                  
  0        -> mutex_owned                     
  0        <- mutex_owned                     
  0        -> rctl_set_find                   
  0          -> mutex_owned                   
  0          <- mutex_owned                   
  0        <- rctl_set_find                   
  0      <- rctl_test_entity                  
  0      -> rctl_test                         
  0      <- rctl_test                         
  0      -> rctl_test_entity                  
  0        -> mutex_owned                     
  0        <- mutex_owned                     
  0        -> rctl_set_find                   
  0          -> mutex_owned                   
  0          <- mutex_owned                   
  0        <- rctl_set_find                   
  0      <- rctl_test_entity                  
  0      -> rm_asrss                          
  0      <- rm_asrss                          
  0    <- clock_tick                          
  0    -> callout_schedule                    
  0      -> callout_schedule_1                
  0      <- callout_schedule_1                
  0      -> softcall                          
  0        -> mutex_vector_enter              
  0        <- mutex_vector_enter              
  0        -> mutex_vector_exit               
  0        <- mutex_vector_exit               
  0      <- softcall                          
  0      -> siron                             
  0      <- siron                             
  0      -> setsoftint                        
  0      <- setsoftint                        
  0      -> callout_schedule_1                
  0      <- callout_schedule_1                
  0    <- callout_schedule                    
  0   | clock:return                          
  0  <- clock                                 

However, as great as it is to see the above, the shape of the call sequence isn't very apparent and that is a simple routine. Using something such as a ufs_write() you soon see how this gets very complex. It is much better to draw a directed graph of the calls and you can immediately see what is being called by what. An example of a single ufs_write() call can be seen in this image . The image was generated using DTrace to feed a public domain piece of software called JzGraph which is a great Java implementation of the wonderful Graphviz algorithms.

However, the above is very static in its nature and we wanted to generate something much more dynamic. What we have done is to create a Java application which is a dtrace consumer (i.e it links a JNI library which uses libdtrace(3LIB)). Therefore we can interact with DTrace at will and be much more dynamic in the way we present and manipulate the data. The current version of our application allows you to dynamically generate graphs for any kernel function, step through the call sequence, zoom in and out of graphs, collapse and expand nodes, display timing data, display source code for the nodes in a graph (amongst other things).

We have many other ideas of what to do and hopefully we'll get around to them soon. Sun is developing some Java bindings for DTrace and they're looking really good at the moment (see here for Bryans comment on this).

I for one am looking forward to a future in which complex systemic interactions will be visible to all as the data is now readily available. It has to be seen to be believed.

Nice work jon!

Posted by Jon Anderson on August 27, 2004 at 02:38 AM GMT+00:00 #

Howdy, Would it be possible to include a "shared" directory in Solaris 10 with commonly used dscripts? A set of out of the box examples would be helpful. Also, any thoughts of integrating dtrace into the visual debuggers? That could be rather helpful :) Thanks, - Ryan

Posted by Ryan Matteson on August 28, 2004 at 06:21 AM GMT+00:00 #

Ryan - apologies for the delay in this reply. I'm not sure exactly what the current plan of action is regarding distribution of examples. I'll ask the chaps and see what they say. Regarding the studio tools - a kernel profiling tool has been written in DTrace so you can use the analyzer on a kernel profile (however, I'm not sure whether or not that will be productised). Let us know if there are enhancements that you'd like to see that you think DTrace might bring to that particular party. Cheers, Jon.

Posted by Jon Haslam on September 01, 2004 at 03:36 PM GMT+00:00 #

Post a Comment:
Comments are closed for this entry.



« April 2014