Friday Apr 16, 2010

Kernel and user profiling with dtrace

Just put together a short dtrace script for profiling both userland and kernel activity.

#!/usr/sbin/dtrace -s
#pragma D option quiet


profile-97
/arg1/
{
  @[pid,execname,ufunc(arg1)]=count();
}

profile-98
/arg0/
{
  @k[pid,execname,func(arg0)]=count();
}

tick-1s
{
  trunc(@,25);
  trunc(@k,25);
  printf("%5s %20s %20s %10s\\n","PID","EXECNAME","FUNC","COUNT");
  printa("%5d %20s %20A %10@d\\n",@);
  printa("%5d %20s %#20a %10@d\\n",@k);
  trunc(@);
  trunc(@k);
}

The script samples the current pc for both user land and kernel about 100x per second. There's some risk of over counting since there's one probe for user and one probe for kernel. Every second the code prints out the top 25 user and kernel routines, broken down by pid and executable name. The output looks like:

  PID             EXECNAME                 FUNC                  COUNT
  556                 Xorg libpixman-1.so.0`pixman_image_unref   1
  556                 Xorg libpixman-1.so.0`pixman_fill          1
  556                 Xorg libc.so.1`memcpy                      1
    0                sched unix`dispatch_softint                 1
    0                sched unix`dispatch_hardint                 2
    0                sched unix`mach_cpu_idle                    91

Tuesday Nov 27, 2007

Adding dtrace probes to user code (part 2)

Adam Leventhal pointed out in the comments to my post on adding dtrace userland probes that there is an improved approach to adding userland dtrace probes. He describes this approach on his blog.

The approach solves two problems. First, that C++ name mangling makes it hard to add dtrace probes for that language. Second, that code with dtrace probes inserted in it will not compile on systems that do not have the necessary dtrace support.

So going back to the example code, I'll try to show the problem and the solution. Here's app.cc:

#include <stdio.h>
#include <sys/sdt.h>

void func(int a, int b)
{
  DTRACE_PROBE2(myapp,func_call,a,b);
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

When compiled with the C compiler the following symbols get defined:

$ cc -c app.cc
$ nm app.o
app.o:

[Index]   Value      Size    Type  Bind  Other Shndx   Name
...
[10]    |         0|       0|FUNC |GLOB |0    |UNDEF  |__dtrace_myapp___func_call
...

When compiled with the C++ compiler the following happens:

$ CC -c app.cc
$ nm app.o
app.o:

[Index]   Value      Size    Type  Bind  Other Shndx   Name
...
[7]     |         0|       0|FUNC |GLOB |0    |UNDEF  |__1cbA__dtrace_myapp___func_call6FLL_v_
...

Because the call to the dtrace probe is not declared as being extern 'C' the compiler mangles the C++ function name.

The new approach that Adam describes involves dtrace preprocessing the probe description file to generate a header file, and the including the header file in the source code. The big advantage of having the header file is that it's now possible to declare the dtrace probes to have extern 'C' linkage, and avoid the name mangling issue. The syntax for preprocessing the probe description file is:

$ dtrace -h -s probes.d

This generates the following header file:

/\*
 \* Generated by dtrace(1M).
 \*/

#ifndef _PROBES_H
  #define _PROBES_H

  #include <unistd.h>

  #ifdef  __cplusplus
    extern "C" {
  #endif

  #if _DTRACE_VERSION
    #define MYAPP_FUNC_CALL(arg0, arg1)  __dtrace_myapp___func_call(arg0, arg1)
    #define MYAPP_FUNC_CALL_ENABLED()    __dtraceenabled_myapp___func_call()
    extern void __dtrace_myapp___func_call(int, int);
    extern int __dtraceenabled_myapp___func_call(void);
  #else
    #define MYAPP_FUNC_CALL(arg0, arg1)
    #define MYAPP_FUNC_CALL_ENABLED() (0)
  #endif

  #ifdef  __cplusplus
    }
  #endif

#endif  /\* _PROBES_H \*/

The other advantage is that the header file can protect the definitions of the dtrace probes with #if _DTRACE_VERSION; which enables the same source to be compiled on systems which do not support dtrace.

The source code needs to be modified to support this syntax:

#include &kt;stdio.h>
#include "probes.h"

void func(int a, int b)
{
  MYAPP_FUNC_CALL(a,b);
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

The rest of the process is the same as before.

Tuesday Nov 20, 2007

Adding dtrace probes to user code

The process of adding dtrace probes to userland code is described in the dynamic tracing guide. However, there's no better way of learning how to do it, than trying it out on a snippet of code.

Here's a short bit of code that calls a function twice, each time with different parameters. The plan is to insert a probe that can report the passed parameters.

#include <stdio.h>

void func(int a, int b)
{
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

The first change is to add the <sys/sdt.h> header file. This file has definitions for the DTRACE_PROBE<N> macro. N represents the number of parameters that are to be reported by the probe. In this case we are going to pass two parameters (a and b) to the probe. As well as the parameters that are to be passed to the dtrace probe, the macro takes the name to be used tof the application provide (in this case the name will be myapp) and the name of the probe (in this case func_call). The modified source code looks as follows:

#include <stdio.h>
#include <sys/sdt.h>

void func(int a, int b)
{
  DTRACE_PROBE2(myapp,func_call,a,b);
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

The next step is to write a probe description file which dtrace will use to produce the probes. A full file would describe the stability of the probe in more detail, but a lightweight file just describes the probes defined by the provider application:

provider myapp
{
  probe func_call(int, int);
};

Having completed this, it's necessary to compile and link the application. Initially each source file needs to be compiled, and then before the application is linked, dtrace needs to be invoked to modify the object files, removing the calls to the probes, but leaving space for them to be reinserted. dtrace also needs to compile the probe description file into an object file. Finally the modified object files and the probe description file can be linked to produce the executable. As follows:

$ cc -c app.c
$ dtrace -G -32 -s probes.d app.o
$ cc probes.o app.o

The resulting code in the application looks like:

func()
        113a0:  9d e3 bf a0  save       %sp, -96, %sp
        113a4:  f0 27 a0 44  st         %i0, [%fp + 68]
        113a8:  f2 27 a0 48  st         %i1, [%fp + 72]
        113ac:  d0 07 a0 44  ld         [%fp + 68], %o0
        113b0:  01 00 00 00  nop
        113b4:  d2 07 a0 48  ld         [%fp + 72], %o1
        113b8:  11 00 00 45  sethi      %hi(0x11400), %o0
        113bc:  90 12 22 60  bset       608, %o0        ! 0x11660
        113c0:  d2 07 a0 44  ld         [%fp + 68], %o1
        113c4:  40 00 42 c7  call       printf  ! 0x21ee0
        113c8:  d4 07 a0 48  ld         [%fp + 72], %o2
        113cc:  81 c7 e0 08  ret
        113d0:  81 e8 00 00  restore

The nop at 0x113b0 is there for dtrace to dynamically patch with a call instruction that will enable the dtrace probe.

Finally, the following is an example of using the new probe:

$ more script.d
myapp$target:::func_call
{
  @[arg0,arg1]=count();
}
$ dtrace -s script.d -c a.out
dtrace: script 'script.d' matched 1 probe
a=1, b=2
a=2, b=3
dtrace: pid 22355 has exited

                1                2                1
                2                3                1

The script just aggregates the parameters used in the function call. When the application terminates the aggregation is printed out - showing the expected result of two calls to the routine each call with different parameters.

About

Darryl Gove is a senior engineer in the Solaris Studio team, working on optimising applications and benchmarks for current and future processors. He is also the author of the books:
Multicore Application Programming
Solaris Application Programming
The Developer's Edge

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
5
6
8
9
10
12
13
14
15
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
The Developer's Edge
Solaris Application Programming
Publications
Webcasts
Presentations
OpenSPARC Book
Multicore Application Programming
Docs