Monday Dec 30, 2013

Privileges

Before I start, this is not about security, it's probably the antithesis of security. So I'd recommend starting by reading about how using privileges can break the security of your system.

There are three tools that I regularly use that require escalated privileges: dtrace, cpustat, and busstat. You can read up on the way that Solaris manages privileges. But if you know what you want to do, the process to figure out how to get the necessary privileges is reasonable straightforward.

To find out what privileges you have you can use the ppriv -v $$ command. This will report all the privileges for the current shell.

To find out what privileges are stopping you from running a command, you should run it under ppriv -eD command. For example:

ppriv -eD cpustat -c instruction_counts 1 1
cpustat[13222]: missing privilege "sys_resource" (euid = 84945, syscall = 128) needed at rctl_rlimit_set+0x98
cpustat[13222]: missing privilege "cpc_cpu" (euid = 84945, syscall = 5) needed at kcpc_open+0x4
...

It is also possible to list all the privileges on the system using ppriv -l. This is helpful if the privilege is has a name that maps onto what you want to do. The privileges for dtrace are good examples of this:

$ ppriv -l|grep dtrace
dtrace_kernel
dtrace_proc
dtrace_user

You can then use usermod -K ... to assign the necessary privileges to a user. For example:

$ usermod -K defaultpriv=basic,sys_resource,cpc_cpu username

Information about privileges for users is recorded in /etc/user_attr, so it is possible to directly edit that file to add or remove privileges.

Using this approach you can determine that busstat needs sys_config, cpustat needs sys_resource and cpc_cpu, and dtrace needs dtrace_kernel, dtrace_proc, and dtrace_user.

Wednesday Jun 09, 2010

Runtime analysis in the Solaris Studio IDE

I was pleasantly surprised to find support for runtime analysis embedded in the Solaris Studio IDE. This analysis uses the Performance Analyzer to gather data as the code is running and then presents this data both as timeline views over the runtime of the application, and also source code annotations. Here's the view as the data is gathered.


The tool gathers profile data which is shown as an aggregation of time spent in each routine, and also annotated against each line of source.


The other thing the tool is able to track is memory leaks, again reporting the amount leaked, and attributing the leaks to the lines of source where the data was allocated.


Friday Apr 16, 2010

Kernel and user profiling with dtrace

Just put together a short dtrace script for profiling both userland and kernel activity.

#!/usr/sbin/dtrace -s
#pragma D option quiet


profile-97
/arg1/
{
  @[pid,execname,ufunc(arg1)]=count();
}

profile-98
/arg0/
{
  @k[pid,execname,func(arg0)]=count();
}

tick-1s
{
  trunc(@,25);
  trunc(@k,25);
  printf("%5s %20s %20s %10s\\n","PID","EXECNAME","FUNC","COUNT");
  printa("%5d %20s %20A %10@d\\n",@);
  printa("%5d %20s %#20a %10@d\\n",@k);
  trunc(@);
  trunc(@k);
}

The script samples the current pc for both user land and kernel about 100x per second. There's some risk of over counting since there's one probe for user and one probe for kernel. Every second the code prints out the top 25 user and kernel routines, broken down by pid and executable name. The output looks like:

  PID             EXECNAME                 FUNC                  COUNT
  556                 Xorg libpixman-1.so.0`pixman_image_unref   1
  556                 Xorg libpixman-1.so.0`pixman_fill          1
  556                 Xorg libc.so.1`memcpy                      1
    0                sched unix`dispatch_softint                 1
    0                sched unix`dispatch_hardint                 2
    0                sched unix`mach_cpu_idle                    91

Thursday Sep 11, 2008

Times for CMG workshop

The time and location information for the CMG workshop are up on the website.

Sunday Workshop
Sunday 8:30 AM - 12:00 PM
Room: Concorde C

Friday Aug 29, 2008

Using dtrace to locate floating point traps

The easiest way to check whether a system is handling floating point traps is to use kstat:

$ kstat |grep fpu_unfinished
        fpu_unfinished_traps            178164

or

$ kstat -s fpu_unfinished_traps
module: unix                            instance: 0     
name:   fpu_traps                       class:    misc                          
        fpu_unfinished_traps            178164

This reports the number since boot time, so to see if traps are happening, you need to run the command twice and look at the difference.

An alternative way of doing this is to use dtrace to count the number of traps that occur:

$ dtrace -n fbt:genunix:_fp_fpu_simulator:entry'{@a[probefunc]=count();}'
dtrace: description 'fbt:genunix:_fp_fpu_simulator:entry' matched 1 probe
\^C

  _fp_fpu_simulator                                            57050 

Then its easy to record the pid of the processes generating the traps together with their frequency:

dtrace -n fbt:genunix:_fp_fpu_simulator:entry'{@a[pid]=count();}'
dtrace: description 'fbt:genunix:_fp_fpu_simulator:entry' matched 1 probe
\^C

    1686                4
    1526            19670
    1589            20924 

The final thing to do is to look at the function names:

$ dtrace -n fbt:genunix:_fp_fpu_simulator:entry'{@a[ustack(1)]=count();}'
dtrace: description 'fbt:genunix:_fp_fpu_simulator:entry' matched 1 probe
\^C


              app`func1
             13426 

Thursday Mar 27, 2008

Performance visualization discussion group

Saw this on Neil Gunther's blog, a discussion group for performance visualization. I've no idea whether it will be an interesting group or not, but there's certainly many possibilities for making performance data more readily understandable. So far there look to be about 12 folks signed up to the group.

One visualization tool that I think shows promise is chime, which is built upon dtrace.

Tuesday Mar 25, 2008

Adding dtrace probes to user code (part 3)

I've previously discussed how to add dtrace USDT probes into user code. The critical step is to run the object files through dtrace, for dtrace to record the instrumentation points and to modify the object files prior to linking. The output of this step is an object file that also needs to be linked into the executable. Here's an example:

$ cc -O -c app.c
$ cc -O -c app1.c
$ dtrace -G -32 -s probes.d app.o app1.o
$ cc -O probes.o app.o app1.o

The results from running the example code under a suitable dtrace script are:

$ sudo dtrace -s script.d -c a.out
dtrace: script 'script.d' matched 10 probes
a=1, b=2
a=1, b=2
a=1, b=2
a=2, b=3
dtrace: pid 20655 has exited

                2                3                1
                1                2                3

One question that has come up is whether it's necessary to run a single call to dtrace which instruments all the object files, or whether it's possible to use multiple calls.

The object file that dtrace produces probes.o is going to be over written with each call to dtrace, so it's no surprise that the naive approach of multiple calls to dtrace each call generating the same object file does not work:

$ dtrace -G -32 -s probes.d app.o
$ dtrace -G -32 -s probes.d app1.o
$ cc -O app.o app1.o probes.o
$ sudo dtrace -s script.d -c a.out
dtrace: script 'script.d' matched 9 probes
a=1, b=2
a=1, b=2
a=1, b=2
a=2, b=3
dtrace: pid 20725 has exited

                2                3                1
                1                2                2

The next thing to try is whether changing the generated object file works:

$ dtrace -G -32 -s probes.d -o probe0.o app.o
$ dtrace -G -32 -s probes.d -o probe1.o app1.o
$ cc -O probes.o app.o app1.o probe1.o
$ sudo dtrace -s script.d -c a.out
dtrace: script 'script.d' matched 9 probes
a=1, b=2
a=1, b=2
a=1, b=2
a=2, b=3
dtrace: pid 20673 has exited

                2                3                1
                1                2                2

And if we wanted more proof, swapping the order of the object files generates the following:

$ cc -O app.o app1.o probe1.o probe0.o
$ sudo dtrace -s script.d -c a.out
dtrace: script 'script.d' matched 1 probe
a=1, b=2
a=1, b=2
a=1, b=2
a=2, b=3
dtrace: pid 20683 has exited

                1                2                1

So the conclusion is that the only way it will work is by putting all the object files onto the commandline of a single call to dtrace.

Monday Dec 17, 2007

Call stack visualisation using dtrace

Nice demonstration of using dtrace to extract callstack information.

Tuesday Nov 27, 2007

Adding dtrace probes to user code (part 2)

Adam Leventhal pointed out in the comments to my post on adding dtrace userland probes that there is an improved approach to adding userland dtrace probes. He describes this approach on his blog.

The approach solves two problems. First, that C++ name mangling makes it hard to add dtrace probes for that language. Second, that code with dtrace probes inserted in it will not compile on systems that do not have the necessary dtrace support.

So going back to the example code, I'll try to show the problem and the solution. Here's app.cc:

#include <stdio.h>
#include <sys/sdt.h>

void func(int a, int b)
{
  DTRACE_PROBE2(myapp,func_call,a,b);
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

When compiled with the C compiler the following symbols get defined:

$ cc -c app.cc
$ nm app.o
app.o:

[Index]   Value      Size    Type  Bind  Other Shndx   Name
...
[10]    |         0|       0|FUNC |GLOB |0    |UNDEF  |__dtrace_myapp___func_call
...

When compiled with the C++ compiler the following happens:

$ CC -c app.cc
$ nm app.o
app.o:

[Index]   Value      Size    Type  Bind  Other Shndx   Name
...
[7]     |         0|       0|FUNC |GLOB |0    |UNDEF  |__1cbA__dtrace_myapp___func_call6FLL_v_
...

Because the call to the dtrace probe is not declared as being extern 'C' the compiler mangles the C++ function name.

The new approach that Adam describes involves dtrace preprocessing the probe description file to generate a header file, and the including the header file in the source code. The big advantage of having the header file is that it's now possible to declare the dtrace probes to have extern 'C' linkage, and avoid the name mangling issue. The syntax for preprocessing the probe description file is:

$ dtrace -h -s probes.d

This generates the following header file:

/\*
 \* Generated by dtrace(1M).
 \*/

#ifndef _PROBES_H
  #define _PROBES_H

  #include <unistd.h>

  #ifdef  __cplusplus
    extern "C" {
  #endif

  #if _DTRACE_VERSION
    #define MYAPP_FUNC_CALL(arg0, arg1)  __dtrace_myapp___func_call(arg0, arg1)
    #define MYAPP_FUNC_CALL_ENABLED()    __dtraceenabled_myapp___func_call()
    extern void __dtrace_myapp___func_call(int, int);
    extern int __dtraceenabled_myapp___func_call(void);
  #else
    #define MYAPP_FUNC_CALL(arg0, arg1)
    #define MYAPP_FUNC_CALL_ENABLED() (0)
  #endif

  #ifdef  __cplusplus
    }
  #endif

#endif  /\* _PROBES_H \*/

The other advantage is that the header file can protect the definitions of the dtrace probes with #if _DTRACE_VERSION; which enables the same source to be compiled on systems which do not support dtrace.

The source code needs to be modified to support this syntax:

#include &kt;stdio.h>
#include "probes.h"

void func(int a, int b)
{
  MYAPP_FUNC_CALL(a,b);
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

The rest of the process is the same as before.

Tuesday Nov 20, 2007

Adding dtrace probes to user code

The process of adding dtrace probes to userland code is described in the dynamic tracing guide. However, there's no better way of learning how to do it, than trying it out on a snippet of code.

Here's a short bit of code that calls a function twice, each time with different parameters. The plan is to insert a probe that can report the passed parameters.

#include <stdio.h>

void func(int a, int b)
{
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

The first change is to add the <sys/sdt.h> header file. This file has definitions for the DTRACE_PROBE<N> macro. N represents the number of parameters that are to be reported by the probe. In this case we are going to pass two parameters (a and b) to the probe. As well as the parameters that are to be passed to the dtrace probe, the macro takes the name to be used tof the application provide (in this case the name will be myapp) and the name of the probe (in this case func_call). The modified source code looks as follows:

#include <stdio.h>
#include <sys/sdt.h>

void func(int a, int b)
{
  DTRACE_PROBE2(myapp,func_call,a,b);
  printf("a=%i, b=%i\\n",a,b);
}

void main()
{
  func(1,2);
  func(2,3);
}

The next step is to write a probe description file which dtrace will use to produce the probes. A full file would describe the stability of the probe in more detail, but a lightweight file just describes the probes defined by the provider application:

provider myapp
{
  probe func_call(int, int);
};

Having completed this, it's necessary to compile and link the application. Initially each source file needs to be compiled, and then before the application is linked, dtrace needs to be invoked to modify the object files, removing the calls to the probes, but leaving space for them to be reinserted. dtrace also needs to compile the probe description file into an object file. Finally the modified object files and the probe description file can be linked to produce the executable. As follows:

$ cc -c app.c
$ dtrace -G -32 -s probes.d app.o
$ cc probes.o app.o

The resulting code in the application looks like:

func()
        113a0:  9d e3 bf a0  save       %sp, -96, %sp
        113a4:  f0 27 a0 44  st         %i0, [%fp + 68]
        113a8:  f2 27 a0 48  st         %i1, [%fp + 72]
        113ac:  d0 07 a0 44  ld         [%fp + 68], %o0
        113b0:  01 00 00 00  nop
        113b4:  d2 07 a0 48  ld         [%fp + 72], %o1
        113b8:  11 00 00 45  sethi      %hi(0x11400), %o0
        113bc:  90 12 22 60  bset       608, %o0        ! 0x11660
        113c0:  d2 07 a0 44  ld         [%fp + 68], %o1
        113c4:  40 00 42 c7  call       printf  ! 0x21ee0
        113c8:  d4 07 a0 48  ld         [%fp + 72], %o2
        113cc:  81 c7 e0 08  ret
        113d0:  81 e8 00 00  restore

The nop at 0x113b0 is there for dtrace to dynamically patch with a call instruction that will enable the dtrace probe.

Finally, the following is an example of using the new probe:

$ more script.d
myapp$target:::func_call
{
  @[arg0,arg1]=count();
}
$ dtrace -s script.d -c a.out
dtrace: script 'script.d' matched 1 probe
a=1, b=2
a=2, b=3
dtrace: pid 22355 has exited

                1                2                1
                2                3                1

The script just aggregates the parameters used in the function call. When the application terminates the aggregation is printed out - showing the expected result of two calls to the routine each call with different parameters.

Tuesday Jun 12, 2007

Identifying misaligned loads in 32-bit code using dtrace

A previous blog entry talks about handling and detecting misaligned memory accesses. For 64-bit code this is easy to achieve using the Performance Analyzer, for 32-bit code the analysis is a bit more tricky. Fortunately it is possible to do the 32-bit analysis with dtrace

Consider the following program which has a misaligned memory access. The default mode of the compiler (since Sun Studio 9) will compile the binary to trap to fix the misalignment and continue

% more align.c
void main()
{
  volatile char a[10];
  int i;
  for (i=0; i<100000000; i++) {(\*(int\*)(&a[1]))++;}
}

The following dtrace script will instrument the misaligned data access trap handler and report all the pids that trigger this

% more tr.d
fbt::do_unaligned:entry
{
  @p[pid]=count();
}

It can be run with

% sudo dtrace -s tr.d
dtrace: script 'tr.d' matched 1 probe
\^C


    14873           260932

The script returns the pid which is having misalignment issues. This information is useful, in that it is trivial to recompile the binary with a different setting for -xmemalign and avoid the behaviour. But it would be very useful to know where the traps are occuring in the binary - perhaps most of the traps only happen in one place, and that place can be fixed in the source.

% more tr.d
fbt::do_unaligned:entry
{
  @[ustack()]=count();
}

This script produces output that identifies the locations in the binary where the traps are being generated. For the simple test code there are two locations - the load and the store.

sudo dtrace -s tr.d
dtrace: script 'tr.d' matched 1 probe
\^C
              align`main+0x10
              align`_start+0x108
           130466

              align`main+0x18
              align`_start+0x108
           130466

The disassembly for the loop is as follows

main()
        10b80:  9d e3 bf 90  save       %sp, -112, %sp
...
        10b90:  d0 07 bf f7  ld         [%fp - 9], %o0  <<<<<< misaligned
        10b94:  90 02 20 01  inc        %o0
        10b98:  d0 27 bf f7  st         %o0, [%fp - 9]  <<<<<< misaligned
        10b9c:  ba 07 60 01  inc        %i5
        10ba0:  80 a7 40 09  cmp        %i5, %o1
        10ba4:  06 bf ff fb  bl         main+0x10       ! 0x10b90
        10ba8:  01 00 00 00  nop

Friday May 25, 2007

Calling dtrace from Java

Recently been playing with calling dtrace scripts from Java. Its a nice api. The initial start page for learning about this is here. However, it may be more effective to look at the documentation that is included in the source. The source for some examples is also available.

About

Darryl Gove is a senior engineer in the Solaris Studio team, working on optimising applications and benchmarks for current and future processors. He is also the author of the books:
Multicore Application Programming
Solaris Application Programming
The Developer's Edge

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
5
6
8
9
10
12
13
14
15
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
The Developer's Edge
Solaris Application Programming
Publications
Webcasts
Presentations
OpenSPARC Book
Multicore Application Programming
Docs