Baby's first DTrace

At the Solaris 10 launch on Monday I was talking to a sysadmin about DTrace. He was clearly very excited about it -- finally he could end a fight between the database guys and the appserver guys about whose stuff was to blame -- but he had one reservation: Where do I start? Since DTrace lets you look at almost anything on the system, it can be hard knowing the first thing to look at, here's what I told him:

start with the tools you know

You've probably used truss(1) or mpstat(1M) or prstat(1) or iostat(1M) or whatever. They give you a static view of what's happening on the system -- static in that you can't get any more, you can't get any other degree of detail, and you can't dive deeper. So start from those points, and go deeper. Each statistic in those observability tools has at least one associated probe in DTrace. If you're looking at mpstat(1M) output, maybe cross-calls (xcal) are high, or spins on mutexes (smtx) are high. You don't have to guess anymore; you can actually drill down and figure out what application or what user or what zone they correspond to by enabling their corresponding DTrace probes (sysinfo:::xcalls and lockstat:::\*-spin respectively) and trace the data you want.

figure out what functions are being called

When you're trying to optimize an application, it helps to know where the app is spending its time. A simple DTrace invocation like this:

# dtrace -n 'pid$target:::entry{ @[probefunc] = count() }' -p <process-id>

can give you a coarse idea of where you're spending time. When you do this, a lot of it will make sense, but some of it will probably be a surprise: "Why am I calling malloc(3C) a bazillion times?" So find those aberrant cases and figure out what's going on: "OK, how much are we allocating each time?" (dtrace -n 'pid$target::malloc:entry{ @ = quantize(arg0) }' -p <process-id>).

look for lock contention

In multi-threaded apps, lock contention can be huge performance killer. Run the new plockstat(1) command to see if your app suffers from lock contention. If it does, you'll see long spin and contention times. These are pretty easy problems to solve, but if you can't track down the source of the problem, plockstat -- of course -- lets you dig deeper by using the plockstat provider.

Those are a few places I've started from in the past, but, of course, every application is different. DTrace isn't meant to supplant your knowledge about your app and the system at large, rather it should complement it and let you do more with what you already know.

Comments:

Thanks for putting these hints up. I've been reading on DTrace (the big manual) but hadn't gotten to the point where I tought I could put it into practice (beside some basic sysadmin stuff) for development per say. I do count on that in the following months to increase performance of a web based application. Turns out that one needs to start using dtrace to realise its potential... I've been running povray under Solaris x86. Noticed it was slower than expected. For background info, see: http://groups.yahoo.com/group/solarisx86/message/15480 Well, today, I had a few minutes, stumbled on your blog and decided to try and see if I'd get anywhere. I did: http://groups.yahoo.com/group/solarisx86/message/15708 I dont have the Sun compiler so I'll wait on Dennis (from Blastwave) to recompile a new version with this enabled, and I fully expect a performance gain of 30%+. I'll keep you posted. Francois

Posted by Francois Dion on November 19, 2004 at 05:47 AM PST #

Francois,
I'm delighted that you found this so immediately helpful. I'll try to put together more such first steps (and hope the timing of my blog post coincides with some performance problem cropping up on a open source discussion).

Posted by Adam Leventhal on November 20, 2004 at 02:22 AM PST #

How about using DTrace to learn about the internals of Solaris? Kinda like what Richard Stevens did to learn networking on unix (ie. discovery by experimentation). This would actually be helpful to learn about DTrace \*and\* Solaris. Can you engage your tech writers on this?

Posted by Denis T. on November 25, 2004 at 05:41 AM PST #

Hey Denis,
You might want to check out Solaris Internals. It currently doesn't include anything about DTrace (or Solaris 10), but Jim Mauro is working on rolling DTrace examples into various chapters.

Posted by Adam Leventhal on November 25, 2004 at 01:51 PM PST #

Post a Comment:
Comments are closed for this entry.
About

Adam Leventhal, Fishworks engineer

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today