Wednesday Sep 30, 2009

DTrace and unexploded ordnance

The Sun Corp User Group meeting yesterday went well. We had a fair turn out, some good questions and I have been asked to do a couple of company specific sessions as follow on.

Sometimes you don't always give the best answer when answering cold. Once you have had a little time to mull it over you think of a possible better answer or an equally valid alternative approach(or you were just wrong the 1st time, it happens sometimes).

To the gentleman with the umount problem which fails every few mounts prior to a backup, I would also make sure the assumption about why umount is failing with a busy file. If you are getting an error message in the log which points that way, then fine, if not, it would be a good idea to get the error code of the umount2 system call when it does fail. We could use DTrace, but we could also wrap the umount command inside truss in the script

truss -t umount2 -v umount2 umount /wibble
make sure it is logged. Still think the live dump is the right way to chase this if you can script it right and don't want to sit by the machine for months on end at 2am when the backup kicks off. This is still the wrong answer as ZFS snapshots would allow you to avoid the umount altogether.

To the people concerned about Oracle performance on ZFS, the offer of 30 minutes or so of SharedShell ( still stands. One question I should of also asked is if the ZFS intent log is split out and on separate fast storage.

On the way home I went for a run near New Rador. Very nice area indeed and great running. I did come across a flag pole with a red flag and some signage which discussed explosives and this being a restricted area. From later discussions with a local this was a test range and people with metal detectors turn up, hoist the flag and dispose of the various bits of scattered ordnance and you really know when they are there.

I am staying to the paths next time. The thought does not appeal of either arriving on a world war 2 morter shell at the pearly gates or being an minor character in 1 episode of Old Harry's Game who spent eternity debugging Windows and Linux performance issues with the promise of DTrace port always a few days away.

View New Radnor in a larger map

In spite of the possible objective danger, some great running but I am keeping to the paths until the clearance work is complete in 2011.

Friday Sep 04, 2009

DTrace for breakfast

The UK Corporate Solaris Users Group on Tuesday 29th September has a breakfast meeting where we are going to sprinkle a little DTrace on your cereal. DTrace was a requested topic at the last meeting, so in keeping with my preferred style of delivery, it will be demo only. Diagnosis and performance analysis is a full contact sport, so best to show it as it really is.

More details can be found are here

Saturday Sep 06, 2008

VirtualBox 2.0 and DTrace now play nice together

The very simple DTrace one liner

dtrace -n fbt:::entry'{@[probefunc] = count()}'

caused the VirtualBox versions prior to 2.0 to reset. I am pleased to report that VirtualBox 2.0 and DTrace function boundary probes now work together very well.

This had been an inhibitor to using VirtualBox on my Mac laptop for DTrace demo's. Should I ever find no better use for my time, I am now able to use DTrace in OSX to trace the calls made by VirtualBox while using DTrace on OpenSolaris to understand the behaviour of an application.

Next race is the Nant Peris Horseshoe which starts and finishes in Llanberis.

Tuesday Sep 11, 2007

sdxx to cxtxdx conversion for IO latency by colour DTrace script

A couple of people commented that they would like a format of c0t0d0 rather than sd0 for the script in I/O latency by colour script I posted a few weeks ago.

So, change the line

       @[args[1]->dev_pathname] = lquantize(this->elapsed / MS, 0, 200, 50);
       @[args[1]->dev_statname] = lquantize(this->elapsed / MS, 0, 200, 50);
The DTrace IO provider does not appear to provide the format cxtxdx(if you know different, please add a comment), so a bit of klunky post processing is needed.
#!/usr/bin/perl -w

use strict;
my %regex= ();

system("iostat -E | awk '/Soft/ { print \\$1}' > /tmp/a");
system("iostat -En | awk '/Soft/ { print \\$1 }' > /tmp/b");
system("paste /tmp/a /tmp/b > /tmp/c");

open(F, "/tmp/c") || die ("no /tmp/c");
while() {
    my ($sd,$ctd) = split;
    $regex{$sd} = $ctd;

while (<>) {
    for my $sd (keys %regex) {
Then run as
 pfexec dtrace -s ./io_latency_by_colour.d | ./
and enjoy the colours and the easier to read disk format. I shall have to talk to Brother Jon to determine if an addition to the IO provider makes sense, though given the amount of work iostat goes through to get this format, might explain why its not already part if the IO provider.



« April 2014