What better way to celebrate the launch of OpenSolaris than to talk about DTrace ! One of the wonderful features that has made debugging a lot more easier ! :)

Memory leaks have always been difficult problems to deal with. There are various solutions available ranging from recompiling the source with specific probes to preloading with special libraries.

During a recent encounter with memoryleaks I thought of trying DTrace as a tools to narrow down on the problem. Much had and has been talked about DTrace and I thought I should give it a shot. And it did not disappoint me :-)

Here is the DTrace script (Dscript) I came up with:

-- snip --
# cat ./memleak.d

#!/usr/sbin/dtrace -s

pid$1:libc.so.1:malloc:entry
{
        self->trace = 1;
        self->size = arg0;
}
pid$1:libc.so.1:malloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Size=%d", arg1, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:realloc:entry
{
        self->trace = 1;
        self->size = arg1;
        self->oldptr = arg0;
}

pid$1:libc.so.1:realloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Oldptr=0x%p Size=%d", arg1, self->oldptr, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:calloc:entry
/self->trace == 1/
{
        self->trace = 1;
        self->size = arg1;
}

pid$1:libc.so.1:calloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Size=%d", arg1, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:free:entry
{
        printf("Ptr=0x%p ", arg0);
}
-- snip --

• On entry to malloc()/calloc()/realloc(), it captures the size of the allocation.
• On return from malloc()/calloc()/realloc(), it prints :
  • the pointer that is returned
  • the size of the buffer that was allocated (captured during entry)
  • the stack trace (which tells us where the memory was allocated).
  • incase of realloc(), we print the old pointer and the new pointer.
• On entry to free() we just print the pointer which is being freed.

The arguments that this Dscript takes is the pid of the process for which you want to trace the memory leaks.

 
  eg. ./memleak.d <pid>  > /var/tmp/memleak.data

So, memleak.data has the output from the Dscript.

Once we know that the process has leaked memory (can be confirmed by monitoring the SIZE of the process in prstat) we can stop the Dscript.

Now, we need to filter the output obtained (ie. memleak.data). And to do this here is a Perl script (Thanks to my collegue Venky who wrote this !) :

-- snip --
#!/usr/bin/perl

# findleaks.pl

use Data::Dumper;

my %hash = ();

while (<>) {
        if ((/malloc:return Ptr=([\^ ]\*) Size=(.\*)/) || 
            (/calloc:return Ptr=([\^ ]\*) Size=(.\*)/)) {
                $hash{$1} = { size => $2 };
		while (<>) {
			last if /\^$/;
			$hash{$1}->{stack} .= $_;
		}
        }
        elsif (/free:entry Ptr=([\^ ]\*)/) {
                if (exists $hash{$1} and $hash{$1}) {
                        $hash{$1} = '';
                }
        }
	elsif (/realloc:entry Ptr=([\^ ]\*) Oldptr=([\^ ]\*) Size=(.\*)/) {
		if ($1 eq $2) {
			if (exists $hash{$1} and $hash{$1}) {
				$hash{$1} = { size => $3 };
				$hash{$1}->{stack} = '';
				while (<>) {
		                        last if /\^$/;
               			         $hash{$1}->{stack} .= $_;
                		}
			} 
		} else {
				$hash{$1} = '';
				$hash{$2}= { size => $3 };
				$hash{$2}->{stack} = '';
				while (<>) {
		                        last if /\^$/;
               			         $hash{$2}->{stack} .= $_;
                		}
		}
	}

}

foreach my $key (keys %hash) {
	next if not $hash{$key}->{size};
	print "Ptr=$key Size=", $hash{$key}->{size}, "\\n";
	print $hash{$key}->{stack}, "\\n---------\\n";
}
-- snip --

./findleaks.pl ./memleak.data.

The output of findleaks.pl contains all the buffers that were allocated but never freed. It also prints the stack trace (of where the buffer was allocated) and the size of the buffer.

So, these are the potential leaks (I say potential as these could be in use). However, if we see a particular stack repeating frequently, then I would suspect it. Well, now that we know where the potential leaks are we need to do some code reading find the actual leaks.

libumem is another good approach. This helps in finding out many other memory management related issues. The details for this are available on Adam Leventhal's weblog at : http://blogs.sun.com/roller/page/ahl/?anchor=solaris_10_top_11_20

However, if you don't like to preload the libraries and want to just identify memory leaks then, the DTrace scripts would be ideal.

So, important advantages of using DTrace comapred to other methods that I know of :

• No recompilation of the binaries needed.
• No need to interpose/preload a library. Which means you don't need to restart the process and also avoid any possible issues with preloading the library.
• The overhead of DTrace is much lesser.
• The tracing can turned ON or OFF as needed. Unlike in other cases where you would have to stop the process and restart it (in case of preloading the libraries).

And all this took a couple of hours to write and we could narrow down on the problem quickly.

Although DTrace may not be the best tool to narrow down on the memory leaks, it is definetly one of the easiest one !

Technorati Tag: OpenSolaris
Technorati Tag: Solaris
Technorati Tag: DTrace