Thursday Sep 23, 2010

Highly Recommended Solaris Tools book

Ever since I've been given a copy of the "Solaris Performance and Tools" book I've found it more and more useful. It has a lot of good information regarding various diagnostic tools that are native to Solaris including dtrace and mdb. If you ever need to examine a core dump or kernel panic the mdb section of this book is fantastic. Same goes for using dtrace.

Tuesday Jun 14, 2005

Playing with Solaris memory debuggers

Playing with Solaris memory debuggers

Playing with Solaris memory debuggers

The following are notes that I've made for myself as I used various Solaris memory debugging libraries. Given the following was originally for my consumption I can not vouch for the correctness of the grammar.

====================================================================
UserSpace info:

I was playing with both watchmalloc.so.1, libumem.so.1 to see what they can do.
Here's what I observe:

Using environment variables:

LD_PRELOAD=watchmalloc.so.1 MALLOC_DEBUG=WATCH,RW

watchmalloc can find bugs like (using MALLOC_DEBUG=RW):

p=malloc(128);
free(p);
foo=\*p; /\* invalid read \*/

or:

p=malloc(128);
free(p);
\*p=1; /\* invalid write \*/

or:

p=malloc(1);
memset(p, 1, 10); /\* write past buffer \*/

Note, watchmalloc will core dump when it detects an error.  And it causes
programs to run MUCH slower.

===========================================================

libumem does have some memory guards to detect invalid writes but it requires
use of the mdb command ::umem_verify on the core.  It does not catch bad reads
like watchmalloc but it does have guards with patterns like 0xdeadbeef and
0xbaddcafe.  And it does not slow down program execution like watchmalloc.

Use LD_PRELOAD=libumem.so.1
With libumem one can do memory leak detection (using
UMEM_LOGGING=transaction UMEM_DEBUG=default):

p=malloc(128);
p=malloc(128);
abort();

then do 'echo ::findleaks|mdb core' to see: 

CACHE     LEAKED   BUFCTL CALLER
000bb088       1 000cb608 main+8
----------------------------------------------------------------------
   Total       1 buffer, 320 bytes

then use:

echo '000cb608::bufctl_audit' | mdb core

to see the stack trace where the leak allocation took place.
Note, LEAKED is the number of times a leak occurred.

Note, a core dump is not necessary.  Use "mdb -o nostop -p PID" where PID
is the proc. ID of the running process and then do the findleaks stuff:

echo '::findleaks' | mdb -o nostop -p $(pgrep gssd)
echo '000cb608::bufctl_audit' | mdb -o nostop -p $(pgrep gssd)

This is good for daemons.  Or use "gcore <PID>" to get a core dump of a
running process.  This is useful to look for leaks in daemons like krb5kdc.

Also, to watch the memory size of a running daemon to see if it is growing over
time use:

prstat -p <PID of daemon> 300 > /tmp/prstat.out
---------------------------------------------------------------------------
Both watchmalloc and libumem will core dump on double free()'s.

Based on the above it seems like it would be good to use watchmalloc for
some memory corruption testing and use libumem for memory leak
detection.

Note use:

print ::umem_status|mdb core

to see umem status for user space core when debugging with libumem.

Use: 

print ::umalog | mdb core

to see umem transaction log and stack traces.

It's also good do do:

print ::stack | mdb core

and look at the stack trace (note, the values in each function listed
are the input registers %i0-5).
Look at umalog to see if it's possible to determine if memory was
free'ed earlier.  Use:

print "<address>::dis" | mdb core

to see the assembly around the stack function address to see where the
call was made (look for other call's).

===========================================================
Kernel memory debugging info:

In /etc/system do:

    \* kmem lite flag, must use independently of other kmem flags
    \* set kmem_flags = 0x100
    \* kmem flags: audit, test, redzone
    set kmem_flags = 0x7

and reboot system.

Use:

echo "::dcmds" | mdb unix.0 vmcore.0

to see debugging commands (look for kmem stuff).

Note, the kernel kmem outputs debug messages to syslog.

Technorati Tag:
Technorati Tag:
Technorati Tag:
About

user12615206

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today