Debugging with libumem and MDB
By jwadams on Jun 14, 2006
In celebration of OpenSolaris's birthday, I thought I would do some more blogging about libumem, one of my favorite parts of it. In particular, I'll cover some of its debugging features, which borrow heavily from the kmem dcmds and walkers written by Bryan, Dan, and others.
Much of the debugging power of libumem comes from its mdb(1M) debugger module. Anyone familiar with the kmem dcmds used for kernel debugging will see a lot of similarities (modulo some 'u's where 'k's used to be). You can see a list of everything it provides by doing:
> ::dmods -l libumem.so.1 libumem.so.1 dcmd allocdby - given a thread, print its allocated buffers dcmd bufctl - print or filter a bufctl dcmd bufctl_audit - print a bufctl_audit dcmd findleaks - search for potential memory leaks dcmd freedby - given a thread, print its freed buffers dcmd ugrep - search user address space for a pointer dcmd umalog - display umem transaction log and stack traces dcmd umastat - umem allocator stats dcmd umausers - display current medium and large users of the umem allocator dcmd umem_cache - print a umem cache dcmd umem_debug - toggle umem dcmd/walk debugging dcmd umem_log - dump umem transaction log dcmd umem_malloc_dist - report distribution of outstanding malloc()s dcmd umem_malloc_info - report information about malloc()s by cache dcmd umem_status - Print umem status and message buffer dcmd umem_verify - check integrity of umem-managed memory dcmd vmem - print a vmem_t dcmd vmem_seg - print or filter a vmem_seg dcmd whatis - given an address, return information walk allocdby - given a thread, walk its allocated bufctls walk bufctl - walk a umem cache's bufctls walk bufctl_history - walk the available history of a bufctl walk freectl - walk a umem cache's free bufctls walk freedby - given a thread, walk its freed bufctls walk freemem - walk a umem cache's free memory walk leak - given a leak ctl, walk other leaks w/ that stacktrace walk leakbuf - given a leak ctl, walk addr of leaks w/ that stacktrace walk umem - walk a umem cache walk umem_alloc_112 - walk the umem_alloc_112 cache ... more umem_alloc_\* caches ... walk umem_bufctl_audit_cache - walk the umem_bufctl_audit_cache cache walk umem_bufctl_cache - walk the umem_bufctl_cache cache walk umem_cache - walk list of umem caches walk umem_cpu - walk the umem CPU structures walk umem_cpu_cache - given a umem cache, walk its per-CPU caches walk umem_hash - given a umem cache, walk its allocated hash table walk umem_log - walk the umem transaction log walk umem_magazine_1 - walk the umem_magazine_1 cache ... more umem_magazine_\* caches ... walk umem_slab - given a umem cache, walk its slabs walk umem_slab_cache - walk the umem_slab_cache cache walk umem_slab_partial - given a umem cache, walk its partially allocated slabs (min 1) walk vmem - walk vmem structures in pre-fix, depth-first order walk vmem_alloc - given a vmem_t, walk its allocated vmem_segs walk vmem_free - given a vmem_t, walk its free vmem_segs walk vmem_postfix - walk vmem structures in post-fix, depth-first order walk vmem_seg - given a vmem_t, walk all of its vmem_segs walk vmem_span - given a vmem_t, walk its spanning vmem_segsThere's a lot of meat here, but I'll start by focusing on the most important dcmds and walkers.
|addr::whatis||Reports information about a given buffer. At the moment, the
support for this is kind of anemic; it will only give information
about buffers under libumem's control. There's an RFE to do
4706502 enhance ::whatis for libumem
But even as it is, it is still quite handy for debugging. In Solaris 10 and later, ::whatis will automatically provide the bufctl or vmem_seg address for buffers with debugging information attached to them:
without UMEM_DEBUG set > f3da8::whatis f3da8 is f3da8+0, allocated from umem_alloc_32 with UMEM_DEBUG=default > f3da8::whatis f3da8 is f3da8+0, bufctl fd150 allocated from umem_alloc_32 > fd150::bufctl -v ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS fd150 f3da8 d8c3823401f60 1 e2788 82920 0 libumem.so.1`umem_cache_alloc+0x218 libumem.so.1`umem_alloc+0x58 libumem.so.1`malloc+0x28 nam_putval+0x3dc nam_fputval+0x1c env_namset+0x228 env_init+0xb4 main+0xa8 _start+0x108This allows for a quick answer to the question "what is this buffer, and who was the last one to allocate/free it". See the descriptions of ::bufctl and ::vmem_seg, below, for more information on their use. (In Solaris 9, you need to use '-b' to get the bufctl address)
Searches the entire address space for a particular pointer value.
The value must be properly aligned. There are options to loosen
the search; -d dist searches for
[addr, addr + dist) instead of an exact match,
-m mask only compares the bits selected in mask,
Display's information from a 'umem_bufctl_audit_t' pointer, which
includes the buffer address, timestamp, thread, and caller. With
the '-v' switch ("verbose" mode), it also includes the cache,
transaction log pointer, contents log pointer, and stack trace.
This dcmd can also be used in a pipeline to filter the bufctls it prints, by address (-a addr), function or function+offset in the stack trace (-c caller), timestamp (-e earliest / -l latest), or thread (-t thread)
In Solaris Nevada, you can also get the full history for a bufctl by using the -h flag.
On Solaris 9, the equivalent of ::bufctl -v is addr::bufctl_audit.
Display's information from a 'vmem_seg_t' pointer, which
includes the type, start address and end addresses of the segment.
For ALLoCated segments, it also includes the top stack from
the stacktrace. With the '-v' switch, ("verbose" mode), it
also includes (for ALLoCated segments only) the thread,
timestamp, and stack trace recorded at allocation time.
Does a conservative garbage-collection of the entire process in
order to find memory leaks. The memory leaks are then grouped
by stack trace and either (umem cache) or (allocation size). To
dump all of the leak stack traces, you can use the -d flag.
Prints a report of all of the umem-managed memory in the system,
grouped by umem cache and vmem arena. This can be used to see
which allocation sizes are chewing up the most memory.
Verifies the consistancy of the umem heap. If debugging is enabled,
this will find instances of modified free buffers and writes past
the end of the buffer. To get a detailed report of corrupted buffer
addresses, take the cache pointer from a line with
"n corrupted buffers", and do addr::umem_verify.
Lists all of the umem caches in the system, in tabular form. This
is often the easiest way to get a cache's address.
That's all for now, but I'll follow up with more descriptions and examples later.