Debugging with libumem and MDB

In celebration of OpenSolaris's
birthday, I thought I would do some more blogging about libumem,
one of my favorite parts of it. In particular, I'll cover some of its debugging features,
which borrow heavily from the kmem dcmds and walkers written by href="/bmc">Bryan, Dan, and others.

Much of the debugging power of libumem comes from its mdb(1M)
debugger module.
Anyone familiar with the kmem dcmds used for kernel debugging will see a
lot of similarities (modulo some 'u's where 'k's used to be). You can
see a list of everything it provides by doing:

> ::dmods -l libumem.so.1
dcmd allocdby - given a thread, print its allocated buffers
dcmd bufctl - print or filter a bufctl
dcmd bufctl_audit - print a bufctl_audit
dcmd findleaks - search for potential memory leaks
dcmd freedby - given a thread, print its freed buffers
dcmd ugrep - search user address space for a pointer
dcmd umalog - display umem transaction log and stack traces
dcmd umastat - umem allocator stats
dcmd umausers - display current medium and large users of the
umem allocator
dcmd umem_cache - print a umem cache
dcmd umem_debug - toggle umem dcmd/walk debugging
dcmd umem_log - dump umem transaction log
dcmd umem_malloc_dist - report distribution of outstanding malloc()s
dcmd umem_malloc_info - report information about malloc()s by cache
dcmd umem_status - Print umem status and message buffer
dcmd umem_verify - check integrity of umem-managed memory
dcmd vmem - print a vmem_t
dcmd vmem_seg - print or filter a vmem_seg
dcmd whatis - given an address, return information
walk allocdby - given a thread, walk its allocated bufctls
walk bufctl - walk a umem cache's bufctls
walk bufctl_history - walk the available history of a bufctl
walk freectl - walk a umem cache's free bufctls
walk freedby - given a thread, walk its freed bufctls
walk freemem - walk a umem cache's free memory
walk leak - given a leak ctl, walk other leaks w/ that
walk leakbuf - given a leak ctl, walk addr of leaks w/ that
walk umem - walk a umem cache
walk umem_alloc_112 - walk the umem_alloc_112 cache
... more umem_alloc_\* caches ...
walk umem_bufctl_audit_cache - walk the umem_bufctl_audit_cache cache
walk umem_bufctl_cache - walk the umem_bufctl_cache cache
walk umem_cache - walk list of umem caches
walk umem_cpu - walk the umem CPU structures
walk umem_cpu_cache - given a umem cache, walk its per-CPU caches
walk umem_hash - given a umem cache, walk its allocated hash table
walk umem_log - walk the umem transaction log
walk umem_magazine_1 - walk the umem_magazine_1 cache
... more umem_magazine_\* caches ...
walk umem_slab - given a umem cache, walk its slabs
walk umem_slab_cache - walk the umem_slab_cache cache
walk umem_slab_partial - given a umem cache, walk its partially allocated
slabs (min 1)
walk vmem - walk vmem structures in pre-fix, depth-first order
walk vmem_alloc - given a vmem_t, walk its allocated vmem_segs
walk vmem_free - given a vmem_t, walk its free vmem_segs
walk vmem_postfix - walk vmem structures in post-fix, depth-first
walk vmem_seg - given a vmem_t, walk all of its vmem_segs
walk vmem_span - given a vmem_t, walk its spanning vmem_segs

There's a lot of meat here, but I'll start by focusing on the most important
dcmds and walkers.

Important dcmds

addr::whatisReports information about a given buffer. At the moment, the
support for this is kind of anemic; it will only give information

about buffers under libumem's control. There's an RFE to do



enhance ::whatis for libumem

But even as it is, it is still quite handy for debugging. In
Solaris 10 and later, ::whatis will automatically
provide the bufctl or vmem_seg address for buffers with
debugging information attached to them:

without UMEM_DEBUG set
> f3da8::whatis
f3da8 is f3da8+0, allocated from umem_alloc_32
with UMEM_DEBUG=default
> f3da8::whatis
f3da8 is f3da8+0, bufctl fd150 allocated from umem_alloc_32
> fd150::bufctl -v
fd150 f3da8 d8c3823401f60 1
e2788 82920 0

This allows for a quick answer to the question "what is this buffer,

and who was the last one to allocate/free it". See the descriptions

of ::bufctl and ::vmem_seg, below, for more

information on their use. (In Solaris 9, you need to use '-b' to

get the bufctl address)


Searches the entire address space for a particular pointer value.

The value must be properly aligned. There are options to loosen

the search; -d dist searches for

[addr, addr + dist) instead of an exact match,

-m mask only compares the bits selected in mask,



Display's information from a 'umem_bufctl_audit_t' pointer, which

includes the buffer address, timestamp, thread, and caller. With

the '-v' switch ("verbose" mode), it also includes the cache,

transaction log pointer, contents log pointer, and stack trace.

This dcmd can also be used in a pipeline to filter the bufctls it

prints, by address (-a addr), function or function+offset

in the stack trace (-c caller),

timestamp (-e earliest / -l latest), or thread

(-t thread)

In Solaris Nevada, you can also get the full history for a bufctl

by using the -h flag.

On Solaris 9, the equivalent of ::bufctl -v is



Display's information from a 'vmem_seg_t' pointer, which

includes the type, start address and end addresses of the segment.

For ALLoCated segments, it also includes the top stack from

the stacktrace. With the '-v' switch, ("verbose" mode), it

also includes (for ALLoCated segments only) the thread,

timestamp, and stack trace recorded at allocation time.


Does a conservative garbage-collection of the entire process in

order to find memory leaks. The memory leaks are then grouped

by stack trace and either (umem cache) or (allocation size). To

dump all of the leak stack traces, you can use the -d flag.


Prints a report of all of the umem-managed memory in the system,

grouped by umem cache and vmem arena. This can be used to see

which allocation sizes are chewing up the most memory.


Verifies the consistancy of the umem heap. If debugging is enabled,

this will find instances of modified free buffers and writes past

the end of the buffer. To get a detailed report of corrupted buffer

addresses, take the cache pointer from a line with

"n corrupted buffers", and do addr::umem_verify.


Lists all of the umem caches in the system, in tabular form. This

is often the easiest way to get a cache's address.

That's all for now, but I'll follow up with more descriptions and
examples later.

Tags: [

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.