Debugging with libumem and MDB

In celebration of OpenSolaris's birthday, I thought I would do some more blogging about libumem, one of my favorite parts of it. In particular, I'll cover some of its debugging features, which borrow heavily from the kmem dcmds and walkers written by Bryan, Dan, and others.

Much of the debugging power of libumem comes from its mdb(1M) debugger module. Anyone familiar with the kmem dcmds used for kernel debugging will see a lot of similarities (modulo some 'u's where 'k's used to be). You can see a list of everything it provides by doing:

> ::dmods -l
  dcmd allocdby             - given a thread, print its allocated buffers
  dcmd bufctl               - print or filter a bufctl
  dcmd bufctl_audit         - print a bufctl_audit
  dcmd findleaks            - search for potential memory leaks
  dcmd freedby              - given a thread, print its freed buffers
  dcmd ugrep                - search user address space for a pointer
  dcmd umalog               - display umem transaction log and stack traces
  dcmd umastat              - umem allocator stats
  dcmd umausers             - display current medium and large users of the
                              umem allocator
  dcmd umem_cache           - print a umem cache
  dcmd umem_debug           - toggle umem dcmd/walk debugging
  dcmd umem_log             - dump umem transaction log
  dcmd umem_malloc_dist     - report distribution of outstanding malloc()s
  dcmd umem_malloc_info     - report information about malloc()s by cache
  dcmd umem_status          - Print umem status and message buffer
  dcmd umem_verify          - check integrity of umem-managed memory
  dcmd vmem                 - print a vmem_t
  dcmd vmem_seg             - print or filter a vmem_seg
  dcmd whatis               - given an address, return information
  walk allocdby             - given a thread, walk its allocated bufctls
  walk bufctl               - walk a umem cache's bufctls
  walk bufctl_history       - walk the available history of a bufctl
  walk freectl              - walk a umem cache's free bufctls
  walk freedby              - given a thread, walk its freed bufctls
  walk freemem              - walk a umem cache's free memory
  walk leak                 - given a leak ctl, walk other leaks w/ that
  walk leakbuf              - given a leak ctl, walk addr of leaks w/ that
  walk umem                 - walk a umem cache
  walk umem_alloc_112       - walk the umem_alloc_112 cache
... more umem_alloc_\* caches ...
  walk umem_bufctl_audit_cache - walk the umem_bufctl_audit_cache cache
  walk umem_bufctl_cache    - walk the umem_bufctl_cache cache
  walk umem_cache           - walk list of umem caches
  walk umem_cpu             - walk the umem CPU structures
  walk umem_cpu_cache       - given a umem cache, walk its per-CPU caches
  walk umem_hash            - given a umem cache, walk its allocated hash table
  walk umem_log             - walk the umem transaction log
  walk umem_magazine_1      - walk the umem_magazine_1 cache
... more umem_magazine_\* caches ...
  walk umem_slab            - given a umem cache, walk its slabs
  walk umem_slab_cache      - walk the umem_slab_cache cache
  walk umem_slab_partial    - given a umem cache, walk its partially allocated
                              slabs (min 1)
  walk vmem                 - walk vmem structures in pre-fix, depth-first order
  walk vmem_alloc           - given a vmem_t, walk its allocated vmem_segs
  walk vmem_free            - given a vmem_t, walk its free vmem_segs
  walk vmem_postfix         - walk vmem structures in post-fix, depth-first
  walk vmem_seg             - given a vmem_t, walk all of its vmem_segs
  walk vmem_span            - given a vmem_t, walk its spanning vmem_segs
There's a lot of meat here, but I'll start by focusing on the most important dcmds and walkers.

Important dcmds

addr::whatis Reports information about a given buffer. At the moment, the support for this is kind of anemic; it will only give information about buffers under libumem's control. There's an RFE to do better:

4706502 enhance ::whatis for libumem

But even as it is, it is still quite handy for debugging. In Solaris 10 and later, ::whatis will automatically provide the bufctl or vmem_seg address for buffers with debugging information attached to them:
without UMEM_DEBUG set
> f3da8::whatis      
f3da8 is f3da8+0, allocated from umem_alloc_32

with UMEM_DEBUG=default
> f3da8::whatis      
f3da8 is f3da8+0, bufctl fd150 allocated from umem_alloc_32
> fd150::bufctl -v
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
           fd150            f3da8    d8c3823401f60                1
                            e2788            82920                0
This allows for a quick answer to the question "what is this buffer, and who was the last one to allocate/free it". See the descriptions of ::bufctl and ::vmem_seg, below, for more information on their use. (In Solaris 9, you need to use '-b' to get the bufctl address)

addr::ugrep Searches the entire address space for a particular pointer value. The value must be properly aligned. There are options to loosen the search; -d dist searches for [addr, addr + dist) instead of an exact match, -m mask only compares the bits selected in mask, etc.

addr::bufctl Display's information from a 'umem_bufctl_audit_t' pointer, which includes the buffer address, timestamp, thread, and caller. With the '-v' switch ("verbose" mode), it also includes the cache, transaction log pointer, contents log pointer, and stack trace.

This dcmd can also be used in a pipeline to filter the bufctls it prints, by address (-a addr), function or function+offset in the stack trace (-c caller), timestamp (-e earliest / -l latest), or thread (-t thread)

In Solaris Nevada, you can also get the full history for a bufctl by using the -h flag.

On Solaris 9, the equivalent of ::bufctl -v is addr::bufctl_audit.

addr::vmem_seg Display's information from a 'vmem_seg_t' pointer, which includes the type, start address and end addresses of the segment. For ALLoCated segments, it also includes the top stack from the stacktrace. With the '-v' switch, ("verbose" mode), it also includes (for ALLoCated segments only) the thread, timestamp, and stack trace recorded at allocation time.

::findleaks Does a conservative garbage-collection of the entire process in order to find memory leaks. The memory leaks are then grouped by stack trace and either (umem cache) or (allocation size). To dump all of the leak stack traces, you can use the -d flag.

::umastat Prints a report of all of the umem-managed memory in the system, grouped by umem cache and vmem arena. This can be used to see which allocation sizes are chewing up the most memory.

[addr]::umem_verify Verifies the consistancy of the umem heap. If debugging is enabled, this will find instances of modified free buffers and writes past the end of the buffer. To get a detailed report of corrupted buffer addresses, take the cache pointer from a line with "n corrupted buffers", and do addr::umem_verify.

::umem_cache Lists all of the umem caches in the system, in tabular form. This is often the easiest way to get a cache's address.

That's all for now, but I'll follow up with more descriptions and examples later.

Tags: [ , , , ]


Post a Comment:
Comments are closed for this entry.



Top Tags
« November 2015