Monday Mar 14, 2011

::stacks

Over a decade ago in 2000, David Powell used his intimate knowledge of sed(1) to write a shell script called munges, which entered the muscle memory of most of the Solaris Kernel Development staff. Short for munge stacks, it takes the output of ::walk thread | ::findstack, and groups the stacks by stack trace:
> ::walk thread | ::findstack !/home/dep/bin/munges
743     ##################################  tp: 2a100007c80
        taskq_thread_wait+0x38()
        taskq_thread+0x350()
        thread_start+4()

182     ##################################  tp: 30003c60020
        syscall_trap32+0xcc()

65      ##################################  tp: 30003c617c0
        poll_common+0x448()
        pollsys+0xf0()
        syscall_trap32+0xcc()
...
1       ##################################  tp: 2a100077c80 (TS_FREE)
        clock+0x508()
        cyclic_softint+0xbc()
        cbe_level10+8()
        intr_thread+0x168()
        callout_list_expire+0x5c()
        callout_expire+0x14()
        callout_realtime+0x14()

1       ##################################  tp: 2a10003fc80
        kmdbmod`kctl_wr_thread+0x80()
        thread_start+4()

1       ##################################  tp: 2a100017c80
        thread_reaper+0xa8()
        thread_start+4()

1       ##################################  tp: 180e000
        _start+0x108()
While very handy (even after a decade) it has a few drawbacks:
  • Since it is a shell script, it cannot be used directly when debugging with kmdb(1). Copying and pasting thousands of thread stacks from serial-console output was never convenient.
  • You have to use advanced egrep(1) (or less(1)) technology to search for particular threads of interest.
  • Since ::findstack only displays limited thread state information (i.e. whether the thread is free), munges can't uniquify based upon the thread state.
After sitting on my backburner for a long time, I finally implemented:
6799290 need to uniquify stacks in kmdb

Running ::stacks by itself gives output very similar to the above pipeline:
> ::stacks
THREAD           STATE    SOBJ                COUNT
2a10000fc80      SLEEP    CV                    743
                 taskq_thread_wait+0x38
                 taskq_thread+0x350
                 thread_start+4

30003dc43c0      SLEEP    SHUTTLE               182
                 syscall_trap32+0xcc
...
2a10055dc80      ONPROC   <NONE>                  1
                 idle+0x120
                 thread_start+4

3004a4d0740      ONPROC   <NONE>                  1
                 trap+0x1b30
                 user_rtt+0x20

180e000          STOPPED  <NONE>                  1
                 _start+0x108

After my initial putback (in 2009), Greg Price added 6802742 (adding module filtering), and Bryan Cantrill added 6935550 (which added ::stacks support to userland debugging).

Here's the help message for ::stacks, which has a lot of details on its use:

> ::help stacks

NAME
  stacks - print unique kernel thread stacks

SYNOPSIS
  [ addr ] ::stacks [-afiv] [-c func] [-C func] [-m module] [-M module] 
      [-s sobj | -S sobj] [-t tstate | -T tstate]

DESCRIPTION
  
  ::stacks processes all of the thread stacks on the system, grouping
  together threads which have the same:
  
    \* Thread state,
    \* Sync object type, and
    \* PCs in their stack trace.
  
  The default output (no address or options) is just a dump of the thread
  groups in the system.  For a view of active threads, use "::stacks -i",
  which filters out FREE threads (interrupt threads which are currently
  inactive) and threads sleeping on a CV. (Note that those threads may still
  be noteworthy; this is just for a first glance.)  More general filtering
  options are described below, in the "FILTERS" section.
  
  ::stacks can be used in a pipeline.  The input to ::stacks is one or more
  thread pointers.  For example, to get a summary of threads in a process,
  you can do:
  
    procp::walk thread | ::stacks
  
  When output into a pipe, ::stacks prints all of the threads input,
  filtered by the given filtering options.  This means that multiple
  ::stacks invocations can be piped together to achieve more complicated
  filters.  For example, to get threads which have both 'fop_read' and
  'cv_wait_sig_swap' in their stack trace, you could do:
  
    ::stacks -c fop_read | ::stacks -c cv_wait_sig_swap_core
  
  To get the full list of threads in each group, use the '-a' flag:
  
    ::stacks -a
  
OPTIONS
  
    -a    Print all of the grouped threads, instead of just a count.
    -f    Force a re-run of the thread stack gathering.
    -v    Be verbose about thread stack gathering.
  
FILTERS
  
    -i    Show active threads; equivalent to '-S CV -T FREE'.
    -c func[+offset]
          Only print threads whose stacks contain func/func+offset.
    -C func[+offset]
          Only print threads whose stacks do not contain func/func+offset.
    -m module
          Only print threads whose stacks contain functions from module.
    -M module
          Only print threads whose stacks do not contain functions from
          module.
    -s {type | ALL}
          Only print threads which are on a 'type' synchronization object
          (SOBJ).
    -S {type | ALL}
          Only print threads which are not on a 'type' SOBJ.
    -t tstate
          Only print threads which are in thread state 'tstate'.
    -T tstate
          Only print threads which are not in thread state 'tstate'.
  
     SOBJ types: mutex rwlock cv sema user user_pi shuttle
  Thread states: free sleep run onproc zomb stopped wait panic

ATTRIBUTES

  Target: kvm
  Module: genunix
  Interface Stability: Unstable

About

jwadams

Search

Top Tags
Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today