Solaris Crash Analysis Tool 5.0 Release Notes
By danaf on Jun 18, 2008
Are you sitting down? We're very close to releasing Solaris CAT 5.0 to the public. We're waiting for one more approval and we'll push it out. The plan is to release the package at the end of June but I think we'll make mid-July. As an appetizer, here are the release notes for 5.0 so you can see what's coming.
The tool now supports the analysis of core dumps from Solaris 10 and above. This requires that they are analyzed on an x86 or x64 system for 32 bit core dumps and an x64 system for 64 bit core dumps.
Sun Cluster Support
clust command has been added to retrieve cluster specific information.
The new command syntax is as follows:
clust [flag] <cmd> [arg]
||reports available dbg options which can be enabled|
||reports important cluster addresses|
||reports enabled dbg buf which can be displayed|
||dumps specified dbg_buf, -a dumps possible previous data from from buffer when it has wrapped wrap|
||calculates delta between hrtime and tm (timestamp) found in the debug buffers.|
||finds and reports our did devices|
||reports heartbeat data|
||reports invocation flags add thread reporting|
||reports current cluster members|
||reports current paths and associated data|
||reports path manager data|
||dumps specified dbg_buf (no size calculation)|
zone command has been added to display
zone information in the core. Additionally, the
vfstab commands now have options to include
the zone name for the process/vfstab entry, or to select processes or vfstab
entries by zone. The new
task commands also have per-project options.
New Structure Command Flags
New flags were added to the
skma commands to
enable display of more details of structures elements. Those include:
||display the address of each structure element|
||display the offset of each structure element|
||display the size of each structure element|
||display the CTF module of each structure element|
||display the type of each structure element|
||display the index of each structure array element|
||display any gaps between structure elements|
||allows specification of the array element size (
Additionally, new options were added to the
||display the origin of the type. This could be from the CTF data in the core, or the stabs file name.|
||display all of the CTF versions of the type, including the names of the modules which include that version|
||display the module for each element of the type shown|
||display gaps between structure elements|
stype xck will compare all available CTF versions of a type and
display a list of the differing versions of the type in the core with a
list of the modules which use that version. The parenthesized numbers show
the number of elements, followed by the size of the type.
Additionally, types which are "dissonant" are shown. If two version of a type are dissonant, then they are not only different, but they have differing offsets or sizes for the same named members of that type.
Finally, if there is a region of memory where a pointer to another known type
was found, the "stype field" subcommand will help search for types which exist
in the CTF data which has that type at a known offset.
Some commands which have been documented as deprecated for a long time have now been eliminated from the tool. Specifically:
findstkcommand still exists as an alias, although it cannot be searched for with the
bigdump command is now deprecated.
Support has been added for user-defined symbols. These will supersede any
in-core symbols that would normally be shown. This allows specification of
names that might be useful to the user for regions of memory, including
treating it an an array. See the
section for further details.
This new command displays the autofs configuration in the core by traversing
the fnnode tree.
This new command lists the threads waiting for a condition variable.
This new command lists the NFS shares for the system.
This new command displays the pools present in the system. It can also be
used to get information about a specific pool. Pools may be selected by
name, pool id, or
This command prints a table of projects on the system. It can also be used
to list the tasks in a project. A project may be selected by name, project
kprojectstructure address, or those in a zone specified by name, zone id, or
This command is used to display resource controls. They can be displayed
rctl_setstructure address, or for a process specified by PID or
This command simply lists the threads waiting in a specified sleepq specified
by the provided sleepq_head structure address.
This command prints a table of tasks on the system. It can also be used to
display processes in the task. A task may be specified by task id,
taskstructure address, project id, or
This command is similar to the
sfmmucommand for displaying kernel virtual to physical address translation structures, but is for the x86/x64 architecture. It can be used to display PTEs, hments, htables, ptables, hat structures, or searching for translations for a specified virtual address.
zfs command has been added to retrieve zfs specific information.
This command lists the zones, including their id, name, root path, and status. If the -l flag is included, more data about the zone is
displayed. If the -z <zone> option is included, then only the specified zone is displayed. The <zone> may be specified a
s a zone address, id, or name.
colorcommand's options for
info2,now include the ability to set background colors and attributes.
Two new color classifications were added:
bad. The former is used to indicate the portion of data
which is supposed to be the redzone. The latter indicates "bad" data
such as data which should be 0xdeadbeef but is not in "kma buf" output.
The basic command format is:
color <class> <foreground> <background> <attrubutes>The <foreground> and <background> may be any one of
white. The options are positional, so the word
nonecan be used in place of a color to indicate not to set a color.
Attributes supported are
strikethrough. These may be
combined in a comma-separated list.
Some examples are:
color fatal red none bold color warning none none underline,bold color bad white red color alert green color info white greenNote that attribute support is dependent on the terminal emulator in use. For example, some of the attributes,
strikethroughin particular, may not be supported by the terminal emulator you are using. Some may display these attributes differently, such as using a brighter color instead of making the font bold. Other possibilities is that the terminal emulator may not support certain combinations of attributes.
In Solaris Nevada, the chip structure and notation was eliminated. A new
facility called the processor group (pgroup) was implemented in its place.
cpu -c [<pgroup>]
-cflag's meaning was changed to mean a pgroup specifier instead of a chip specifier on that version of Solaris.
This new flag allows display of CPUs by their hardware sharing
relationship. This matches against the pghw_hw type field, and dumps
the CPUs by this grouping. Valid <pghw_type>s are PGHW_IPIPE,
PGHW_CACHE, PGHW_FPU, PGHW_MPIPE, PGHW_MEMORY, and PGHW_CHIP. This
feature only works on Solaris Nevada.
cpu -h [<pghw_type>]
If the <pghw_type> is omitted, then a summary of the hardware relationships is displayed, only listing the CPU IDs and their troups. For example, from a system with an UltraSPARC-T1:
PGR_PHYSICAL, class:cmt(id:1), PGHW_IPIPE 0/0: 0 1 2 3 3/68: 4 5 6 7 4/70: 8 9 10 11 5/72: 12 13 14 15 6/74: 16 17 18 19 7/76: 20 21 22 23 8/78: 24 25 26 27 9/80: 28 29 30 31 PGR_PHYSICAL, class:cmt(id:1), PGHW_FPU 1/0: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 PGR_PHYSICAL, class:cmt(id:1), PGHW_MPIPE 2/0: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31The X/Y represents the pgroup ID and the instance number.
This new option for
dev node [<addr>]
devdisplays the tree of in_node structures rooted at
e_ddi_inst_state.ins_root, which is a fully-populated parallel tree to the dev_info tree.
Two new options have been added to this command to allow searching for
files based on
vnode, and based on
-n option for
findfiles allows searching
for processes using files whose
vnode matches the supplied
<vnode_addr>. This matching includes a
fn_dest, and an
-f option for
findfiles allows searching
for processes using files whose
<vfs_addr>. This matching includes a
v_vfsp, and an
The flags for findval have been altered to make the size specifiers
into single-character flags.
-vaoption has been replaced by
-V. Correspondingly, the
-paoption has been replaced by
A new option was added to allow specification of the size of the object
to be read for the (implicit)
-a case where a value is read
from the core. By default, sizeof(ulong) is read and displayed. The
-S <size> flag allows specification of 1, 2, 4, or 8-byte
chunks of data to be read instead.
ifconfcommand's output now includes IPMP group names.
kma statoutput, an asterisk at the end of the line for a cache is now used to indicate that setting
kmem_flagswould not make it into this cache's
cache_flagsdue to the
cache_cflagsset. This means that setting
kma usersto be run against this cache.
This new flag for
kma users -b <ltstamp> <htstamp>
kma usersallows selection of only data from the specified time range [
<htstamp>). The specified range is compared against the
bc_timestamp. The values are in the same units as the
kma users -f
kma usersprocesses allocated buffers. The new flag
-fcauses it to process free buffers instead. This is accomplished by walking the cache's magazines and slab freelists. Note that the cache must have one of the KMF_BUFTAG flags enabled to be able to get from a given free buffer to its kmem_bufctl_audit structure which contains the stack and thread data which
kstat xck [<filename>]
xcksubcommand causes the entire set of kstats to be walked, looking for any that match a class/module/name specification. Any that match are checked against a condition, and if they fail that check, they cause a message to be displayed, and the values which triggered it.
By default, the list of specifications are kept in the file
You can specify a different rules file on the command line. When it successfully
loads a rules file, it reports the number of rules found, then runs the checks.
The format of the rules file is a colon-separated list of the following six fields:
||the kstat class|
||the kstat module|
||the kstat name|
||the rule checked - see below for details|
||the severity of the rule if it fails, 0-9. This may be used to emphasize higher (numeric) severities over lower.|
||the error message to be displayed if the rule fails|
The rule is matched against the
kstat_named_t entries in the kstat.
A rule can specify <name><comparator><value> where <name>
is used to match against the
kstat_named_t's name, and it's value
is then compared against <value>. <comparator> can be any of
">", "<", ">=", "<=", "==", or "!=". <value> can be an integer, the
name of another
kstat_named_t within that kstat, or a string
enclosed in quotes.
<value> may also include a percentage calculation for a name.
device_error:::Hard Errors>0:9:device had hard errorsany kstat's whose class is "device_error", matching any module or name, and has a
kstat_named_tnamed "Hard Errors" which has a value greater than zero, report "disk had hard errors" using severity 9.
An example of a hit on this rule would be:
st54,err:Hard Errors>0(33):device had hard errorsThis shows the name of the kstat that triggered the report, the check that failed, the value in parentheses, and the error message.
net:::duplex!="full":9:not full duplexmatches class "net"'s
kstat_named_tof "duplex" and has a string value that is not "full", and reports "not full duplex" at severity 9.
A third example:
::biostats:buffers_locked_by_someone>1%buffer_cache_lookups:9:This matches any kstat named "biostats" and compares a
kstat_named_twith the name "buffers_locked_by_someone" against 1% of one named "buffer_cache_lookups", and reports if it's greater at severity 9. In this case, no specific error message is printed, just the rule and values.
mdumpnow prints a leading asterisk (\*) instead of a line which is a repeat of the previous line. This is done to help point out "interesting" data. See the
scatenv mdump_compressionsection for further details.
mdump now displays the "next" address at the end
of its output. This is done to show what the "next" address would be, and,
in the case where we ended with multiple compressed lines, to show that it
continued to the end of the requested data.
-Poption to the
mdumpand all of the
rdcommands allows reading from the core or system by physical address instead of virtual address.
Two new flags were added to this command which can be useful when the errorq
data isn't in the "normal" places. The -raw flag causes all queues to be
processed, which may include old errors, or possibly even junk.
This version of the
meminfo tree <proc>
meminfocommand works similarly to the
useroption, but only walks the process tree under the specified
<proc>, giving that trees totals.
Previously, you could select processes by their process address or PIDs.
You can now instead match by the command string. This is done by matching
the psargs for the process against the provided <cmd> substring.
meminfo user <cmd>
The process address/PID specification is distinguished from the command
substring based on the first character. If it is numeric, it is considered
a process address/PID list, otherwise, a command substring.
This new option to the
meminfo -m user
meminfocommand will display the memory layout for all the processes on the system.
This command was added to display the contents of an nvlist structure.
This new option to the
pagecommand will examine the page_freelists and page_cachelists and display a list of how many pages are in each of the per-color buckets, organized by page size.
This new option to the
page frag [<sizecode>]
pagecommand walks the
page_countersarrays and counts the free base constituent pages for a given pagesize. The output shows a number of free base constituent pages, followed by a count of the number of pages of size <sizecode> that have that many constituent pages free.
For example, if the chart showed:
free page count ==== ========== 512 35 511 112 ...that would mean that there are 35 large pages that have 512 of their constituent pages free, and 112 that have 511 free.
The last line of output for pages with 0 constituent pages free also includes a count of those pages by their pagesize. Since they're entirely in-use, it's possible that they're in-use as large pages already. The list shows how many are at each pagesize. Note that pages larger than the sizecode being displayed are factored by their size.
For example, when requesting size 1(64KB):
0 978433 8K:5927, 64K:284434, 512K:143528, 4M:104064, 32M:413150 (213M unknown)would appear to show 413150 32MB pages, when this is more than there are in the system. So instead, that value is divided by 512, since there are 512 32MB pages in a 64KB page. The result is:
0 978433 8K:5927, 64K:284434, 512K:17941, 4M:1626, 32M:808 (209M unknown)But in this case, the sum of the pages doesn't equal 978433.
The "unknown" entry is for entries in the page_counters for which a page cannot be found. These are typically parts of memory for which no page structure is allocated (memory used during boot), and thus the page size cannot be easily determined.
Two fragmentation percentages are calculated:
coalesce fragmentation - how fragmented completely-free large pages are into free constituent pages
relocate fragmentation - how non-free constituent pages are fragmenting large pages of the selected size
The coalesce fragmentation measure is based on 100% representing all constituent pages being the smallest pages. For the scaled fragmentation calculation, constituent pages which are sized between the smallest and the one being examined are counted at 1/rate where "rate" is how many smallest pages make up the current size. For example, fragmentation into 512K pages are counted at 1/64 the fragmentation of 8K pages.
For example, from the following fragmentation of 4M pages:
512 12964 free 8K:2067328, 64K:13552, 512K:4508, 4M:8151the calculation would be:
(2067328 + 13552 + 4508) / (12964 \* 512)Which would make 31.4179% of the total possible fragmentation. For the scaled fragmentation, using the same numbers, the calculation would be:
(2067328 + 13552 / 8 + 4508 / 64) / (12964 \* 512)which would make 31.1724% scaled coalesce fragmentation.
Relocate fragmentation is measured by summing the slots using this formula:
(count of pages in this slot / total number of partially-free pages) \* (count of subpages free for this slot / number of subpages in a page)Fully in-use (non-free) pages at slot 0, are not counted.
page mappers -P
page mappershas been replaced by the
Two new options were added to the
By default, only allocated buffers from the specified cache are scanned
for packet matches. The
-f flag causes both allocated and
free buffers to be scanned.
-s flag causes
pkma to display a summary of
the packet data seen. This includes counts of the following:
ethernet type ethernet source ethernet destination IPv4 protocol IPv4 source IPv4 destination IPv6 protocol IPv6 source IPv6 destination ICMPv4 type/code ICMPv6 type/code TCP source port TCP destination port UDP source port UDP destination port (R)ARP op ARP sender ethernet ARP target ethernet RARP sender ethernet RARP target ethernet ARP sender IP ARP target IP RARP sender IP RARP target IP
The output for each is shown with a count of occurrences in brackets. For example:
ethernet type: 0x800(IPv4), 0x806(ARP)The new scatenv setting
pdump_min_pktaffects how many of a given item must be seen before it is displayed. That is, if less than
pdump_min_pktwould be displayed in the brackets, the entry is not displayed. See the
pdump_min_pktsection for more details on this setting.
-l flag adds two more categories:
source packet half destination packet halfThe
-Lflag further adds the whole packet as a category.
proccommand has some options added to display processes' zone (
-z), project ID (
-j), task ID (
-k), or contract ID (
Additionally, processes may be selected by using the uppercase versions
of the above flags. To select processes by zone, use
-Z <zone_name>. Processes within a project may be
-J <proj_id>. Similarly, processes
within a task may be selected with
-K <task_id>. Finally,
processes may be selected by contract ID with
When dumping arrays or the array of hash buckets, the array members may
be separated by something other than the element size. This new option
allows specification of the size that the command will increment the
address of array elements.
sarray/shash -e <size>
By default, most of the time when the CPU structures are walked by the
tool, it starts at
cpu_listand walks the list of CPUs via the
Due to x86/x64 instructions being variably-sized, it can sometimes be
useful to see the actual bytes that make up the instruction. When this
is enabled, the bytes of the instruction are displayed with it.
Due to x86/x64 instructions being variably-sized, it can sometimes be
useful to see the size of the instruction. When this is enabled, the size
of the instruction is displayed with it.
dispqcommand previously displayed only CPUs with threads in their dispatch queues. This change shows all CPUs unless the
dispq_emptyis enabled. By default it is disabled.
Additionally, if any threads are pinned by the cpu_thread, they are also
shown in the
dispq output. Finally, if the cpu_dispthread doesn't
match the cpu_thread, and hasn't shown up in the pinned thread(s), it is also
mdumpnow compresses repeated lines to help find "interesting" data in the output. It now prints a leading asterisk (\*) instead of a line which is a repeat of the previous line. Further repititions are omitted. This behavior can be reverted to the original by disabling the
This also effects the output of
kma buf and
panic kmem similarly.
This setting affects how many of an item in
pkma -smust be seen before it is shown in the output. The default is 100. See the
pkma -ssection for more details on that command.
Some of the microstate information is now present in thread/lwp output if the
TP_MSACCT flag is set in the t_proc_flag - the t_mstate, ms_prev,
ms_state_start, and ms_start.
These are also displayed if the
thr_microstate setting is
This new subcommand lists segments with softlocks, and a total of softlocks
active in the system.
rwlock -L, this new option searches segkp for threads with the specified semaphore, eliminates those in the sleep queue for that semaphore, and lists those remaining as possible semaphore holders.
slistcommand now accepts a type of "none". This is intended to allow walking linked lists where the offset of the link pointer is known, but the type is not.
Only the element number and address are displayed for each. The
-c flag also works with this type name.
skmadumps buffers which are allocated by default, and if the
-fflag is used, then both free and allocated buffers are displayed. This new option to
skmaallows dumping only the buffers within a specified cache which are free.
Solaris 10 has a new streams feature called squeues. This new subcommand
stream [-l|-s] [-d] squeue [<squeue_addr>]
streamcommand allows examination of the squeues present.
In its base form,
stream squeue lists all the squeues on
the system, including the decoded fields in the structures.
The mblks present in the squeues may be displayed by specifying
stream -l sqeueue. This display honors the
str_pdump flags - thus it will not display the mblks
-l is specified and
A one-line summary of squeues present is displayed if the
flag is specified. This includes a summary of the mblks present on the
squeue. This flag overrides
-d is included, only squeues which have data (mblks) in
them are displayed.
Sometimes the crash copy of the SVM data may have invalid device sizes. The "-i"
option was added to instruct svm to ignore these errors and display info on
the meta device.
New subcommands were added to the
symbolscommand to allow the user to specify their own symbols. The command has three options:
add subcommand is how user-defined symbols are created.
By default it creates a word-sized symbol. The user can additionally specify
a size if something other than word size is desired.
If the symbol size is specified, the number of elements can be specified. In this case, the region is treated as an array with each element the size specified.
del subcommand is used to remove symbols from the
user-specified symbol table.
This new subcommand for
tlistdumps threads whose process has SKILLED set in their p_flag, indicating that a SIGKILL has been posted to the process.
This is implemented in
tlist rather than
often the process won't be present in the process list any longer, yet the
threads linger in the thread list, holding a pointer to the involved process.
This new subcommand for
tlistdumps threads which hae a t_pctcpu greater than the specified value. If no value is specified, 90% is used.
vfstabcommand now includes an option (
-F <fstype>) to specify the filesystem type. This will only list filesystems of the specified type
The device or mount point may now also be specified directly in the command.
If specified, only filesystems which exactly match the
whatishas been replaced by the
svm [-s <set>] [-d <devnum>]
-doption was added to support translation of Solaris Volume Manager's minor device numbers. In some cases these device numbers will seem odd and this option should help by providing the metaset wherein the device is defined and device instance within that metaset.
On Solaris 9 and up, the metaset name can now by provided as an argument to
The structures associated with SVM change frequently. This causes Solaris CAT
to dump incorrect information. We are in the process of changing the SVM code
to use CTF instead of static structures. In this release the metaset and
metadb info is now read using CTF.