Thursday Apr 30, 2015

Few Random Solaris Commands : intrstat, croinfo, dlstat, fmstat, ..

Target: Solaris 11 and later. Some of these commands may work on earlier versions too.


Interrupt Statistics : intrstat utility

intrstat utility can be used to monitor interrupt activity generated by various hardware devices along with the CPU that serviced the interrupt and the CPU time spent in servicing those interrupts on a system. On a busy system, intrstat reported stats may help figure out which devices are busy at work, and keeping the system busy with interrupts.


.. [idle system] showing the interrupt activity on first two vCPUs ..

# intrstat -c 0-1 5

      device |      cpu0 %tim      cpu1 %tim
      cnex#0 |         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0
    hermon#0 |         0  0.0         0  0.0
    hermon#1 |         0  0.0         0  0.0
    hermon#2 |         0  0.0         0  0.0
    hermon#3 |         0  0.0         0  0.0
       igb#0 |         0  0.0         0  0.0
     ixgbe#0 |         0  0.0         0  0.0
   mpt_sas#0 |        18  0.0         0  0.0
      vldc#0 |         0  0.0         0  0.0

      device |      cpu0 %tim      cpu1 %tim
      cnex#0 |         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0
    hermon#0 |         0  0.0         0  0.0
    hermon#1 |         0  0.0         0  0.0
    hermon#2 |         0  0.0         0  0.0
    hermon#3 |         0  0.0         0  0.0
       igb#0 |         0  0.0         0  0.0
     ixgbe#0 |         0  0.0         0  0.0
   mpt_sas#0 |        53  0.2         0  0.0
      vldc#0 |         0  0.0         0  0.0

Check the outputs of the following as well.

# echo ::interrupts | mdb -k
# echo ::interrupts -d | mdb -k


Physical Location of Disk : croinfo & diskinfo commands

Both croinfo and diskinfo commands provide information about the chassis, receptacle, and occupant relative to all disks or a specific disk. Note that croinfo and diskinfo utilities share the same executable binary and function in a identical manner. The main difference being the defaults used by each of the utilities.


# croinfo
D:devchassis-path               t:occupant-type  c:occupant-compdev
------------------------------  ---------------  ---------------------
/dev/chassis//SYS/MB/HDD0/disk  disk             c0t5000CCA0125411FCd0
/dev/chassis//SYS/MB/HDD1/disk  disk             c0t5000CCA0125341F0d0
/dev/chassis//SYS/MB/HDD2       -                -
/dev/chassis//SYS/MB/HDD3       -                -
/dev/chassis//SYS/MB/HDD4/disk  disk             c0t5000CCA012541218d0
/dev/chassis//SYS/MB/HDD5/disk  disk             c0t5000CCA01248F0B8d0
/dev/chassis//SYS/MB/HDD6/disk  disk             c0t500151795956778Ed0
/dev/chassis//SYS/MB/HDD7/disk  disk             c0t5001517959567690d0

# diskinfo -oDcpd
D:devchassis-path               c:occupant-compdev     p:occupant-paths                                                               d:occupant-devices
------------------------------  ---------------------  -----------------------------------------------------------------------------  -----------------------------------------
/dev/chassis//SYS/MB/HDD0/disk  c0t5000CCA0125411FCd0  /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca0125411fd,0  /devices/scsi_vhci/disk@g5000cca0125411fc
/dev/chassis//SYS/MB/HDD1/disk  c0t5000CCA0125341F0d0  /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca0125341f1,0  /devices/scsi_vhci/disk@g5000cca0125341f0
/dev/chassis//SYS/MB/HDD2       -                      -                                                                              -
/dev/chassis//SYS/MB/HDD3       -                      -                                                                              -
/dev/chassis//SYS/MB/HDD4/disk  c0t5000CCA012541218d0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca012541219,0  /devices/scsi_vhci/disk@g5000cca012541218
/dev/chassis//SYS/MB/HDD5/disk  c0t5000CCA01248F0B8d0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca01248f0b9,0  /devices/scsi_vhci/disk@g5000cca01248f0b8
/dev/chassis//SYS/MB/HDD6/disk  c0t500151795956778Ed0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@4/disk@w500151795956778e,0  /devices/scsi_vhci/disk@g500151795956778e
/dev/chassis//SYS/MB/HDD7/disk  c0t5001517959567690d0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@8/disk@w5001517959567690,0  /devices/scsi_vhci/disk@g5001517959567690


Monitoring Network Traffic Statistics : dlstat command

dlstat command reports network traffic statistics for all datalinks or a specific datalink on a system.


# dlstat -i 5 net0
           net0  163.12M   39.93G  206.14M   43.63G
           net0      312  196.59K      146  370.80K
           net0      198  172.18K      121  121.98K
           net0      168   91.23K       93  195.57K

For the complete list of options along with examples, please consult the Solaris Documentation.


Fault Management : fmstat utility

Solaris Fault Manager gathers and diagnoses problems detected by the system software, and initiates self-healing activities such as disabling faulty components. fmstat utility can be used to check the statistics associated with the Fault Manager.

fmadm config lists out all active fault management modules that are currently participating in fault management. -m option can be used to report the diagnostic statistics related to a specific fault management module. fmstat without any option report stats from all fault management modules.


# fmstat 5
module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
cpumem-retire            0       0  1.0 8922.5  96   0     0     0    12b      0
disk-diagnosis        1342       0  1.1 8526.0  96   0     0     0      0      0
disk-transport           0       0  1.0 8600.3  96   1     0     0    56b      0
zfs-diagnosis          139      75  1.0 8864.5  96   0     4    12   672b   608b
zfs-retire             608       0  0.0   15.2   0   0     0     0     4b      0

# fmstat -m cpumem-retire 5
                NAME VALUE            DESCRIPTION
           auto_flts 0                auto-close faults received
            bad_flts 0                invalid fault events received
     cacheline_fails 0                cacheline faults unresolveable
      cacheline_flts 0                cacheline faults resolved
    cacheline_nonent 0                non-existent retires
   cacheline_repairs 0                cacheline faults repaired
      cacheline_supp 0                cacheline offlines suppressed


InfiniBand devices : List & Show Information about each device

ibv_devices lists out all available IB devices whereas ibv_devinfo shows information about all devices or a specific IB device.


# ibv_devices
    device                 node GUID
    ------              ----------------
    mlx4_0              0021280001cee63a
    mlx4_1              0021280001cee492
    mlx4_2              0021280001cee4aa
    mlx4_3              0021280001cee4ea

# ibv_devinfo -d mlx4_0
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.7.8130
        node_guid:                      0021:2800:01ce:e63a
        sys_image_guid:                 0021:2800:01ce:e63d
        vendor_id:                      0x02c9
        vendor_part_id:                 26428
        hw_ver:                         0xB0
        board_id:                       SUN0160000002
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 56
                        port_lid:               95
                        port_lmc:               0x00
                        link_layer:             IB

                port:   2
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 56
                        port_lid:               96
                        port_lmc:               0x00
                        link_layer:             IB

Other commands and utilities such as ibstatus, fwflash or cfgadm can also be used to retrieve similar information.


PCIe Hot-Plugging : hotplug command

When the hotplug service is enabled on a Solaris system, hotplug command to bring hot pluggable devices online or offline without physically adding or removing the device from the system.

The following command lists out the all physical [hotplug] connectors along with the current status.


# hotplug list -c
Connection           State           Description
IOU2-EMS2            ENABLED         PCIe-Native
IOU2-PCIE6           ENABLED         PCIe-Native
IOU2-PCIE7           EMPTY           PCIe-Native
IOU2-PCIE4           EMPTY           PCIe-Native
IOU2-PCIE1           EMPTY           PCIe-Native

For detailed instructions to hotplug a device, check the Solaris documentation out.

Fancy Separator Credit: jkneb

Wednesday Mar 26, 2014

Software Availability : Solaris Studio 12.4 Beta & ORAchk

First off, these are two unrelated softwares.

Solaris Studio 12.4 Beta

Nearly two-and-a-half years after the release of Solaris Studio 12.3, Oracle is gearing up for the next major release 12.4. In addition to the compiler and library optimizations to support the latest and greatest SPARC & Intel x64 hardware such as SPARC T5, M5, M6, Fujitsu's M10, and Intel's Ivy Bridge and Haswell line of servers, support for C++ 2011 language standard is one of the highlights of this forthcoming release. The complete list of features and enhancements in release 12.4 are documented in the What's New page.

Those who feel compelled to give the updated/enhanced compilers and tools a try, can get started right away by downloading the beta bits from the following location. This software is available for Solaris 10 & 11 running on SPARC, x86 hardware; and Linux 5 & 6 runnin g on x86/x64 hardware. Anyone can download this software for free.

     Oracle Solaris Studio 12.4 Beta Download

Don't forget to check the Release Notes out for the installation instructions, known issues, limitations and workarounds, features that were removed in this release and so on.

Here's a pointer to the documentation (preview): Oracle Solaris Studio 12.4 Information Library

Finally, should you run into any issue(s) or if you have questions about anything related, feel free to use the Solaris Studio Community Forum.

ORAchk 2.2.4 (formerly known as EXAchk)

ORAchk, the Oracle Configuration Audit Tool, enhances EXAchk tool's functionality, and replaces the existing & popular RACcheck tool. In addition to the top issues reported by users/customers, ORAchk proactively scans for known problems within Oracle Database, Sun systems (especially engineered systems) and Oracle E-Business Suite Financials.

While checking, ORAchk covers a wide range of areas such as OS kernel settings, database installations (single instance and RAC), performance, backup and recovery, storage setting, and so on.

ORAchk generated reports (mostly high level) show the system health risks with the ability to drill down into specific problems and offers recommendations specific to the environment and product configuration. Those who do not like sending this data back to Oracle should be happy to know that there is no phone home feature in this release.

Note that ORAchk is available only for the Oracle Premier Support Customers - meaning only those customers with appropriate support contracts can use this tool. So, if you are a Oracle customer with the ability to access the Oracle Support website, check the following pages out for additional information.

     ORAchk - Oracle Configuration Audit Tool
     ORAchk user's guide

Feel free to use the community forum to ask any related questions.

Saturday Dec 21, 2013

Measuring Network Bandwidth Using iperf

iperf is a simple, open source tool to measure the network bandwidth. It can test TCP or UDP throughput. Tools like iperf are useful to check the performance of a network real quick, by comparing the achieved bandwidth with the expectation. The example in this blog post is from a Solaris system, but the instructions and testing methodology are applicable on all supported platforms including Linux.

Download the source code from iperf's home page, and build the iperf binary. Those running Solaris 10 or later, can download the pre-built binary (file size: 245K) from this location to give it a quick try (right click and "Save Link As .." or similar option).

Testing methodology:

iperf's network performance measurements are based on the client-server communication model - hence requires establishing both a server and a client. The same iperf binary can be used to run the process in server and client modes.

  1. Start iperf in server mode
    iperf -s -i <interval>

    Option -s or --server starts the process in server mode. -i or --interval is the sampling interval in seconds.

  2. Start iperf in client mode, and test the network connection between client and the server with arbitrary data transfers.

    iperf -n <bytes> -i <interval> -c <ServerIP>

    Option -c or --client starts the process in client mode. Option -n or --bytes specify the number of bytes to transmit in bytes, KB (use suffix K) or MB (use suffix M). -i or --interval is the sampling interval in seconds. The last option is the IP address or the hostname of the server to connect to. By default, client connects to the server using TCP. -u or --udp switches to UDP.

  3. Check the network link speed on server and client, and compare the throughput achieved.

Check the man page out for the full list of options supported by iperf in client and server modes.

Here is a simple demonstration.

On server node:

iperfserv% dladm show-phys net0
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0

iperfserv% ifconfig net0 | grep inet
        inet netmask ffffff00 broadcast

iperfserv% ./iperf -v
iperf version 3.0-BETA5 (28 March 2013)SunOS iperfserv 5.11 11.1 sun4v sparc sun4v

iperfserv% ./iperf -s -i 1
Server listening on 5201

On client node:

client% dladm show-phys net0
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0

client% ifconfig net0 | grep inet
        inet netmask ffffff00 broadcast

client% ./iperf  -n 1024M  -i 1 -c
Connecting to host, port 5201
[  4] local port 63507 connected to port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   105 MBytes   875 Mbits/sec
[  4]   1.01-2.02   sec   112 MBytes   934 Mbits/sec
[  4]   2.02-3.00   sec   110 MBytes   934 Mbits/sec
[  4]   8.02-9.01   sec   110 MBytes   933 Mbits/sec
[  4]   9.01-9.27   sec  30.0 MBytes   934 Mbits/sec
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-9.27   sec  1.00 GBytes   927 Mbits/sec
[  4]   0.00-9.27   sec  1.00 GBytes   927 Mbits/sec

iperf Done.

At the same time, somewhat similar messages are written to stdout on the server node.

iperfserv% ./iperf  -s -i 1
Server listening on 5201
Accepted connection from, port 33457
[  5] local port 5201 connected to port 63507
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   104 MBytes   874 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   934 Mbits/sec
[  5]   2.00-3.00   sec   111 MBytes   934 Mbits/sec
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-9.28   sec  1.00 GBytes   927 Mbits/sec
[  5]   0.00-9.28   sec  1.00 GBytes   927 Mbits/sec
Server listening on 5201

The link speed is specified in Mbps (megabit per second). In the above example, the network link is operating at 1000 Mbps speed, and the achieved bandwidth is 927 Mbps, which is 92.7% of the advertised bandwidth.


  • It is not necessary to execute iperf in client and server modes as root or privileged user
  • In server mode, iperf uses port 5201 by default. It can be changed to something else using -p or --port option
  • Restart iperf server after each client test to get reliable, consistent results
  • Using iperf is just one of many ways to measure the network bandwidth. There are other tools such as uperf, ttcp, netperf, bwping, udpmon, tcpmon, .. just to name a few. Research and pick the one that best suits your requirement.

Benchmark announcements, HOW-TOs, Tips and Troubleshooting


« May 2015