Saturday Jun 11, 2016

Solaris API for Locality Group Observability

[ Related: 1st part @ Locality Group Observability on Solaris ]

As of this writing, Solaris has limited support for lgroup observability using programmatic interfaces. Applications can use the lgroup API to traverse the locality group hierarchy, discover the contents of each locality group and even affect thread and memory placement on desired lgroups.

The man page for liblgrp(3LIB) lists out the currently supported public interfaces, and a brief description for most of those interfaces can be obtained by running man -k lgroup in a shell. In order to use this API, applications must link with locality group library, liblgrp(3LIB).

The following sample code demonstrates lgroup API usage by making several lgrp_*() calls including lgrp_device_lgrps() to find the locality groups that are closest to the specified I/O device.

# cat -n iodevlgrp.c

     1  #include <stdio.h>
     2  #include <stdlib.h>
     3  #include <assert.h>
     4  #include <sys/lgrp_user.h>
     5  #include <sys/types.h>
     6
     7  int main(int argc, char **argv) {
     8
     9          if (argc != 2) {
    10                  fprintf(stderr, "Usage: %s \n", argv[0]);
    11                  exit(1);
    12          }
    13
    14          /*  lgroup interface version check */
    15          if (lgrp_version(LGRP_VER_CURRENT) != LGRP_VER_CURRENT) {
    16              fprintf(stderr, "\nBuilt with unsupported lgroup interface %d",
    17                  LGRP_VER_CURRENT);
    18              exit(1);
    19          }
    20
    21          lgrp_cookie_t cookie =lgrp_init(LGRP_VIEW_OS);
    22          lgrp_id_t node = lgrp_root(cookie);
    23
    24          /* refresh the cookie if stale */
    25          if ( lgrp_cookie_stale(cookie) ) {
    26                  lgrp_fini(cookie);
    27                  cookie =lgrp_init(LGRP_VIEW_OS);
    28          }
    29
    30          int nlgrps = lgrp_nlgrps(cookie);
    31          if ( nlgrps == -1 ) {
    32                  perror("\n lgrp_nlgrps");
    33                  lgrp_fini(cookie);
    34                  exit(1);
    35          }
    36          printf("Number of locality groups on the system: %d\n", (nlgrps-1));
    37
    38          /* lgroups closest to the target device */
    39          int numlgrps = lgrp_device_lgrps(argv[1], NULL, 0);
    40          if (numlgrps == -1 ){
    41                  fprintf(stderr, "I/O device: %s. ", argv[1]);
    42                  perror("lgrp_device_lgrps");
    43          } else {
    44                  printf("lgroups closest to the I/O device %s: ", argv[1]);
    45                  lgrp_id_t *lgrpids = (lgrp_id_t *)calloc(numlgrps, sizeof(lgrp_id_t));
    46                  assert(lgrpids != NULL);
    47                  lgrp_device_lgrps(argv[1], lgrpids, numlgrps);
    48                  for (int i = 0; i < numlgrps; ++i) {
    49                          printf(" %d ", lgrpids[i]);
    50                  }
    51                  free(lgrpids);
    52          }
    53          lgrp_fini(cookie);
    54          printf("\n");
    55          return 0;
    56
    57  }


% cc -o iodevlgrp -llgrp iodevlgrp.c

% ./iodevlgrp /dev/ixgbe0
Number of locality groups on the system: 2
lgroups closest to the I/O device /dev/ixgbe0:  1

% lgrpinfo -d /dev/ixgbe0
lgroup ID : 1

% ./iodevlgrp /dev/ixgbe1
Number of locality groups on the system: 2
lgroups closest to the I/O device /dev/ixgbe1:  2

% lgrpinfo -d /dev/ixgbe1
lgroup ID : 2

Source: Oracle Solaris 11.3 Programming Interfaces Guide

Saturday Apr 23, 2016

Solaris 11: Few Random Commands

Logical Domains
Dependencies & Dependents
Network & Power Consumption Stats
HBA Listing

Dependencies

In the following example, dummy domain depends on two domains -- primary domain for virtual disks and virtual network devices, and dom2 for virtual disks and SR-IOV virtual functions (VF).

# ldm list-dependencies dummy
DOMAIN            DEPENDENCY        TYPE
dummy             primary           VDISK,VNET
                  dom2              VDISK,IOV

-l option displays the actual devices.

Dependents

-r option in list-dependencies sub-command shows the dependents for the logical domain(s). -l option displays the actual devices in here too.

eg.,

# ldm list-dependencies -r -l dom2
DOMAIN            DEPENDENT         TYPE           DEVICE
dom2              dummy             VDISK          primary-vds0/vdisk0:vol_dummy_disk0
                                    VDISK          service-vds0/vdisk0:vol_dummy_disk0
                                    IOV            /SYS/IOU1/PCIE14/IOVIB.PF0.VF1
                                    IOV            /SYS/IOU1/EMS4/CARD/NET0/IOVNET.PF0.VF1

Network Statistics

list-netstat sub-command in ldm utility displays the statistics for all the network devices that are configured in the domain(s).

eg.,

# ldm ls-netstat dom2
DOMAIN
dom2

NAME               IPACKETS     RBYTES       OPACKETS     OBYTES
----               --------     ------       --------     ------
net4               444.42M      61.27G       444.42M      7.12G
net1               13.20M       4989.22M     0            0
net0               0            0            0            0
..
ldoms-net0.vf1     37.62M       6.44G        49.15K       3.97M
ldoms-net0.vf0     523.34K      90.15M       62.85K       14.67M
ldoms-net1.vf15    0            0            0            0
..

List HBAs

list-hba sub-command in ldm utility lists out the physical SCSI HBA initiator ports in the target domain. -t option shows SCSI transport medium type such as FC or SAS.

eg.,

# ldm ls-hba -t  primary
NAME                                                 VSAN
----                                                 ----
/SYS/IOU0/EMS1/CARD/SCSI/HBA0/PORT1
        init-port w5080020000016331
        Transport Protocol SAS
/SYS/IOU0/EMS1/CARD/SCSI/HBA0/PORT2
        init-port w5080020000016331
        Transport Protocol SAS
...

Check the latest man page for ldm(1M) for the complete list of options for those sub-commands.

Power Consumption Statistics

ldmpower command shows the breakdown of power consumed by processors and memory in different domains or in a given domain. Note that ldmpower is an independent command, not part of ldm utility.

eg.,

# ldmpower -c processors -c memory -l primary
Processor Power Consumption in Watts
DOMAIN          15_SEC_AVG      30_SEC_AVG      60_SEC_AVG
primary         313             315             317

Memory Power Consumption in Watts
DOMAIN          15_SEC_AVG      30_SEC_AVG      60_SEC_AVG
primary         594             595             595

As usual, check the man page ldmpower(1M) for details.


PCI Devices

Scan & List

scanpci utility scans PCI buses and reports configuration settings for each PCI device that it finds.

eg.,

# scanpci -v

pci bus 0x0002 cardnum 0x06 function 0x00: vendor 0x111d device 0x80ba
 Integrated Device Technology, Inc. [IDT] Device unknown
 CardVendor 0x0000 card 0x0000 (Card unknown)
  STATUS    0x0010  COMMAND 0x0147
  CLASS     0x06 0x04 0x00  REVISION 0x03
  BIST      0x00  HEADER 0x01  LATENCY 0x00  CACHE 0x10
  MAX_LAT   0x00  MIN_GNT 0x03  INT_PIN 0x00  INT_LINE 0x00
  Bus: primary=02, secondary=03, subordinate=72, sec-latency=0
  I/O behind bridge: 00000000-03ffffff
  Memory behind bridge: 00100000-2000ffff
  Prefetchable memory behind bridge: 100000000-777f0ffff

pci bus 0x0003 cardnum 0x00 function 0x00: vendor 0x8086 device 0x1521
 Intel Corporation I350 Gigabit Network Connection
 CardVendor 0x108e card 0x7b18 (Oracle/SUN, Quad Port GbE PCIe 2.0 Low Profile Adapter, UTP)
  STATUS    0x0010  COMMAND 0x0146
  CLASS     0x02 0x00 0x00  REVISION 0x01
  BIST      0x00  HEADER 0x80  LATENCY 0x00  CACHE 0x10
  BASE0     0x00100000 SIZE 1048576  MEM
  BASE3     0x00200000 SIZE 16384  MEM
  BASEROM   0x00280000 SIZE 524288
  MAX_LAT   0x00  MIN_GNT 0x00  INT_PIN 0x01  INT_LINE 0x00

...

SRU
IDRs

Detect & Show Installed

Identifying Solaris 11 SRU level is one of the popular topics and it was covered in Solaris 11 documentation, My Oracle Support (MOS) and elsewhere by numerous bloggers. After all this, we still have to wonder why there isn't a conscious effort by the development team that owns pkg utility to make this a bit easier for normal human beings. In any case, pkg list entire or pkg info entire commands show a cryptic version string with a bunch of information encoded including the SRU level.

The second numeral in "branch" version represents the Solaris update whereas the third numeral is the SRU.

eg.,

In the following example, Solaris update is 3 (running Solaris 11 - hence it is a Solaris 11.3 system) and current SRU is 5. (I underlined every alternate field)

# pkg info entire | grep -i branch
        Branch: 0.175.3.5.0.6.0

IDRs

IDRs (Interim Diagnostic Reliefs) are in reality package updates whose names usually start with "idr" - therefore, just checking for the string "idr" in the list of installed packages on the system is sufficient to list out what IDRs were installed on the target system.

# pkg list idr*
pkg list: No packages matching 'idr*' installed <-- no IDRs installed

(Ticket Stub CSS Theme Credit: Kia Skretteberg)

Tuesday Jun 30, 2015

Oracle Database : Profiling PL/SQL Code

AWR gathers and reports performance statistics that are useful in monitoring and tuning databases. Even though the combination of AWR/ASH reports and explain plans are very useful in analyzing the performance of queries, their usefulness is limited when dealing with PL/SQL-heavy applications. AWR and ASH reports help figure out if the database is spending significant amount of time and/or CPU cycles in executing PL/SQL code and in what PL/SQL modules -- however to identify performance bottlenecks withinin PL/SQL modules/routines and to pin-point hot spots in PL/SQL code at line level, DBMS_PROFILER, a database package, might be the right tool to use. Also check the documentation of DBMS_HPROF package out (Credit to guest for the tip/suggestion).

Rest of this post lists out the steps involved (on a high level) in generating the profile using DBMS_PROFILER; and subsequently generating a report or extracting relevant pieces of information.

Steps:

  1. [Pre-11g env only] Create PLAN_TABLE as SYS user

    SQL> @?/rdbms/admin/utlxplan.sql
    SQL> GRANT ALL ON sys.plan_table TO public;
    SQL> CREATE PUBLIC SYNONYM plan_table FOR sys.plan_table;
    
  2. Create PL/SQL Profiler tables under the application schema

    As application user:

    SQL> @?/rdbms/admin/proftab.sql
    
  3. Install DBMS_PROFILER package as SYS user, if missing

    SQL> @?/rdbms/admin/profload.sql
    
  4. Start PL/SQL Profiler

    As application user:

    SQL> EXEC DBMS_PROFILER.START_PROFILER('SOME_COMMENT');
    
  5. Execute one or more transactions to be profiled

  6. Stop the PL/SQL Profiler

    As application user:

    SQL> EXEC DBMS_PROFILER.STOP_PROFILER;
    
  7. At this point, there are couple of options to analyze the profile.

    • Easy option - if you have access to My Oracle Support (MOS), download the profiler.sql from MOS Document ID 243755.1 "Script to produce HTML report with top consumers out of PL/SQL Profiler DBMS_PROFILER data", and execute.

      As application user:

      SQL> profiler.sql
      

      Select appropriate runid from the list displayed on stdout. profiler.sql script generates an HTML report that shows top time consumers for the duration of the execution of the PL/SQL Profiler run.

    • Not-so-Easy option - refer to the DBMS_PROFILER documentation to extract relevant information and details such as top time consumers, top lines sorted by total time, profiled PL/SQL module list, top modules etc.,

      Checking the documentation out for the following three key tables is a good start -- PLSQL_PROFILER_RUNS (information related to a profiling session), PLSQL_PROFILER_UNITS (information about each PL/SQL unit that was executed during the profiler session) and PLSQL_PROFILER_DATA (execution statistics for each line of code in PL/SQL units). Ancillary tables: DBA_SOURCE and PLAN_TABLE.

Thursday Apr 30, 2015

Few Random Solaris Commands : intrstat, croinfo, dlstat, fmstat, ..

Target: Solaris 11 and later. Some of these commands may work on earlier versions too.

-1-


Interrupt Statistics : intrstat utility

intrstat utility can be used to monitor interrupt activity generated by various hardware devices along with the CPU that serviced the interrupt and the CPU time spent in servicing those interrupts on a system. On a busy system, intrstat reported stats may help figure out which devices are busy at work, and keeping the system busy with interrupts.

eg.,

.. [idle system] showing the interrupt activity on first two vCPUs ..

# intrstat -c 0-1 5

      device |      cpu0 %tim      cpu1 %tim
-------------+------------------------------
      cnex#0 |         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0
    hermon#0 |         0  0.0         0  0.0
    hermon#1 |         0  0.0         0  0.0
    hermon#2 |         0  0.0         0  0.0
    hermon#3 |         0  0.0         0  0.0
       igb#0 |         0  0.0         0  0.0
     ixgbe#0 |         0  0.0         0  0.0
   mpt_sas#0 |        18  0.0         0  0.0
      vldc#0 |         0  0.0         0  0.0

      device |      cpu0 %tim      cpu1 %tim
-------------+------------------------------
      cnex#0 |         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0
    hermon#0 |         0  0.0         0  0.0
    hermon#1 |         0  0.0         0  0.0
    hermon#2 |         0  0.0         0  0.0
    hermon#3 |         0  0.0         0  0.0
       igb#0 |         0  0.0         0  0.0
     ixgbe#0 |         0  0.0         0  0.0
   mpt_sas#0 |        53  0.2         0  0.0
      vldc#0 |         0  0.0         0  0.0
^C

Check the outputs of the following as well.

# echo ::interrupts | mdb -k
# echo ::interrupts -d | mdb -k

-2-


Physical Location of Disk : croinfo & diskinfo commands

Both croinfo and diskinfo commands provide information about the chassis, receptacle, and occupant relative to all disks or a specific disk. Note that croinfo and diskinfo utilities share the same executable binary and function in a identical manner. The main difference being the defaults used by each of the utilities.

eg.,

# croinfo
D:devchassis-path               t:occupant-type  c:occupant-compdev
------------------------------  ---------------  ---------------------
/dev/chassis//SYS/MB/HDD0/disk  disk             c0t5000CCA0125411FCd0
/dev/chassis//SYS/MB/HDD1/disk  disk             c0t5000CCA0125341F0d0
/dev/chassis//SYS/MB/HDD2       -                -
/dev/chassis//SYS/MB/HDD3       -                -
/dev/chassis//SYS/MB/HDD4/disk  disk             c0t5000CCA012541218d0
/dev/chassis//SYS/MB/HDD5/disk  disk             c0t5000CCA01248F0B8d0
/dev/chassis//SYS/MB/HDD6/disk  disk             c0t500151795956778Ed0
/dev/chassis//SYS/MB/HDD7/disk  disk             c0t5001517959567690d0

# diskinfo -oDcpd
D:devchassis-path               c:occupant-compdev     p:occupant-paths                                                               d:occupant-devices
------------------------------  ---------------------  -----------------------------------------------------------------------------  -----------------------------------------
/dev/chassis//SYS/MB/HDD0/disk  c0t5000CCA0125411FCd0  /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca0125411fd,0  /devices/scsi_vhci/disk@g5000cca0125411fc
/dev/chassis//SYS/MB/HDD1/disk  c0t5000CCA0125341F0d0  /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca0125341f1,0  /devices/scsi_vhci/disk@g5000cca0125341f0
/dev/chassis//SYS/MB/HDD2       -                      -                                                                              -
/dev/chassis//SYS/MB/HDD3       -                      -                                                                              -
/dev/chassis//SYS/MB/HDD4/disk  c0t5000CCA012541218d0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca012541219,0  /devices/scsi_vhci/disk@g5000cca012541218
/dev/chassis//SYS/MB/HDD5/disk  c0t5000CCA01248F0B8d0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca01248f0b9,0  /devices/scsi_vhci/disk@g5000cca01248f0b8
/dev/chassis//SYS/MB/HDD6/disk  c0t500151795956778Ed0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@4/disk@w500151795956778e,0  /devices/scsi_vhci/disk@g500151795956778e
/dev/chassis//SYS/MB/HDD7/disk  c0t5001517959567690d0  /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@8/disk@w5001517959567690,0  /devices/scsi_vhci/disk@g5001517959567690

-3-


Monitoring Network Traffic Statistics : dlstat command

dlstat command reports network traffic statistics for all datalinks or a specific datalink on a system.

eg.,

# dlstat -i 5 net0
           LINK    IPKTS   RBYTES    OPKTS   OBYTES
           net0  163.12M   39.93G  206.14M   43.63G
           net0      312  196.59K      146  370.80K
           net0      198  172.18K      121  121.98K
           net0      168   91.23K       93  195.57K
^C

For the complete list of options along with examples, please consult the Solaris Documentation.

-4-


Fault Management : fmstat utility

Solaris Fault Manager gathers and diagnoses problems detected by the system software, and initiates self-healing activities such as disabling faulty components. fmstat utility can be used to check the statistics associated with the Fault Manager.

fmadm config lists out all active fault management modules that are currently participating in fault management. -m option can be used to report the diagnostic statistics related to a specific fault management module. fmstat without any option report stats from all fault management modules.

eg.,

# fmstat 5
module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
cpumem-retire            0       0  1.0 8922.5  96   0     0     0    12b      0
disk-diagnosis        1342       0  1.1 8526.0  96   0     0     0      0      0
disk-transport           0       0  1.0 8600.3  96   1     0     0    56b      0
...
...
zfs-diagnosis          139      75  1.0 8864.5  96   0     4    12   672b   608b
zfs-retire             608       0  0.0   15.2   0   0     0     0     4b      0
...
...

# fmstat -m cpumem-retire 5
                NAME VALUE            DESCRIPTION
           auto_flts 0                auto-close faults received
            bad_flts 0                invalid fault events received
     cacheline_fails 0                cacheline faults unresolveable
      cacheline_flts 0                cacheline faults resolved
    cacheline_nonent 0                non-existent retires
   cacheline_repairs 0                cacheline faults repaired
      cacheline_supp 0                cacheline offlines suppressed
	...
	...

-5-


InfiniBand devices : List & Show Information about each device

ibv_devices lists out all available IB devices whereas ibv_devinfo shows information about all devices or a specific IB device.

eg.,

# ibv_devices
    device                 node GUID
    ------              ----------------
    mlx4_0              0021280001cee63a
    mlx4_1              0021280001cee492
    mlx4_2              0021280001cee4aa
    mlx4_3              0021280001cee4ea

# ibv_devinfo -d mlx4_0
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.7.8130
        node_guid:                      0021:2800:01ce:e63a
        sys_image_guid:                 0021:2800:01ce:e63d
        vendor_id:                      0x02c9
        vendor_part_id:                 26428
        hw_ver:                         0xB0
        board_id:                       SUN0160000002
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 56
                        port_lid:               95
                        port_lmc:               0x00
                        link_layer:             IB

                port:   2
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 56
                        port_lid:               96
                        port_lmc:               0x00
                        link_layer:             IB

Other commands and utilities such as ibstatus, fwflash or cfgadm can also be used to retrieve similar information.

-6-


PCIe Hot-Plugging : hotplug command

When the hotplug service is enabled on a Solaris system, hotplug command to bring hot pluggable devices online or offline without physically adding or removing the device from the system.

The following command lists out the all physical [hotplug] connectors along with the current status.

eg.,

# hotplug list -c
Connection           State           Description
________________________________________________________________________________
IOU2-EMS2            ENABLED         PCIe-Native
IOU2-PCIE6           ENABLED         PCIe-Native
IOU2-PCIE7           EMPTY           PCIe-Native
IOU2-PCIE4           EMPTY           PCIe-Native
IOU2-PCIE1           EMPTY           PCIe-Native

For detailed instructions to hotplug a device, check the Solaris documentation out.



Fancy Separator Credit: jkneb

Tuesday Dec 23, 2014

Solaris Studio : C/C++ Dynamic Analysis

First, a reminder - Oracle Solaris Studio 12.4 is now generally available. Check the Solaris Studio 12.4 Data Sheet before downloading the software from Oracle Technology Network.

Dynamic Memory Usage Analysis

Code Analyzer tool in Oracle Solaris Studio compiler suite can analyze static data, dynamic memory access data, and code coverage data collected from binaries that were compiled with the C/C++ compilers in Solaris Studio 12.3 or later. Code Analyzer is supported on Solaris and Oracle Enterprise Linux.

Refer to the static code analysis blog entry for a quick summary of steps involved in performing static analysis. The focus of this blog entry is the dynamic portion of the analysis. In this context, dynamic analysis is the evaluation of an application during runtime for memory related errors. Main objective is to find and debug memory management errors -- robustness and security assurance are nice side effects however limited their extent is.

Code Analyzer relies on another primary Solaris Studio tool, discover, to find runtime errors that are often caused by memory mismanagement. discover looks for potential errors such as accessing outside the bounds of the stack or an array, unallocated memory reads and writes, NULL pointer deferences, memory leaks and double frees. Full list of memory management issues analyzed by Code Analyzer/discover is at: Dynamic Memory Access Issues

discover performs the dynamic analysis by instrumenting the code so that it can keep track of memory operations while the binary is running. During runtime, discover monitors the application's use of memory by interposing on standard memory allocation calls such as malloc(), calloc(), memalign(), valloc() and free(). Fatal memory access errors are detected and reported immediately at the instant the incident occurs, so it is easy to correlate the failure with actual source. This behavior helps in detecting and fixing memory management problems in large applications with ease somewhat. However the effectiveness of this kind of analysis highly depends on the flow of control and data during the execution of target code - hence it is important to test the application with variety of test inputs that may maximize code coverage.

High-level steps in using Code Analyzer for Dynamic Analysis

Given the enhancements and incremental improvements in analytical tools, Solaris Studio 12.4 is recommended for this exercise.

  1. Build the application with debug flags

    –g (C) or -g0 (C++) options generate debug information. It enables Code Analyzer to display source code and line number information for errors and warnings.

    • Linux users: specify –xannotate option on compile/link line in addition to -g and other options
  2. Instrument the binary with discover

    % discover -a -H <filename>.%p.html -o <instrumented_binary> <original_binary>

    where:

    • -a : write the error data to binary-name.analyze/dynamic directory for use by Code Analyzer
    • -H : write the analysis report to <filename>.<pid>.html when the instrumented binary was executed. %p expands to the process id of the application. If you prefer the analysis report in a plain text file, use -w <filename>.%p.txt instead
    • -o : write the instrumented binary to <instrumented_binary>

    Check Command-Line Options page for the full list of discover supported options.

  3. Run the instrumented binary

    .. to collect the dynamic memory access data.

    % ./<instrumented_binary> <args>

  4. Finally examine the analysis report for errors and warnings

Example

The following example demonstrates the above steps using Solaris Studio 12.4 C compiler and discover command-line tool. Same code was used to demonstrate static analysis steps as well.

Few things to be aware of:

  • If the target application preloads one or more functions using LD_PRELOAD environment variable that discover tool need to interpose on for dynamic analysis, the resulting analysis may not be accurate.
  • If the target application uses runtime auditing using LD_AUDIT environment variable, this auditing will conflict with discover tool's use of auditing and may result in undefined behavior.

Reference & Recommended Reading:

  1. Oracle Solaris Studio 12.4 : Code Analyzer User's Guide
  2. Oracle Solaris Studio 12.4 : Discover and Uncover User's Guide

Friday Nov 28, 2014

Solaris Studio 12.4 : C/C++ Static Code Analysis

First things first -- Oracle Solaris Studio 12.4 is now generally available. One of the key features of this release is the support for the latest industry standards including C++11, C11 and OpenMP 4.0. Check the Solaris Studio 12.4 Data Sheet before downloading the software from Oracle Technology Network.

Static Code Analysis

Code Analyzer tool in Oracle Solaris Studio compiler suite can analyze static data, dynamic memory access data, and code coverage data collected from binaries that were compiled with the C/C++ compilers in Solaris Studio 12.3 or later. Code Analyzer is supported on Solaris and Oracle Enterprise Linux.

Primary focus of this blog entry is the static code analysis.

Static code analysis is the process of detecting common programming errors in code during compilation. The static code checking component in Code Analyzer looks for potential errors such as accessing outside the bounds of the array, out of scope variable use, NULL pointer deferences, infinite loops, uninitialized variables, memory leaks and double frees. The following webpage in Solaris Studio 12.4: Code Analyzer User's Guide has the complete list of errors with examples.

    Static Code Issues analyzed by Code Analyzer

High-level steps in using Code Analyzer for Static Code analysis

Given the enhancements and incremental improvements in analysis tools, Solaris Studio 12.4 is recommended for this exercise.

  1. Collect static data

    Compile [all source] and link with –xprevise=yes option.

    • when using Solaris Studio 12.3 compilers, compile with -xanalyze=code option.
    • Linux users: specify –xannotate option on compile/link line in addition to -xprevise=yes|-xanalyze=code.

    During compilation, the C/C++ compiler extracts static errors automatically, and writes the error information to the sub-directory in <binary-name>.analyze directory.

  2. Analyze the static data

    Two options available to analyze and display the errors in a report format.

Example

The following example demonstrates the above steps using Solaris Studio 12.4 C compiler and codean command-line tool.

Few things to be aware of:

  • compilers may not be able to detect all of the static errors in target code especially if the errors are complex.
  • some errors depend on data that is available only at runtime -- perform dynamic analysis as well.
  • some errors are ambiguous, and also might not be actual errors -- expect few false-positives.

Reference & Recommended Reading:
    Oracle Solaris Studio 12.4 Code Analyzer User's Guide

Saturday May 10, 2014

Solaris 11.2 Highlights [Part 2] in 4 Minutes or Less

Part 1: Solaris 11.2 Highlights in 6 Minutes or Less

Highlights contd.,

Package related ..

Minimal Set of System Packages

For the past few years, it is one of the hot topics -- what is the bare minimum [set of packages] needed to run applications. There were a number of blog posts and few technical articles around creating minimal Solaris configurations. Finally users/customers who wish to have their OS installed with minimal set of required system packages for running most of the applications in general, can just install solaris-minimal-server package and not worry about anything else such as removing unwanted packages.

# pkg install pkg:/group/system/solaris-minimal-server

Oracle Database Pre-requisite Package

Until Solaris 11.1, it is up to the users to check the package dependencies and make sure to have those installed before attempting to install Oracle database software especially using graphic installer. Solaris 11.2 frees up the users from the burden of checking and installing individual [required] packages by providing a brand new package called oracle-rdbms-server-12cR1-preinstall. Users just need to install this package for a smoother database software installation later.

# pkg install pkg:/group/prerequisite/oracle/oracle-rdbms-server-12cR1-preinstall

Mirroring a Package Repository

11.2 provides the ability to create local IPS package repositories and keeps them in synch with the IPS package repositories hosted publicly by Oracle Corporation. The key in achieving this is the SMF service svc:/application/pkg/mirror. The following webpage has the essential steps listed on a high-level.

How to Automatically Copy a Repository From the Internet

Another enhancement is the cloning of a package repository using --clone option of pkgrecv command.

Observability related ..

Network traffic diagnostics:

A brand new command, ipstat(1M), reports IP traffic statistics.

# ipstat -?
Usage:	ipstat [-cmnrt] [-a address[,address...]] [-A address[,address...]]
[-d d|u] [-i interface[,interface...]] [-l nlines] [-p protocol[,protocol...]]
[-s key | -S key] [-u R|K|M|G|T|P] [-x opt[=val][,opt[=val]...]]

# ipstat -uM 5

SOURCE                     DEST                       PROTO    INT        BYTES
etc5mdbadm01.us.oracle.com etc2m-appadm01.us.oracle.c TCP      net8       76.3M
etc2m-appadm01.us.oracle.c etc5mdbadm01.us.oracle.com TCP      net8        0.6M
dns1.us.oracle.com         etc2m-appadm01.us.oracle.c UDP      net8        0.0M
169.254.182.76             169.254.182.77             UDP      net20       0.0M
...

Total: bytes in: 76.3M bytes out:  0.6M

Another new command, tcpstat(1M), reports TCP and UDP traffic statistics.

# tcpstat -?
Usage:	tcpstat [-cmnrt] [-a address[,...]] [-A address[,...]] [-d d|u] [-i pid[,...]] 
[-l nlines] [-p port[,...]] [-P port[,...]] [-s key | -S key] [-x opt[=val][,...]] 
[-z zonename[,...]] [interval [count]]

# tcpstat 5

ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global      1267 TCP    etc5mdbadm01.us.  42972 etc2m-appadm01.u     22   84.3M
global      1267 TCP    etc2m-appadm01.u     22 etc5mdbadm01.us.  42972   48.0K
global      1089 UDP    169.254.182.76      161 169.254.182.77    33436  137.0 
global      1089 UDP    169.254.182.77    33436 169.254.182.76      161   44.0 
...
...

Total: bytes in: 84.3M bytes out: 48.4K

# tcpstat -i 43982 5		<-- TCP stats for a given pid

ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global     43982 TCP    etc2m-appadm01.u  43524 etc5mdbadm02.us.     22   73.7M
global     43982 TCP    etc5mdbadm02.us.     22 etc2m-appadm01.u  43524   41.9K

Total: bytes in: 42.1K bytes out: 73.7M

Up until 11.1, it is not so straight-forward to figure out what process created a network endpoint -- one has to rely on a combination of commands such as netstat, pfiles or lsof and proc filesystem (/proc) to extract that information. Solaris 11.2 attempts to make it easy by enhancing the existing tool netstat(1M). Enhanced netstat(1M) shows what user, pid created and control a network endpoint. -u is the magic flag.

#  netstat -aun			<-- notice the -u flag in netstat command; and User, Pid, Command columns in the output

UDP: IPv4
   Local Address        Remote Address      User    Pid      Command       State
-------------------- -------------------- -------- ------ -------------- ----------
      *.*                                 root        162 in.mpathd      Unbound
      *.*                                 netadm      765 nwamd          Unbound
      *.55388                             root        805 picld          Idle
	...
	...

TCP: IPv4
   Local Address        Remote Address      User     Pid     Command     Swind  Send-Q  Rwind  Recv-Q    State
-------------------- -------------------- -------- ------ ------------- ------- ------ ------- ------ -----------
10.129.101.1.22      10.129.158.100.38096 root       1267 sshd           128872      0  128872      0 ESTABLISHED
192.168.28.2.49540   192.168.28.1.3260    root          0       2094176      0 1177974      0 ESTABLISHED
127.0.0.1.49118            *.*            root       2943 nmz                 0      0 1048576      0 LISTEN
127.0.0.1.1008             *.*            pkg5srv   16012 httpd.worker        0      0 1048576      0 LISTEN
	...

[x86 only] Memory Access Locality Characterization and Analysis

Solaris 11.2 introduced another brand new tool, numatop(1M), that helps in characterizing the NUMA behavior of processes and threads on systems with Intel Westmere, Sandy Bridge and Ivy Bridge processors. If not installed by default, install the numatop package as shown below.

# pkg install pkg:/diagnostic/numatop

Performance related ..

This is a grey area - so, just be informed that there are some ZFS and Oracle database related performance enhancements.

Starting with 11.2, ZFS synchronous write transactions are committed in parallel, which should help improve the I/O throughput.

Database startup time has been greatly improved in Solaris 11 releases -- it's been further improved in 11.2. Customers with databases that use hundreds of Gigabytes or Terabyte(s) of memory will notice the improvement to the database startup times. Other changes to asynchronous I/O, inter-process communication using event ports etc., help improve the performance of the recent releases of Oracle database such as 12c.

Miscellaneous ..

Java 8

Java 7 is still the default in Solaris 11.2 release, but Java 8 can be installed from the IPS package repository.

eg.,

# pkg install pkg:/developer/java/jdk-8		<-- Java Development Kit
# pkg install pkg:/runtime/java/jre-8		<-- Java Runtime

Bootable USB Media

Solaris 11.2 introduces the support for booting SPARC systems from USB media. Use Solaris Distribution Constructor (requires distribution-constructor package) to create the USB bootable media, or copy a bootable/installation image to the USB media using usbcopy(1M) and dd(1M) commands.

Oracle Hardware Management Pack

Oracle Hardware Management Pack is a set of tools that are integrated into the Solaris OS distribution, that show the existing hardware configuration, help configure hardware RAID volumes, update server firmware, configure ILOM service processor, enable monitoring the hardware using existing tools etc., Look for pkg:/system/management/hmp/hmp-* packages.

Few other interesting packages:

Parallel implementation of bzip2 : compress/pbzip2
NVM Express (nvme) utility : system/storage/nvme-utilities
Utility to administer cluster of servers : terminal/cssh

Tuesday Apr 29, 2014

Solaris 11.2 Highlights [Part 1] in 6 Minutes or Less

This is not the complete list, of course. Just a few hand-picked ones.

First things first, Solaris 11.2 beta is out.

URLs: Download | What's New in Solaris 11.2 | Information Library (documentation)

Highlights:

Zones related ..

Kernel Zones

Kernel Zones bring the ability to run a non-global/local zone at a different kernel version from the global zone and can be patched or updated independently without the need to reboot the global zone. In other words, kernel zones are independent and isolated environments with a full kernel and user environment.

Creating a Kernel Zone:

  1. If not available, install the kernel zone brand
    # pkg install brand/brand-solaris-kz
    
  2. Create and install a kernel zone using the existing zonecfg and zoneadm commands. The only difference compared to creating a non-kernel zone (the zones we have been creating for the past 10 years) is the template to be used -- by default, SYSdefault template is used. To create a kernel zone, use SYSsolaris-kz template instead.

    # zonecfg -z <zone-name> create –t SYSsolaris-kz
    # zoneadm –z <zone-name> install
    # .. continue with the rest of the steps to complete zone configuration ..
    

Kernel Zones can be used in combination with logical domains (Oracle VM for SPARC), but cannot be used in combination with other virtualization solutions such as Oracle VM VirtualBox that does not support nested virtualization.

Live Zone Re-configuration

This release (11.2) added support for the dynamic re-configuration of local zones. Now the following configuration changes do not require a zone reboot.

  • Resource controls and pools
  • Network configuration
  • Adding or removing file systems
  • Adding or removing virtual and physical devices

Read-Only Global Zones

Recent releases of Solaris have support for Immutable Non-Global Zones already. Solaris 11.2 extends the immutable zone support to Global Zones. Immutable zones will have a read-only zone root.

Make a Global Zone Read-Only/Immutable by:

# zonecfg -z global set file-mac-profile=fixed-configuration

Installing Packages across multiple Non-Global Zones from the Global Zone

  • -r option of pkg can be used to install/update/uninstall software packages into/in/from all non-global zones from the global zone.
  • Use -Z option along with -r to exclude a zone in applying the package operation. Similarly use -z along with -r to apply the intended package operation only in a specific zone

Multiple Boot Environments for Solaris 10 Zones

Multiple BE support has been extended to Solaris 10 Zones in this release. This feature is useful when performing operations such as patching within an Solaris 10 environment running on a Solaris 11 system.

CMT Aware Zones and Resource Pool Configuration

It is now possible to allocate CMT based resources -- vCPUs, Cores and Sockets, using the existing zonecfg and poolcfg commands. This is useful from performance and/or licensing point of view as it provides flexibility and control for managing licensing boundaries or dedicating hardware resources solely to a zone.

Cloud related ..

Centralized Cloud Management with OpenStack

Solaris 11.2 is the first release to incorporate a complete OpenStack distribution. OpenStack allows managing and sharing compute, network and storage resources in the data center through a centralized web portal. In other words, now administrators can set up an enterprise ready private cloud Infrastructure-as-a-Service (IaaS) environment with ease.

Check this quick How-To article out at Oracle Technology Network -- Getting Started with OpenStack on Oracle Solaris 11.2

Cloning and Disaster Recovery with Unified Archives

Unified Archives is a new native archive type that enables quick cloning for rapid application deployment in the cloud, fast and reliable disaster recovery. Both bare metal and virtual environments are supported. Check the archiveadm(1M) man page for details.

eg.,
Create a clone archive of a system
# archiveadm create ./clone.uar

Create bootable media
# archiveadm create-media ./archive.uar				/* USB image */
# archiveadm create-media -f iso <other options> ./bootarch.uar	/* ISO image */

Create a full system recovery archive
# archiveadm create --recovery ./recovery.uar

Extract information from a Unified Archive
# archiveadm info somearchive.uar

To be continued .. Stay tuned.

Monday Mar 31, 2014

[Solaris] ZFS Pool History, Writing to System Log, Persistent TCP/IP Tuning, ..

.. with plenty of examples and little comments aside.

[1] Check existing DNS client configuration

Solaris 11 and later:

% svccfg -s network/dns/client listprop config
config                      application        
config/value_authorization astring     solaris.smf.value.name-service.dns.client
config/options             astring     "ndots:2 timeout:3 retrans:3 retry:1"
config/search              astring     "sfbay.sun.com" "us.oracle.com" "oraclecorp.com" "oracle.com" "sun.com"
config/nameserver          net_address xxx.xx.xxx.xx xxx.xx.xxx.xx xxx.xx.xxx.xx

Solaris 10 and prior:

Check the contents of /etc/resolv.conf

% cat /etc/resolv.conf
search  sfbay.sun.com us.oracle.com oraclecorp.com oracle.com sun.com
options ndots:2 timeout:3 retrans:3 retry:1
nameserver      xxx.xx.xxx.xx
nameserver      xxx.xx.xxx.xx
nameserver      xxx.xx.xxx.xx

Note that /etc/resolv.conf file exists on Solaris 11.x releases too as of today.

[2] Logical domains: finding out the hostname of control domain

Use virtinfo(1M) command.

root@ppst58-cn1-app:~# virtinfo -a
Domain role: LDoms guest I/O service root
Domain name: n1d2
Domain UUID: 02ea1fbe-80f9-e0cf-ecd1-934cf9bbeffa
Control domain: ppst58-01
Chassis serial#: AK00083297

The above output shows that n1d2 domain is a guest domain, which is also an I/O domain, the service domain and a root I/O domain. Control domain is running on host ppst58-01.

Output from control domain:

root@ppst58-01:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    64    130304M  0.1%  0.1%  243d 2h 
n1d1             active     -n----  5001    448   916992M  0.2%  0.2%  3d 15h 26m
n1d2             active     -n--v-  5002    512   1T       0.0%  0.0%  3d 15h 29m

root@ppst58-01:~# virtinfo -a
Domain role: LDoms control I/O service root
Domain name: primary
Domain UUID: 19337210-285a-6ea4-df8f-9dc65714e3ea
Control domain: ppst58-01
Chassis serial#: AK00083297

[3] Administering NFS configuration

Solaris 11 and later:

Use sharectl(1M) command. Solaris 11.x releases include the sharectl administrative tool to configure and manage file-sharing protocols such as NFS, SMB, autofs.

eg.,
Display all property values of NFS:

# sharectl get nfs
servers=1024
lockd_listen_backlog=32
lockd_servers=1024
grace_period=90
server_versmin=2
server_versmax=4
client_versmin=2
client_versmax=4
server_delegation=on
nfsmapid_domain=
max_connections=-1
listen_backlog=32
..
..

# sharectl status
autofs  online client
nfs     disabled

eg.,
Modifying the nfs v4 grace period from the default 90s to 30s:

# sharectl get -p grace_period nfs
grace_period=90
# sharectl set -p grace_period=30 nfs
# sharectl get -p grace_period nfs
grace_period=30

Solaris 10 and prior:

Edit /etc/default/nfs file, and restart NFS related service(s).

[4] Examining ZFS Storage Pool command history

Solaris 10 8/07 and later releases log successful zfs and zpool commands that modify the underlying pool state. All those executed commands can be examined by running zpool history command. Because this command shows the actual zfs commands executed as they are, the 'history' feature is really useful in troubleshooting an error scenario that was resulted from executing some zfs command.

# zpool list
NAME       SIZE  ALLOC  FREE  CAP  DEDUP   HEALTH  ALTROOT
rpool      416G   152G  264G  36%  1.00x   ONLINE  -
zs3actact  848G  17.4G  831G   2%  1.00x   ONLINE  -

# zpool history -l zs3actact
History for 'zs3actact':
2014-03-19.22:02:32 zpool create -f zs3actact c0t600144F0AC6B9D2900005328B7570001d0 [user root on etc25-appadm05:global]
2014-03-19.22:03:12 zfs create zs3actact/iscsivol1 [user root on etc25-appadm05:global]
2014-03-19.22:03:33 zfs set recordsize=128k zs3actact/iscsivol1 [user root on etc25-appadm05:global]

Note that this log is enabled by default, and cannot be disabled.

[5] Modifying TCP/IP configuration parameters

Using ndd(1M) is the old way of tuning TCP/IP parameters, and still supported as of today (in Solaris 11.x releases). However using padm(1M) command is the recommended way to modify or retrieve TCP/IP Internet protocols on Solaris 11.x and later releases.

# ipadm show-prop -p max_buf tcp
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   max_buf               rw   1048576      --           1048576      128000-1073741824

# ipadm set-prop -p max_buf=2097152 tcp

# ipadm show-prop -p max_buf tcp
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   max_buf               rw   2097152      2097152      1048576      128000-1073741824

ndd style (still valid):

# ndd -get /dev/tcp tcp_max_buf
1048576

# ndd -set /dev/tcp tcp_max_buf 2097152

# ndd -get /dev/tcp tcp_max_buf
2097152

One of the advantages of using ipadm over ndd is that the configured/tuned non-default values are persistent across reboots. In case of ndd, we have to re-apply those values either manually or by creating a Run Control script (/etc/rc*.d/S*) to make sure that the intended values are set automatically during a reboot of the system.

[6] Writing to system log from a shell script

Use logger(1) command as shown in the following example.

eg.,

# logger -p local0.warning Big Brother is watching you

# dmesg | tail -1
Mar 30 18:42:14 etc27zadm01 root: [ID 702911 local0.warning] Big Brother is watching you

Check syslog.conf(4) man page for the list of available system facilities and the severity of the condition being logged (levels).

BONUS:

[*] Forceful NFS unmount on Linux

Try the lazy unmount option (-l) on systems running Linux kernel 2.4.11 or later to forcefully unmount a filesystem that keeps throwing Device or resource busy and/or device is busy error(s).

eg.,

# umount -f /bkp
umount2: Device or resource busy
umount: /bkp: device is busy
umount2: Device or resource busy
umount: /bkp: device is busy

# umount -l /bkp
#

Wednesday Mar 26, 2014

Software Availability : Solaris Studio 12.4 Beta & ORAchk

First off, these are two unrelated softwares.

Solaris Studio 12.4 Beta

Nearly two-and-a-half years after the release of Solaris Studio 12.3, Oracle is gearing up for the next major release 12.4. In addition to the compiler and library optimizations to support the latest and greatest SPARC & Intel x64 hardware such as SPARC T5, M5, M6, Fujitsu's M10, and Intel's Ivy Bridge and Haswell line of servers, support for C++ 2011 language standard is one of the highlights of this forthcoming release. The complete list of features and enhancements in release 12.4 are documented in the What's New page.

Those who feel compelled to give the updated/enhanced compilers and tools a try, can get started right away by downloading the beta bits from the following location. This software is available for Solaris 10 & 11 running on SPARC, x86 hardware; and Linux 5 & 6 runnin g on x86/x64 hardware. Anyone can download this software for free.

     Oracle Solaris Studio 12.4 Beta Download

Don't forget to check the Release Notes out for the installation instructions, known issues, limitations and workarounds, features that were removed in this release and so on.

Here's a pointer to the documentation (preview): Oracle Solaris Studio 12.4 Information Library

Finally, should you run into any issue(s) or if you have questions about anything related, feel free to use the Solaris Studio Community Forum.




ORAchk 2.2.4 (formerly known as EXAchk)

ORAchk, the Oracle Configuration Audit Tool, enhances EXAchk tool's functionality, and replaces the existing & popular RACcheck tool. In addition to the top issues reported by users/customers, ORAchk proactively scans for known problems within Oracle Database, Sun systems (especially engineered systems) and Oracle E-Business Suite Financials.

While checking, ORAchk covers a wide range of areas such as OS kernel settings, database installations (single instance and RAC), performance, backup and recovery, storage setting, and so on.

ORAchk generated reports (mostly high level) show the system health risks with the ability to drill down into specific problems and offers recommendations specific to the environment and product configuration. Those who do not like sending this data back to Oracle should be happy to know that there is no phone home feature in this release.

Note that ORAchk is available only for the Oracle Premier Support Customers - meaning only those customers with appropriate support contracts can use this tool. So, if you are a Oracle customer with the ability to access the Oracle Support website, check the following pages out for additional information.

     ORAchk - Oracle Configuration Audit Tool
     ORAchk user's guide

Feel free to use the community forum to ask any related questions.

About

Benchmark announcements, HOW-TOs, Tips and Troubleshooting

Search

Archives
« July 2016
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
      
Today