« Linux | Main | Minamalist »

Solaris

Getting started with Solaris technical support


Browser Alert:   Use Firefox, Chrome or Safari to download files in links below (R-Click, Save-As ). Internet Explorer does not work.

The following scripts are located in obisolaris.tar.gz (2013-07-12)

  • obimon (0.1) - monitor memory and connection growth.
  • pcore.sh (0.1) - miniture version of pstack.sh for Solaris, until we work out access to dbx. Note 1:
  • packcore.sh (0.1) - bundle core file and libraries for Oracle Tech Support
  • pstack.sh (0.4)  - uses dbx to obtain call stacks from a core file.
  • unpackDbx.sh.sh (0.2) - extracts dbx from Studio 12.3 download.  See instructions below.

 Note 1: pcore.sh uses Solaris commands to capture information from a core file.

  • pargs  captures the environnment variables
  • pldd        captures libraries in use
  • pstack   captures pstack for all threads

So what's wrong with pcore.sh output?  It's the Solaris pstack command that limits our understanding:

  1. Doesn't identify the crashing thread
  2. Doesn't display the registers at the time of the crash
  3. Doesn't display disassembly of instructions to understand the nature of the crash
  4. Doesn't demangle C++ object names

Note 2: Submit the pcore output. 

If we need more detailed information,  we'll request packcore.sh output so we can use dbx at Oracle.

Obtaining Solaris dbx

Oracle Solaris Studio 12.3 download site:

  1. Accept the Studio OTN Developer Agreement
  2. Download the tar file appropriate to your environment:
    SPARC (488MB) or x86 (390MB)
    Solaris 10 or Solaris 11
    Linux x86 (368MB)
  3. Package Installation requires registration, root install, and a Support Offering.
    Tarfile Installation contains these notes:
    "This option extracts the installed version of Oracle Solaris Studio software from
    a tarfile (no packages). Elevated OS permissions (e.g. root) are not needed."
    "Oracle support is not available for this configuration nor can the software be updated with patches."
  4. Do not unpack the Studio bz2 file.  Use unpackDbx.sh
    It is 1.6GB unpacked.  This script will extract the dbx tools, which are just 22MB
    $ time unpackDbx.sh ran less than 5 minutes on my machine ( real 4m43.050s ).

Example Installation:

cd /opt
unpackDbx.sh
... tar output ...
------------------------ Done
Add studio12.3/bin to PATH.  Add studio12.3/man to MANPATH
Studio commands available:
   c++filt dbx dem dwarfdump getmsg version whatdir

PATH=$PATH:/opt/studio12.3/bin
MANPATH=$PATH:/opt/studio12.3/man
dbx -V
whatdir dbx
  /opt/studio12.3/bin

unpackDbx.sh script uses md5 to check the download file (SolarisStudio12.3-solaris-sparc-bin.tar.bz2)

With dbx installed, we can obtain much more informative pstack output.
Make sure that pstack.sh and dbx are available by setting $PATH in your .profile at login.

Download pstack.sh, and place it in /opt/studio12.3/bin directory, so it is available from $PATH.

If $obiCollect variable is set to a common directory, pstack output is directed there.  Place the scripts from obisolaris.tar.gz into $obiCollect/bin.

$ obiCollect=/net/host-name/obiCollect ; export obiCollect
$ mkdir $obiCollect/bin
$ pstack.sh core_name
$ ls $obiCollect
  bin core_name.nqsserver.pstack  

Core file names

Solaris coreadm is used to change the characteristics of core files from process crashes.
The default name "core" creates a problem when communicating with Tech Support about specific crashes, because we need to relate a specific crash with the obiee server logs to analyze the cause of the server crash.    As a starting point,  this root command will provide unique core file names, and create a record of the crashes in Solaris log.

  coreadm -i core.%f.%p -e log -e global

For more details:  man coreadm

It's possible to direct all core files to a specific directory using the -g option. (e.g., -g /var/core/core.%f.%p )

For rapid restart of obiee processes, the directory should be on a fast, local filesystem,  and the obiCollect directory should be on a network shared directory available to all nodes in the obiee cluster.

Test drive: coreadm and pstack.sh

  1. Studio 12.3 downloaded and processed with unpackDbx.sh into /opt
    Assume bash shell below.  pstack.sh in /opt/studio12.3/bin
  2. root:  coreadm -i core.%f.%p -e log -e global    
  3. export    PATH=/opt/studio12.3/bin:$MANPATH
    export MANPATH=/opt/studio12.3/man:$MANPATH
  4. sleep 300 &
    [1] 1908
    kill -6 1908
    [1]+  Abort (core dumped) sleep 300 
  5. dmesg | grep sleep
    May 29 15:37:53 ... core_log: sleep[1908] no global core file pattern exists
  6. pstack.sh core.sleep.1908
     ls -l core.sleep* | cut -c 32-80
      1994184 May 29 07:10 core.sleep.29780
         2730 May 29 07:10 core.sleep.29780.pstack

2012-05-30:  modifications to unpackDbx.sh and pstack.sh
2013-02-11: scripts repackaged into obisolaris.tar.gz
Setup instructions have changed, consolidating the scripts to $obiCollect/bin

« Linux | Main | Minamalist »
Comments:

Post a Comment:
Comments are closed for this entry.
About

Dick Dunbar
is an escalation engineer working in the Customer Engineering & Advocacy Lab (CEAL team)
for Oracle Analytics and Performance Management.
I live and work in Santa Cruz, California.
I'll share the techniques I use to detect, avoid and repair problems.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today