« About | Main | Linux »


Getting started with AIX technical support

I had hoped to start this blog with a nice overview article;  things are moving too fast to wait for that.  Today's urgent problem involves a crash on AIX, and I need this material publicly available right away.  Things are happening much too fast to build the perfect receipe, so consider this the starter set.   The other operating systems will be covered later.

obiee is usually deployed in a cluster, so all of the user scripts I write focus on gathering the information into a single shared directory, that each host machine can write to.   This isn't required, but it will help to organize the work.

export obiCollect=/net/host/obiCollect

Create a directory like this ( the example is for automounted filesystems ); place the export statement in your .profile or .bashrc login script.

No matter what directory you may be in, the scripts must be found, so a convenient place might be in  a ./bin subdirectory.

mkdir -p /net/host/obiCollect/bin

Make that directory available by adjusting the PATH setting in your profile scripts.

export PATH=$PATH:/net/host/obiCollect/bin

The script starter set
Browser Alert: Use Firefox, Chrome, Safari or IE10 to download files in links below (R-Click, Save-As ).  Previous Internet Explorer versions do not work for download ( and IE10 does not work to edit Roller Blogs ( edited with Chrome 3/15/2013)

obiaix.tar.gz now contains all of the files described below.  Unpack the archive into a directory in your $PATH.
   cd /net/host/obiCollect/bin
   gzip -dc obiaix.tar.gz | tar -xvf -

     Some utility functions needed by pstack.sh and obidoc.  getevars, c++filt come from IBM.

  • obidoc  ( formerly known as aixdoc.sh )
    Run this script every time you communicate with Tech Support about a crash.
    The script collects important information about your system, and I use the output to detect AIX and OBIEE configuration issues.   The information is dynamic, so running the script just once is not sufficient.  
    If you are talking to Tech Support about a crash, we need two things:
    1. A new bugid associated with your Service Request (SR) so we can communicate efficiently with the development organizations involved with the crash.
    2. The output from obidoc.   New crash?  You need a new obidoc collection.
  • obimon
    Is a script that documents memory and connection information for a specific process.
    Run this script at the request of Tech Support.  
    The script is "light weight" ... it will not burden your system.
    An obiee server crash often starts from a specific event ( query ), and the output from this script helps us to identify what started the stress on the server before it crashed.
    Use the script like this:
    $ obimon nqsserver
    $ obimon sawserver 
    $ obimon pid#
        ( Java process related to obiee )
    This version of the script is stopped using the "kill" command.
    Future versions will be easier to start/stop/restart the monitor collections.
  • pstack.sh
    If you have a core file, this script will document the important information to submit to Tech Support.
    pstack.sh core_name  
    Produces:  core_name.executable.pstack
    If you have a series of core files, you can capture this diagnostic output for all of them with a command like this:
    errpt -A -J CORE_DUMP | grep '^/' | xargs pstack.sh
  • snapcore.sh
    This is a shell cover script for the AIX snapcore utility.
    It is difficult for most people to follow all the rules to properly include the executable full path.
    Two simplifications:
     1. Invocation:   snapcore.sh core_file_name
     2. Produces:    core_file_name.executable.snapcore.pax.Z
    The script figures out the full path of the executable.
    The resulting pax file has the same name prefix as the original core file
    The timestamp on the pax file is the same as the original core.

  • lsof53.tar.gz , lsof61.tar.gz
    "List Open Files" is needed by obidoc.  These versions were compiled from source for the target 64-bit kernel ( 5.3 and 6.1 ).  The executable has been stripped to reduce size.   Recommended installation directory is in any directory that is commonly in $PATH.   For example, /usr/bin on AIX 6.1: 
    ## Download to /tmp
    $ sudo su
    # cd /usr/bin
    # gzip -dc /tmp/lsof61.tar.gz | tar -xvf - 

That should get us started.   I promise to improve this article and keep the scripts current.  They change frequently as I discover system differences among AIX customers.

2013-04-09, 2013-01-29 updates to obiaix.tar.gz for obimon, pstack.sh

« About | Main | Linux »

Post a Comment:
Comments are closed for this entry.

Dick Dunbar
is an escalation engineer working in the Customer Engineering & Advocacy Lab (CEAL team)
for Oracle Analytics and Performance Management.
I live and work in Santa Cruz, California.
I'll share the techniques I use to detect, avoid and repair problems.


« February 2015