Wednesday Jul 09, 2008

Web Server GDD Debugging Scripts

Sometimes, we can use the ready to go debugging scripts to collect more data in a running web server, see


 In particular, the pkg_app script  is  often used when we found a core dump file.

This pkg_app script will help collect the needed libraries and others, so we at sun can proceed further to examine the core dump file.

The other situation can be a web server hang, and we will collect gcore <web server pid> and pkg_app libraries to examine the running threads of the hung web server at that time.

You can download the latest pkg_app script at


 the above is for Linux and the below is for Solaris,

Then after the download and tar -xf,

apple:/export/home/bits/iws/misc/pkg_app-dir> tar -xf ../SUN-GDD_ALL_pkg_app_solaris.tar
apple:/export/home/bits/iws/misc/pkg_app-dir> ls -lrt
total 62
-rw-r--r--   1 root     other       3581 Sep 12  2007 README.txt
-rw-r--r--   1 root     other       3461 Sep 12  2007 LICENSE.txt
-rwxr-xr-x   1 root     other      19285 Oct  2  2007 pkg_app
-rwxr-xr-x   1 root     other       3329 Oct  2  2007 README.pkg_app

Then you can follow the steps at


  1. Copy the script to a temporary directory on the system where Web
    Server is installed.

  2. Login as superuser.

  3. Execute the pkg_app script in one of the following
    three ways:

    • ./pkg_app pid-of-running-application corefile

    • ./pkg_app pid-of-the-running-application(The pkg_app scripts
      prompts for the corefile name.)

    • ./pkg_app core file    "

 e.g. in my env,

apple:/export/home/iws7.0u3/> ptree 9736
9734  webservd-wdog -d /export/home/iws7.0u3/ -r
  9735  webservd -d /export/home/iws7.0u3/ -r /e
    9736  webservd -d /export/home/iws7.0u3/ -r

and  a run to show the options,

apple:/export/home/bits/iws/misc/pkg_app-dir> ./pkg_app
\* Sun Microsystems RSD pkg_app 2.7 Solaris                      [07/09/2008]
pkg_app 2.7, a Sun Microsystems data gathering utility.
  ./pkg_app [options] -c <core file | pid> -p <full path to process binary> -s [path to write tar file]

Required parameters:
 -c <core file OR pid of a running process>
 -p <full path to process binary> (ns-slapd, imapd, httpd etc.)

Optional parameters:
 -i (Include the core file with the final tar.gz)
 -q (Quiet)
 -d (Debug)
 -s <Storage; path to store the final tar file in>

usage:  ./pkg_app -c <name of the core file> -p <path to process binary>
usage:  ./pkg_app -c <pid of the running app> -p <path to process binary>

Examples: these are examples mixing various parameters

Directory Server
./pkg_app -i -c ./core.14740 -p /var/mps/ds52p4/bin/slapd/server/64

Messaging Server
./pkg_app -c ./core.3496 -p /opt/SUNWmsgsr/lib

Web Server
./pkg_app ./core.1092 -p /space/iws70/lib -s /var/crash

Calendar Server
./pkg_app -i -c ./core -p /opt/SUNWics5/cal/lib

--------------------end options above--------------------

then, I looked up where is my webservd binary in web 7.0u3 ,

apple:/export/home/iws7.0u3/> find /export/home/iws7.0u3 -name webservd

the 1st is 64-bit and 2nd is 32-bit,


apple:/export/home/iws7.0u3/lib/sparcv9> file webservd
webservd:       ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped
apple:/export/home/iws7.0u3/lib/sparcv9> cd ..
apple:/export/home/iws7.0u3/lib> file webservd
webservd:       ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped

so, we will point it to the 2nd one  above which is currently in use in my pid=9736.

You can also check out which webservd by

apple:/export/home/iws7.0u3/lib> pmap 9736  | grep webservd
9736:   webservd -d /export/home/iws7.0u3/ -r /
00010000      8K read/exec         /export/home/iws7.0u3/lib/webservd
00020000      8K read/write/exec   /export/home/iws7.0u3/lib/webservd

which shows the 32-bit binary in above (the 2nd one above).

Then, I ran the pkg_app commands, ./pkg_app -c 9736 -p /export/home/iws7.0u3/lib


apple:/export/home/bits/iws/misc/pkg_app-dir> ./pkg_app -c 9736 -p /export/home/iws7.0u3/lib
\* Sun Microsystems RSD pkg_app 2.7 Solaris                      [07/09/2008]
\* OS release                            [5.8]
\* Platform                              [SUNW,Sun-Blade-1000]
\* Using pid                             [9736]
\* Process Root                          [/export/home/iws7.0u3/]
\* Process binary                        [webservd]
\* webservd binary bit version           [32]
\* Checking path to binary name          [success, path != binary name]
\* Checking path is a directory          [success]
\* Locating webservd                     [success]
\* Checking located webservd is 32 bit   [success]
\* Binary located                        [/export/home/iws7.0u3/lib/webservd]
\* Adding binary to pkg_app.pldd         [success]
\* Grabbing pldd                         [success]
\* Grabbing pstack                       [success]
\* Grabbing pmap                         [success]
\* Grabbing pcred                        [success]
\* Grabbing pflags                       [success]
\* Databin Used                          [/export/home/bits/iws/misc/pkg_app-dir]
\* Using hostid for naming .tar.gz       [83450f78]
\* Writing file                          [pkg_app83450f78-apple-080709-154653.tar.Z]
\* Processing file 74 of 74
\* Done gathering files
\* Writing dbx files                     [success]
\* Creating final tarfile                [success]
\* Compressing tarfile                   [success]

Operations Complete

Upload this file to your Cores Directory at Sun
File located in directory .

                [ pkg_app83450f78-apple-080709-154653.tar.Z ]

                                Thank you.
                                Sun Software Technology Service Center (STSC)

1) You can check for updates to this script here:
        BigAdmin -

2) Release Notes and Guides located here:
        Docs -

3) GDD information located here:
        Docs -

4) Please send all Bugs and RFE's to the following address:
        Subject "pkg_app bug/rfe" -

5) Please send all other questions etc to:
        Subject "pkg_app feedback" -
------------------------------end Outputs ------------------------------

apple:/export/home/bits/iws/misc/pkg_app-dir> ls -lrt
total 48702
-rw-r--r--   1 root     other       3581 Sep 12  2007 README.txt
-rw-r--r--   1 root     other       3461 Sep 12  2007 LICENSE.txt
-rwxr-xr-x   1 root     other      19285 Oct  2  2007 pkg_app
-rwxr-xr-x   1 root     other       3329 Oct  2  2007 README.pkg_app
-rw-rw-rw-   1 root     other    24871809 Jul  9 15:47 pkg_app83450f78-apple-080709-154653.tar.Z 

then send above .Z file to Sun and then we can check out more about the web server running process or core dump file.

The inside of the .Z file is:

apple:/export/home/bits/iws/misc/pkg_app-dir/pkg-app-dump.tmp> ls -lrt
total 110
-rw-rw-rw-   1 root     other          0 Jul  9 15:46
-rw-rw-rw-   1 root     other          0 Jul  9 15:46 pkg_app.errors
-rw-rw-rw-   1 root     other       2508 Jul  9 15:46 pkg_app.pldd
-rw-rw-rw-   1 root     other      24729 Jul  9 15:46 pkg_app.pstack
-rw-rw-rw-   1 root     other      16639 Jul  9 15:46 pkg_app.pmap
-rw-rw-rw-   1 root     other         46 Jul  9 15:47 pkg_app.pcred
-rw-rw-rw-   1 root     other       2560 Jul  9 15:47 pkg_app.pflags
-rwxr-xr-x   1 root     other        443 Jul  9 15:47 opencore
-rw-rw-rw-   1 root     other        614 Jul  9 15:47 dbxrc
-rw-rw-rw-   1 root     other       1137 Jul  9 15:47 runtime-080709-154653.log
drwxrwxrwx   2 root     other       2048 Jul  9 15:50 app

and remember to send in any core dump files or gcore files to Sun support with above .Z file.

Then we can check out more about the running conditions and states inside the webservd at that time.


to share tips and experience in Oracle products, e.g. SunONE / iPlanet Web server, Weblogic server, Web protocols, security and performance.


« April 2016