Wednesday Dec 10, 2008

Using the new wshang.ksh version 3.1

I just downloaded the new  wshang.ksh version 3.1 and tried it. Go to  the download location.

(Note: above url is not valid any more. Please download it from My Oracle Support (MOS) site from note: Doc ID 1325651.1 - Download Location For Wshang Script - Gather Debug Data Script For Troubleshooting Oracle iPlanet Web Server or Oracle iPlanet Web Proxy Server Hangs.)

It is very easy to use.

After I downloaded and gunzipped it, I just ran it,

 APPLE:/export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3> ./wshang.ksh
./wshang.ksh[65]: date+%y%m%d%HH%MM%SS:  not found

 This script collects 3 snapshots of the following information
 at a 15 seconds interval against a web server process:
        prstat -L -a (will try 'top' on pre-Solaris 8 systems)
        pmap -x
        netstat -an
 And collections 30 seconds of:
        truss -feald -vall -wall -rall

Data will be collected to the location /tmp/wshang by default.
Data location should have enough disk space for large core files.
Would you like to specify a different directory for data?

Specify a directory:

The following is a list of web server processes:

Solaris 8 or greater
Web Server 7.0/6.1 processes
    364   webservd -r /export/home/iws6.1sp2 -d /export/home/iws6.1sp2/https-ap
    11478 webservd -r /export/home0/case/11330103-admin-gu/61sp10 -d /export/ho
    11640 webservd -d /export/home0/case/11330103-admin-gu/70u3/https-APPLE.low
    12091 webservd -d /export/home/iws7.0u3/admin-server/config -r /export/home
    12742 webservd -r /export/home0/case/11315437-web-session/FVB/run -d /expor
    12814 webservd -r /export/home0/case/11315437-web-session/FVB/run -d /expor
    12860 webservd -r /export/home/iws6.1sp10 -d /export/home/iws6.1sp10/https-
    12919 webservd -r /export/home/iws6.1sp10 -d /export/home/iws6.1sp10/https-
    13879 webservd -d /export/home/iws7.0u3/https-java-mon/config -r /export/ho
    13573 webservd -d /export/home/iws7.0u3/ -r
    15402 webservd -r /export/home/iws6.1sp10 -d /export/home/iws6.1sp10/https-
    15497 webservd -d /export/home0/case/11330103-admin-gu/70u3-2/admin-server/
Web Server 6.0 processes
Proxy Server 4.0 processes
    14065 proxyd -r /export/home/ips4.0sp8 -d /export/home/ips4.0sp8/proxy-acl/
    15527 proxyd -r /export/home/ips4.0sp7 -d /export/home/ips4.0sp7/proxy-adms
    15867 proxyd -r /export/home/ips4.0sp7 -d /export/home/ips4.0sp7/proxy-rpp/

Enter the PID of the web server process to collect data on
(Note that the correct PID for child webservd/ns-httpd process will not be listed correctly here if you have MaxProcs > 1 in your magnus.conf):

(this is already the needed worker process pid from my ptree outputs,

APPLE:/export/home0> ptree 12919
12893 ./webservd-wdog -r /export/home/iws6.1sp10 -d /export/home/iws6.1sp10/htt
  12918 webservd -r /export/home/iws6.1sp10 -d /export/home/iws6.1sp10/https-se
    12919 webservd -r /export/home/iws6.1sp10 -d /export/home/iws6.1sp10/https-


 12919 ?        1:41 webservd

Gathering data now...

Loop 1 for 15 seconds...

collecting pflags...
collecting pmap -x...
collecting pldd...
collecting prstat -L -a...
collecting pstack...
collecting pfiles...
collecting netstat...

Loop 2 for 15 seconds...

collecting pflags...
collecting pmap -x...
collecting pldd...
collecting prstat -L -a...
collecting pstack...
collecting pfiles...
collecting netstat...

Loop 3 for 15 seconds...

collecting pflags...
collecting pmap -x...
collecting pldd...
collecting prstat -L -a...
collecting pstack...
collecting pfiles...
collecting netstat...

collecting 30 seconds of truss...

Collecting gcore...
gcore: /export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/gcore.out.12919 dumped
\* ----------------------------------------------------------------------------------
\* Sun Microsystems RSD pkgapp 3.0 Solaris                               [12/10/2008]
\* ----------------------------------------------------------------------------------
\* OS release                            [5.8]
\* Platform                              [SUNW,Sun-Blade-1000]
\* Checking [-c] is a core or pid        [using pid 12919]
\* Process Root                          [/export/home/iws6.1sp10/https-sess/config]
\* Databin parameter [-s] checks         [reset to /export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3]
\* Creating Databin directories          [/export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/pkgapp_data]
\* Databin used/created is               [/export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/pkgapp_data/pkgapp-121008-01]
\* Creating temp area                    [/tmp/pkgapp.16081/]
\* Pid used, no corefile to check
\* Process binary                        [webservd]
\* Checking usage history                [not recently run]
\* webservd binary bit version           [32]
\* Checking path [-p] to binary name     [failed, path includes binary name]
\* Resetting path [-p] parameter         [/export/home/iws6.1sp10/bin/https/bin]
\* Checking path [-p] is a directory     [success]
\* Locating webservd                     [success]
\* Checking located webservd is 32 bit   [success]
\* Binary located                        [/export/home/iws6.1sp10/bin/https/bin/webservd]
\* Adding binary to pkgapp.pldd          [success]
\* Grabbing pldd                         [success]
\* Grabbing pstack                       [success]
\* Grabbing pmap                         [success]
\* Grabbing pcred                        [success]
\* Grabbing pflags                       [success]
\* Grabbing pargs                        ./pkgapp[103]: /usr/bin/pargs:  not found
./pkgapp[110]: /usr/bin/pargs:  not found
\* Not Including the core/gcore
\* Javatools [-j] not set                [skipped]
\* Grabbing /var/adm/messages            [success]
\* Grabbing uname -a                     [success]
\* Grabbing date/time                    [success]
\* Grabbing showrev -p                   [success]
\* Grabbing pkginfo -l                   [success]
\* Grabbing /etc/release                 [success]
\* Grabbing coreadm                      [success]
\* Grabbing ulimit                       [success]
\* Grabbing libs                         [success]
\* Making lib paths app/                 [success]
\* Making lib paths libs/                [success]
\* Linking libraries                     [success]
\* Libraries linked                      [101 ttl]
\* Using hostid for naming .tar.gz       [83450f78]
\* Writing file                          [pkgapp-83450f78-APPLE-081210-145852.tar.gz]
\* Done gathering files
\* Writing dbxrc & files     [success]
\* Writing manifest-081210-145852.log    [success]
\* Writing pkgapp-args-081210-145852     [success]
\* Creating final tarfile                [success]
\* Compressing tarfile                   [success]
\* End of runtime logging
\* Saving history info                   [/var/tmp/pkgapp-history/history-081210-145852.log]
\* Saving runtime log                    [/var/tmp/pkgapp-history/runtime-081210-145852.log]
\* Removing [-r] temp area/files         [left alone]
\* Operations Complete
Upload the following file(s) to your Cores Directory at Sun

1) File(s) located in directory /export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/pkgapp_data/pkgapp-121008-01

                [ pkgapp-83450f78-APPLE-081210-145852.tar.gz ]

                                Thank you.
                                Sun Software Technology Service Center (STSC)

1) You can check for updates to this script here:
        BigAdmin -
2) Release Notes and Guides located here:
        Docs -
3) GDD information located here:
        Docs -

4) Please send all Bugs and RFE's to the following address:
        Subject "pkgapp bug/rfe" -

5) Please send all other questions etc to:
        Subject "pkgapp feedback" -

Creating a tar file of the results...
a gcore.out.12919 70628K
a gcore_pldd.out 3K
a gcore_pmap.out 14K
a gcore_pstack.out 71K
a netstat.out.1 30K
a netstat.out.2 30K
a netstat.out.3 30K
a pfiles.out.1 4K
a pfiles.out.2 4K
a pfiles.out.3 4K
a pflags.out.1 8K
a pflags.out.2 8K
a pflags.out.3 8K
a pkgapp_data/ 0K
a pkgapp_data/pkgapp-121008-01/ 0K
a pkgapp_data/pkgapp-121008-01/pkgapp-83450f78-APPLE-081210-145852.tar.gz 14849K
a pldd.out.1 3K
a pldd.out.2 3K
a pldd.out.3 3K
a pmap.out.1 20K
a pmap.out.2 20K
a pmap.out.3 20K
a prstat.out.1 2K
a prstat.out.2 2K
a prstat.out.3 2K
a pstack.out.1 71K
a pstack.out.2 71K
a pstack.out.3 71K
a release.out 1K
a showrev.out 70K
a truss.out 37K
a ulimit 1K
a uname.out 1K

The tar file has been created....

Please send the file /export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/11111111-gdd-wshang-.tar.gz to
Sun Microsystems at
Choose cores as the destination.

Then, I checked my output dir,

 APPLE:/export/home0> ls -l /export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/
total 193098
-rw-rw-rw-   1 root     other    25833674 Dec 10 15:00 11111111-gdd-wshang-.tar.gz
-rw-rw-rw-   1 root     other    72322796 Dec 10 14:58 gcore.out.12919
-rw-rw-rw-   1 root     other       2684 Dec 10 14:58 gcore_pldd.out
-rw-rw-rw-   1 root     other      14161 Dec 10 14:58 gcore_pmap.out
-rw-rw-rw-   1 root     other      72266 Dec 10 14:58 gcore_pstack.out
-rw-rw-rw-   1 root     other      30577 Dec 10 14:57 netstat.out.1
-rw-rw-rw-   1 root     other      30577 Dec 10 14:58 netstat.out.2
-rw-rw-rw-   1 root     other      30421 Dec 10 14:58 netstat.out.3
-rw-rw-rw-   1 root     other       3572 Dec 10 14:57 pfiles.out.1
-rw-rw-rw-   1 root     other       3572 Dec 10 14:58 pfiles.out.2
-rw-rw-rw-   1 root     other       3572 Dec 10 14:58 pfiles.out.3
-rw-rw-rw-   1 root     other       8143 Dec 10 14:57 pflags.out.1
-rw-rw-rw-   1 root     other       8143 Dec 10 14:58 pflags.out.2
-rw-rw-rw-   1 root     other       8143 Dec 10 14:58 pflags.out.3
drwxrwxrwx   3 root     other        512 Dec 10 14:58 pkgapp_data
-rw-rw-rw-   1 root     other       2651 Dec 10 14:57 pldd.out.1
-rw-rw-rw-   1 root     other       2651 Dec 10 14:58 pldd.out.2
-rw-rw-rw-   1 root     other       2651 Dec 10 14:58 pldd.out.3
-rw-rw-rw-   1 root     other      19535 Dec 10 14:57 pmap.out.1
-rw-rw-rw-   1 root     other      19535 Dec 10 14:58 pmap.out.2
-rw-rw-rw-   1 root     other      19535 Dec 10 14:58 pmap.out.3
-rw-rw-rw-   1 root     other       1388 Dec 10 14:57 prstat.out.1
-rw-rw-rw-   1 root     other       1390 Dec 10 14:58 prstat.out.2
-rw-rw-rw-   1 root     other       1392 Dec 10 14:58 prstat.out.3
-rw-rw-rw-   1 root     other      72231 Dec 10 14:57 pstack.out.1
-rw-rw-rw-   1 root     other      72235 Dec 10 14:58 pstack.out.2
-rw-rw-rw-   1 root     other      72233 Dec 10 14:58 pstack.out.3
-rw-rw-rw-   1 root     other        184 Dec 10 15:00 release.out
-rw-rw-rw-   1 root     other      71631 Dec 10 15:00 showrev.out
-rw-rw-rw-   1 root     other      37856 Dec 10 14:58 truss.out
-rw-rw-rw-   1 root     other        206 Dec 10 15:00 ulimit
-rw-rw-rw-   1 root     other         66 Dec 10 15:00 uname.out

Then it is all done and just need to do this ,

"Please send the file /export/home/bits/iws/misc/SUN-GDD-webserver-wshang-solaris-v3/data/11111111-gdd-wshang-.tar.gz to
Sun Microsystems at
Choose cores as the destination.

NOTE : Make sure you rename the above xxx.tar.gz  so that your case SR# is part of the filename,

e.g. 71234567-customer_name-wshang-.tar.gz.

This will ensure the uploaded file will be moved to the right location for your case SR#.

Sun support engineers can then find the uploaded file easily.

Tuesday Dec 02, 2008

New wshang.ksh version 3.0 just released


We got a new troubleshooting tool,  wshang.ksh version 3.0 just released. FYI.

You can download it at

There are some user configurable options inside, e.g.

Some notes on usage:  The customer will have to edit the script to
change a few options as needed.  These options will be made interactive
in the next release of wshang.

# The following four settings can be changed as necessary:

# Change this value to set the duration in secs between data collection


# Set the following variable to y or n to allow a gcore to be taken


# Set the following variable to y or n to allow the pkgapp script to run.

# Assumes the pkgapp script is in the same location as wshang script.


# Set the following variable with your Case Number. If no case number
has been

# opened but wish to use wshang.ksh to trap an issue, you can use your

# Name (no spaces) or all 1's.  Once you get a Sun Case number, you can

# the resulting tar file before sending to Sun.


This will be a useful tool to collect troubleshooting data for web and/or proxy server. 


Wing-Yip Walter Lee


« April 2014