Friday Jun 07, 2013

DT (DirTracer) 7.2.1 and Pkgapp 3.6.2? Oh yeah!

DT 7.2.1 and PA 3.6.1 are now on deck!

Photo © Lee Trujillo

With the switch to Oracle and the hosting for DirTracer and Pkgapp being maintained via a Knowledge Document within Oracle's Support Portal, I've let the blog slide into the abyss.  No more!

I'm not saying I will be updating the blog on a regular basis as there just is no need.

You can view all the changes and download the latest version by logging into the Customer Support Portal and searching for Knowledge Doc # 1291443.1 (DT) and 1274584.1 (PA).

  • Using DirTracer to Gather Debug Data for DSEE or ODSEE Directory Servers. [ID 1291443.1]
  • How to Use Pkgapp to Gather Libraries to Debug Core/Gcore/Crash Files in Solaris and Linux [ID 1274584.1]

Enjoy all!

Lee

Wednesday May 25, 2011

DT (DirTracer) 7.1 has been unleashed!

DT 7.1 is finally here!

Photo © Lee TrujilloIt has been over a year since I released 7.0 and I am pleased to announce 7.1 is finally here. It was a long time coming, but I'm happy with the results.  DT 7.1 has fixes, re-branding and new enhancements including the new Pkgapp 3.5.1 which will help solve Directory Server issues.  There were quite a few items I was hoping to add but as development goes, it's easy to keep adding and adding features. I had to place a line in the sand and release it.

DT 7.1 for Solaris and Linux are now both available from Oracle Knowledge Document 1291443.1

Changes in DT 7.1

DirTracer:

  • Rebranded to Oracle from Sun Microsystems.
  • Included the new Pkgapp 3.5.1.
  • Fixed checkBitVersion to determine correct binary bitversion.
  • Fixed checkDSType for carpet issues.
  • Fixed locateLdapTools to locate tools properly.
  • Created locateCarpet to properly find the carpet command.
  • Gather release file from /etc for all Linux'.
  • Fixed how ns-slapd file is located for 64 bit versions (linux).
  • Created an appCheck function to locate system commands.
  • Fixed DT from trying to grab a gcore when gdb is not present. (linux)
  • Minor bug fixes.
  • Removed the included ldapsearch and ldapmodify commands and their respective libaries from the tar as they are no longer needed.

Configurator:

  • Rebranded to Oracle from Sun Microsystems.
  • Fixed checkBitVersion to match DT7.
  • Changed old LOG_IF_HIGH_CPU attrs to HIGH_CPU etc.
  • Changed how I deal with the Instance Selector. Cleaner interface.
  • Removed the need to select gathering logs when option 7 ot 8 are selected.
  • Fixed bug 6965060 for customers that have massively long instance path names. - 6965060 DirTracer: 7.0 Configurator cannot locate dse.ldif if the full inst path is > 80 chars: a ps limit.

 Compressor:

  • Compressor is a small script used to tar/gzip captured files after the fact if the SKIP_TAR_GZIP option was used. This is version 1.0 and is included in this bundle.  See README.FIRST for more information.

Enjoy all!

Lee

Thursday Apr 14, 2011

Pkgapp 3.5 Released!

Pkgapp 3.5 Released!

Photo © Lee Trujillo

I've fixed a few bugs, added some great enhancements and changed some of the output during run-time.  Please note, all downloads for Pkgapp (and DirTracer) will only be available via the My Oracle Support portal.  I hope to have all references to Pkgapp and DirTracer taken off the old BigAdmin and replaced with a reference to the support portal. 

[Read More]

Thursday Mar 10, 2011

DT7.0 and Pkgapp 3.4 Available via the My Oracle Support portal

Due to popular demand...

Photo © Lee TrujilloI've added the DirTracer and Pkgapp tar files to a Knowledge Portal document so customers now can access and download from a more reliable location.  

Please login to the My Oracle Support portal and search for the following Doc ID.

  • DirTracer7 (DT7) - Oracle Support Doc ID 1291443.1
  • Pkgapp 3.4 - Oracle Support Doc ID 1274584.1

This document is a truncated copy of the included .pdf file included with the Pkgapp bundle.  See the "Download" links near the bottom.

Cheers!

Lee

Thursday Aug 05, 2010

The Windows Question

I've recently received a request for DirTracer and Pkgapp on Windows. Photo © Lee Trujillo

I wrote these applications specifically for Unix based operating systems and will continue to release them on the Unix platforms supported by the product.  Windows has always been an exception to this.

While yes, Sun / Oracle releases versions of its Directory Server (DS) software on these supported platforms, DirTracer and Pkgapp are "not" part of the Software package and as a result, not supported by the normal Sun / Oracle support model.  Support for these is left up to one person; Me, their creator.

This said, Windows does not have the same types of facilities for troubleshooting the DS process as most Unix's do.

Examples; the following utilities can be invoked on a per process basis on a Unix platform:

  • pstack
  • prstat
  • pargs
  • pmap
  • pldd
  • gcore
  • iostats
  • vmstats
  • mpstats

Analysis of a DS process goes well beyond copying a set of config files and access logs.  Windows platforms do not provide equivalent utilities and system based commands.

I also created the DirTracer & Pkgapp code to be fairly portable by using scripting. At one point I toyed with PERL but I cannot guarantee PERL would be installed on all systems so I went for the least common denominator available; unix shell. Yes yes, I know there is Windows PowerShell but it was only released in circa 2008 and two years later...is still only at version 1.0 according to their own FAQ.  In fact, on Microsoft's own Shell Repository I only see six results for Process Monitoring scripts using PowerShell; the majority of users still preferring VBscript. Regardless, even if I could port DirTracer and Pkgapp over to PowerShell I would face the same issues as if I had created these using PERL; the availability of PowerShell installed on the Windows server.

I also thought of using Cygwin's Terminal which is essentially a unix port to run inside an emulator. This would be a perfect way to port my applications, but I face the same hurdles as I do with PERL and PowerShell - Availability.  Lets face it, Windows Administrators do not take it kindly when anyone asks them to install "anything" on their systems in order to troubleshoot problems.

The Stats:

Lets look at the facts. Being in the Support Organization of Oracle I am not informed on the number of customers downloading DSEE for Windows.  I do however have access to the numbers of Service Requests for this Operating System.

  • In the Calendar Q1 of this year less than 2% of the Directory Server cases were identified as for Windows versions of the product.

While I do not know if more of the cases received were actually using Windows, I can honestly say at my level (tier 3) I rarely receive an escalation that is Windows related.  Most DSEE related issues we get stem from Cacao, Windows Services and ISW to ADS synchronization problems. These type of problems are never addressed by DirTracer on any platform.  With such low number of issues coming in for DS related issues, the priority is to keep working on the Unix platform versions of DirTracer and Pkgapp.

Cheers!

Lee T.

Thursday Apr 22, 2010

Pkgapp 3.4 Released

Yes, I've released it!

Photo © Lee TrujilloI've updated the Licensing agreements etc., packaged the files and released internally (only).  If you need it, you can call Sun Oracle Support and have them use the following Internal Only url (do not click this link, it is not available outside of the Sun Oracle network).  I have submitted two new packages (tars) to BigAdmin and we hope they will make it on the page sometime next week.  Keep checking the BigAdmin link.

Pkgapp 3.4 - http://surl/pkgapp

Changes in Pkgapp 3.4

Improvements

  1. On all platforms Pkgapp now utilizes the /proc filesystem to avoid locking issues (as seen on  Solaris) of the process instead of using pldd against the live process.  See “Ptools / Pstack Issues” of the Server Impact section.
  2. Removed old 3.2 code and reworked how the -a parameter checks are displayed (see the pdf guide for more detail).
  3. When -i (include core) is not used, renames the core/gcore file to - pkgapp-<hostid>-<corefile name>.  This will help Sun Oracle employees easily locate the core file based on the name.
  4. Enhanced the error checking with regard to usage options.
  5. Set umask on the tmp bin and databin locations to enhance security.

Fixes

  1. Fixed bug 6918386 pkgapp 3.3 fails when processing argument for parameter -a
  2. Fixed bug 6919793 Pkgapp 3.3: awk fails on excessive string length from a localized binary.
  3. Fixed bug 6920593 Pkgapp 3.3: missed capturing sparcv9 \*_db.so\* on Solaris
  4. Fixed an issue where Pkgapp would ignore the core (when provided via -c) when the -j parameter was given.

Example:

Solaris
root[/opt/pkgapp]#./pkgapp -itr -p 18101 -c /var/tmp/cores/core.18101 -a /opt/dsee7/lib/64/ns-slapd -s /var/tmp/storage
\* ------------------------------------------------------------------------------
\* Oracle Corporation CSD pkgapp 3.4 Solaris                         [04/22/2010]
\* ------------------------------------------------------------------------------
\* Checking install epoch/delta          [good, 772538 seconds]
\* Checking pkgapp path                  [using /opt/304-solaris]
\* Checking for locale en_US.UTF-8       [failed]
\* Checking for locale en_US.ISO8859-1   [failed]
\* Checking for locale en_US             [failed]
\* Checking for locale C                 [success]
\* Checking usage history                [not recently run]
\* OS release [Solaris]                  [5.10]
\* Platform/Type                         [SUNW,Sun-Blade-T6300 sparcv9]
\* Checking [-p] pid.                    [using pid 18101]
\* Checking [-a] path                    [using /opt/dsee7/lib/64/ns-slapd]
\* Process root                          [/opt/dsee7/lib/64/ns-slapd]
\* Checking [-c] core                    [using core /var/tmp/cores/core.18101]
\* Databin parameter [-s] checks         [reset to /var/tmp/storage]
\* Creating Databin directories          [/var/tmp/storage]
\* Databin used/created is               [/var/tmp/storage/pkgapp-042210-01]
\* Creating temp area                    [/tmp/pkgapp.15891/]
\* Checking corefile for a valid pldd    [pldd is good with 80 elements]
\* Checking if corefile is truncated     [CORE core.18101 is GOOD. Core contains 320 segments ]
\* Continuing...                         
\* Process binary                        [ns-slapd]
\* ns-slapd binary bit version           [64]
\* core.18101 bit version                [64]
\* Checking path [-a] to binary name     [success, path includes binary name]
\* Checking path [-a] exists             [success]
\* Locating ns-slapd                     [success]
\* Checking located ns-slapd is 64 bits..[success]
\* Binary located                        [/opt/dsee7/lib/64/ns-slapd]
\* Adding binary to pkgapp.pldd          [success]
\* Grabbing libs from /proc              [success]
\* Grabbing pstack from core             [success]
\* Grabbing pmap from core               [success]
\* Grabbing pcred from core              [success]
\* Grabbing pflags from core             [success]
\* Grabbing pargs from core              [success]
\* Grabbing [-i] core/gcore              [success]
\* Grabbing core/gcore cksum             [success]
\* Javatools [-j] not set                [skipped]
\* Grabbing /var/adm/messages            [success]
\* Grabbing uname -a                     [success]
\* Grabbing date/time                    [success]
\* Grabbing showrev -p                   [success]
\* Grabbing pkginfo -l                   [success]
\* Grabbing /etc/release                 [success]
\* Grabbing coreadm                      [success]
\* Grabbing ulimit -a                    [success]
\* Gathering 64 bit libraries            
\* Grabbing libs                         [success]
\* Making lib paths app/                 [success]
\* Making lib paths libs/                [success]
\* Linking libraries                     [success]
\* Libraries linked                      [109 ttl]
\*                                       
\* Using hostid for naming .tar.gz       [84aaaac8]
\* Writing file                          [pkgapp-84aaaac8-s-6000a-t6300i-zone8-sca11-100422-092800.tar.gz]
\*                                       
\* Done gathering files                  
\* Writing dbxrc & opencore.sh files     [success]
\* Writing coreinfo.sh file              [success]
\* Writing validate-libs.sh file         [success]
\* Writing manifest-100422-092800.log    [success]
\* Writing pkgapp-args-100422-092800     [success]
\* Creating final tarfile                [success]
\* Compressing tarfile                   [success]
\* End of runtime logging                
\* Saving history info                   [success]
\* Saving runtime log                    [success]
\* Saving Library list                   [success]
\* Saving Library list README            [success]
\* Removing [-r] temp area/files         [removed]
\*                                       
\* Operations Complete                   
\* Total runtime                         [1 minutes]
\* ------------------------------------------------------------------------------
\* Upload the following file(s) to http://supportfiles.sun.com Cores Directory at
Oracle
                                        
1) File(s) located in directory /var/tmp/storage/pkgapp-042210-01
                                        
                [ pkgapp-84aaaac8-s-6000a-t6300i-zone8-sca11-100422-092800.tar.gz ]
                                        


                                Thank you.
                                Customer Service Delivery
                                Global Systems Support


NOTES:
1) You can check for updates to this script here:
        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
        Blog - http://blogs.sun.com/Dirtracer

2) GDD information located here:
        Rel Notes - http://docs.sun.com/app/docs/doc/820-0437
        Docs - http://www.sun.com/service/gdd/index.xml
        Video - http://mediacast.sun.com/search?query=gdd
        Video - http://mediacast.sun.com/search?query=dirtracer

3) Please send all Bugs and RFE\\'s to the following address:
        Subject "pkgapp bug/rfe" - pkgapp-feedback_ww@oracle.com

4) See the following reference files.
        /var/tmp/pkgapp-history/runtime-100422-092800.log
        /var/tmp/pkgapp-history/history-100422-092800.log
        /var/tmp/pkgapp-history/library.list-100422-092800
        /var/tmp/pkgapp-history/library.list-100422-092800.README 
Linux
root [/opt/pkgapp]# ./pkgapp -itr -p 11716 -c /var/tmp/cores/core.11716 -a /opt/dsee7/lib/64/ns-slapd -s /var/tmp/storage
\* ------------------------------------------------------------------------------
\* Oracle Corporation CSD pkgapp 3.4 Linux			    [04/22/2010]
\* ------------------------------------------------------------------------------
\* Checking install epoch/delta		 [0 seconds]
\* Checking pkgapp path			 [using /opt/pkgapp]
\* Checking for locale en_US.utf8	 [success]
\* Checking usage history		 [not recently run]
\* OS Release [Linux]			 [2.6.9-5.EL]
\* Platform/Type				 [x86_64 x86_64]
\* Checking [-p] pid			 [using pid 11716]
\* Checking [-a] path			 [using /opt/dsee7/lib64/ns-slapd]
\* Process root				 [/opt/dsee7/lib64/ns-slapd]
\* Checking [-c] core			 [using core /var/tmp/cores/core.11716]
\* Databin parameter [-s] checks		 [reset to /var/tmp/storage]
\* Creating Databin directories		 [/var/tmp/storage]
\* Databin used/created is		 [/var/tmp/storage/pkgapp-042210-01]
\* Creating temp area			 [/tmp/pkgapp.11771/]
\* Process binary			 [ns-slapd]
\* ns-slapd binary bit version		 [64]
\* core.11716 bit version		 [64]
\* Checking path [-a] to binary name	 [success, path includes binary name]
\* Checking path [-a] exists		 [success]
\* Locating ns-slapd			 [success]
\* Checking located ns-slapd is 64 bits.	 [success]
\* Binary located			 [/opt/dsee7/lib64/ns-slapd]
\* Adding binary to pkgapp.pldd		 [success]
\* Grabbing libs from /proc		 [success]
\* Grabbing gdb backtrace		 [success]
\* Grabbing [-i] core/gcore		 [success]
\* Grabbing core/gcore cksum		 [success]
\* Javatools [-j] not available		 [skipped]
\* Grabbing /var/log/messages		 [success]
\* Grabbing uname -a			 [success]
\* Grabbing date/time			 [success]
\* Grabbing rpm -qa			 [success]
\* Grabbing /etc/redhat-release		 [success]
\* Grabbing ulimit -a			 [success]
\* Grabbing libs			 [success]
\* Making lib paths app/		 	 [success]
\* Making lib paths libs/		 [success]
\* Linking libraries1 of 71      	 [success]
\* Libraries linked			 [71 ttl]
\* 					
\* Using hostid for naming .tar.gz	 [120ad308]
\* Writing file				 [pkgapp-120ad308-s-6048a-x6250d-vbox1-gmp03.sun.com-100422-182724.tar.gz]
\* 					
\* Done gathering files			
\* Writing dbxrc & opencore.sh files	 [success]
\* Writing coreinfo.sh file		 [success]
\* Writing validate-libs.sh file		 [success]
\* Writing manifest-100422-182724.log	 [success]
\* Writing pkgapp-args-100422-182724	 [success]
\* Creating final tarfile		 [success]
\* Compressing tarfile			 [success]
\* End of runtime logging		
\* Saving history info			 [success]
\* Saving runtime log			 [success]
\* Saving Library list			 [success]
\* Saving Library list README		 [success]
\* Removing [-r] temp area/files		 [removed]
\* 					
\* Operations Complete			
\* Total runtime				 [0 minutes]
\* ------------------------------------------------------------------------------
\* Upload the following file(s) to http://supportfiles.sun.com Cores Directory at
Oracle
					
1) File(s) located in directory /var/tmp/storage/pkgapp-042210-01
					
		[ pkgapp-120ad308-s-6048a-x6250d-vbox1-gmp03.sun.com-100422-182724.tar.gz ]
					


				Thank you.
				Customer Service Delivery
				Global Systems Support


NOTES:
1) You can check for updates to this script here:
	BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
	Blog - http://blogs.sun.com/Dirtracer

2) GDD information located here:
	Rel Notes - http://docs.sun.com/app/docs/doc/820-0437
	Docs - http://www.sun.com/service/gdd/index.xml
	Video - http://mediacast.sun.com/search?query=gdd
	Video - http://mediacast.sun.com/search?query=dirtracer

3) Please send all Bugs and RFE's to the following address:
	Subject "pkgapp bug/rfe" - pkgapp-feedback_ww@oracle.com

4) See the following reference files.
	/var/tmp/pkgapp-history/runtime-100422-182724.log
	/var/tmp/pkgapp-history/history-100422-182724.log
	/var/tmp/pkgapp-history/library.list-100422-182724
	/var/tmp/pkgapp-history/library.list-100422-182724.README

Enjoy,

Lee

Wednesday Dec 16, 2009

The New DirTracer (DT7) and Pkgapp (3.4) are here!

Just in time for the holidays!

I am pleased to finally announce new versions of not only Pkgapp, but DirTracer as well.Photo © Lee Trujillo

Updated: March 03, 2011 with the Oracle Support Doc ID's for these.

  • DirTracer7 (DT7) - Oracle Support Doc ID 1291443.1
  • Pkgapp 3.4 - Oracle Support Doc ID 1274584.1

Both products have gone through some serious revisions to make them not only easier on the processes being tracked, but also to make their captured data slimmer. Here is a complete list of the changes to each product.  Please see the included Reference Guide for more detail.

DT7 (DirTracer 7.0)

Improvements

  1. Directory Server 7 compatible. DT7 can work with directory server versions 5.1, 5.2.x, 6.x as well as 7.0.
  2. Data capture size reduced by ½. Disk usage with a Basic Capture in DT6.0.7 was 4169 kb.  The same capture size is now 2191 kb.
  3. Internal tar files removed.
  4. Safer when used against ds processes...removed pflags, pldd, pmap, pfiles as they can create locking issues with the ds process on some os versions.  Currently the only use of these proctools are against gcores or cores if provided.
  5. Checks available databin space when a gcore is requested.  Will disable GRAB_GCORE if space is needed.  Helps customers from inadvertently filling disk space to 100% with a gcore.
  6. New validate data script provided.  Checks the manifest against the captured data.
  7. Now uses Pkgapp 3.3.
  8. Pkgapp captures are now included with final tar.  Customers no longer have to upload a second file for data analysis. 
  9. DT7 now has an epoch check feature alerting the user if the DT7 version being used is more than 120 days old.   A simple alert to have the customer check Big Admin for a newer version.
  10. Renamed the databin from a numbered path to include the name dirtracer.
  11. Changed the name of some config parameters.
  12. Changed how the high cpu tracking is seen.
  13. New data captured.
  14. Added an overall timer which is displayed at the end of the DT7 run.
  15. Renamed the README.dirtracer to README.changelog
  16. Created a text based README.FIRST.  This is a Quickstart Guide if you don't have time to read the full Reference Guide.
  17. Renamed the command file from "dirtracer" to dt7.  Also renamed the config file from "dirtracer.config" to "dt.conf" to make commands easier to type.

Fixes

  1. Fixed the nsds50ruv search to be a scope sub search as opposed to one.
  2. Fixed (enhanced) the Control C feature so users can Control C once to exit the Main Loop; this advances DT7 to the Post Loop data gathering phase.  Control C a second time allows the user to exit even the Post Loop.
  3. Fixed a minor issue with how the values were store in the dt.conf.last file for the GATHER_N_XXXXX_LOGS parameters.
  4. Fixed the paths to check “carpet” type for Directory Server 7.
  5. Fixed an issue in Configurator which wouldn't return the stty echo if you used Control C.
  6. Fixed an issue where DT7 couldn't derive the correct bit mode (32 or 64bits) when SERVER_DOWN=1.  Both DT7 and Configurator ask the user for this information if it is not available.
  7. Fixed an issue where Pkgapp isn't gathered if DT7 is set to capture 2 gcores and is exited before the second gcore is gathered.

Pkgapp 3.3

Improvements

  1. Added a new parameter -l.  This takes a known good pldd text file to be used in place of either a pldd against the pid or the core.  This allows admins to use a previously generated pldd text file, instead of attaching to the live pid or a possibly bad core file.
  2. Changed the overall usage. Both can be used on Solaris but the -p <pid> is now required on Linux.  This is due to a limitation of the ldd command and lack of pldd on Redhat and SuSE linux.  ldd misses most libs that a real process runs with. -p is now used for the <pid> and -c is now sed for the <core>.
  3. Added an Epoch (date of the first run) check to alert users when their pkgapp version is 120 days old.   Just a simple reminder to check if a newer version exists.
  4. Uses the /proc file system to quickly gather library names instead of using pldd etc (when a pid is used).  Solaris - uses pldd on the core if available. Linux - requires a pid to use the /proc fs as ldd is not accurate.
  5. Created a validate-libs.sh file which can be used to validate all the libraries gathered (as seen in the manifest) in the capture paths.
  6. Now copies the library-list and a README to the /var/tmp/pkgapp-history/
  7. Updated the post-run NOTES: section to include new links and information.

Fixes

  1. Fixed bug 6881865: 6881865 Pkgapp 3.2 fails to exec gb when pkgapp is run from outside its install path
  2. Fixed bug 6882174: 6882174 Pkgapp 3.2 fails to gather jstack/jinfo

What is next?

Due to time constraints (my regular job, etc.) I had to cut development of DT7 off as DS7 was released.  I had a bigger list of the enhancements I wanted to add but ran out of time.  Here are a few of them I want to explore for DT7.1.

  1. Anonymizer. Data anonymizer for sensitive customer data.
  2. ReptopCk inclusion
  3. Search Debug Feature. Set debug: 4 on, play SRCH/MOD etc, set debug off. Great for debugging problem searches/mods.
  4. Add precise timing for all DT executions. Would get the date before each command ever done as well as adding it to a log...so you can see each command executed in a linear list.
  5. For DS6 and 7 do an ls -laR on the CARPETPATH and INSTANCEPATH
  6. Rewrite the config file processor
  7. Add a new command processor. This would allow for more flexibility to run a specific search or use a Dtrace script.

More on these later.

I hope the new versions help you out, and remember you can always send me feedback for new improvements or for issues encountered.

 Happy Holidays and Regards,

 Lee

Friday Nov 13, 2009

Pkgapp 3.3 Soon to be released!

Wow I've been busy but all for the good.  Photo © Lee Trujillo

I've been in development with SLX, internal Engineer Videos for DirTracer as well as DT7 (DirTracer 7) and now Pkgapp 3.3 development.

I've fixed a couple bugs, added a few enhancements and...changed its usage a bit... 

More on these changes later but here are some example run's in the mean time.

Pkgapp 3.3 Beta 5 Solaris

\* ----------------------------------------------------------------------------------
\* Sun Microsystems TSC pkgapp 3.3b5 Solaris                             [11/13/2009]
\* ----------------------------------------------------------------------------------
\* Checking install epoch/delta          [0 seconds]
\* Checking pkgapp path                  [using /opt/pkgapp]
\* Checking usage history                [not recently run]
\* OS release [Solaris]                  [5.10]
\* Platform/Type                         [SUNW,Sun-Blade-T6300 sparcv9]
\* Checking [-p] pid                     [using pid 20578]
\* Checking [-a] path                    [reset to /usr/lib]
\* Process root                          [/usr/lib]
\* Checking [-c] core                    [using core /var/tmp/cores/core.20578]
\* Databin parameter [-s] checks         [reset to /var/tmp/storage]
\* Creating Databin directories          [/var/tmp/storage]
\* Databin used/created is               [/var/tmp/storage/pkgapp-111309-01]
\* Creating temp area                    [/tmp/pkgapp.27404/]
\* Checking corefile for a valid pldd    [pldd is good with 36 elements]
\* Checking if corefile is truncated     [CORE core.20578 is GOOD. Core contains 106 segments ]
\* Continuing...                         
\* Process binary                        [sendmail]
\* sendmail binary bit version           [32]
\* Checking path [-a] to binary name     [success, path != binary name]
\* Checking path [-a] is a directory     [success]
\* Locating sendmail                     [success]
\* Checking located sendmail is 32 bit   [success]
\* Binary located                        [/usr/lib/sendmail]
\* Adding binary to pkgapp.pldd          [success]
\* Grabbing libs from /proc              [success]
\* Grabbing pstack from core             [success]
\* Grabbing pmap from core               [success]
\* Grabbing pcred from core              [success]
\* Grabbing pflags from core             [success]
\* Grabbing pargs from core              [success]
\* Grabbing [-i] core/gcore              [success]
\* Grabbing core/gcore cksum             [success]
\* Javatools [-j] not set                [skipped]
\* Grabbing /var/adm/messages            [success]
\* Grabbing uname -a                     [success]
\* Grabbing date/time                    [success]
\* Grabbing showrev -p                   [success]
\* Grabbing pkginfo -l                   [success]
\* Grabbing /etc/release                 [success]
\* Grabbing coreadm                      [success]
\* Grabbing ulimit                       [success]
\* Grabbing libs                         [success]
\* Making lib paths app/                 [success]
\* Making lib paths libs/                [success]
\* Linking libraries                     [success]
\* Libraries linked                      [80 ttl]
\*                                       
\* Using hostid for naming .tar.gz       [84aaaac8]
\* Writing file                          [pkgapp-84aaaac8-s-6000a-t6300i-zone8-sca11-091113-125258.tar.gz]
\*                                       
\* Done gathering files                  
\* Writing dbxrc & opencore.sh files     [success]
\* Writing validate-libs.sh file         [success]
\* Writing manifest-091113-125258.log    [success]
\* Writing pkgapp-args-091113-125258     [success]
\* Creating final tarfile                [success]
\* Compressing tarfile                   [success]
\* End of runtime logging                
\* Saving history info                   [success]
\* Saving runtime log                    [success]
\* Saving Library list                   [success]
\* Saving Library list README            [success]
\* Removing [-r] temp area/files         [left alone]
\*                                       
\* Operations Complete                   
\* Total runtime                         [0 minutes]
----------------------------------------------------------------------------------
Upload the following file(s) to your supportfiles.sun.com Cores Directory at Sun
                                        
1) File(s) located in directory /var/tmp/storage/pkgapp-111309-01
                                        
                [ pkgapp-84aaaac8-s-6000a-t6300i-zone8-sca11-091113-125258.tar.gz ]
                                        


                                Thank you.
                                Sun Software Technology Service Center (STSC)


NOTES:
1) You can check for updates to this script here:
        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
        Blog - http://blogs.sun.com/Dirtracer
        Sun Learning Exchange Video's - https://slx.sun.com/search/node/dirtracer

2) Release Notes and Guides located here:
        Docs - http://docs.sun.com/app/docs/doc/820-0437

3) GDD information located here:
        Docs - http://www.sun.com/service/gdd/index.xml

4) Please send all Bugs and RFE\\'s to the following address:
        Subject "pkgapp bug/rfe" - gdd-issue-tracker@sun.com

5) Please send all other questions etc to:
        Subject "pkgapp feedback" - gdd-feedback@sun.com

6) See the following reference files.
        /var/tmp/pkgapp-history/runtime-091113-125258.log
        /var/tmp/pkgapp-history/history-091113-125258.log
        /var/tmp/pkgapp-history/library.list-091113-125258
        /var/tmp/pkgapp-history/library.list-091113-125258.README


Pkgapp 3.3 Beta 5 Redhat (same as SuSe)

\* ----------------------------------------------------------------------------------
\* Sun Microsystems TSC pkgapp 3.3.b5 Redhat				 [11/13/2009]
\* ----------------------------------------------------------------------------------
\* Checking install epoch/delta		 [0 seconds]
\* Checking pkgapp path			 [using /opt/SUNWpkgapp]
\* Checking usage history		 [not recently run]
\* OS Release [Redhat]			 [2.6.9-5.EL]
\* Platform/Type				 [x86_64 x86_64]
\* Checking [-p] pid			 [using pid 686]
\* Checking [-a] path			 [reset to /opt/sun/dir/ds6/lib]
\* Process root				 [/opt/sun/dir/ds6/lib]
\* Checking [-c] core			 [using core /var/cores/core.686]
\* Databin parameter [-s] checks		 [reset to /var/tmp/storage]
\* Creating Databin directories		 [/var/tmp/storage]
\* Databin used/created is		 [/var/tmp/storage/pkgapp-111309-01]
\* Creating temp area			 [/tmp/pkgapp.11773/]
\* Process binary			 [ns-slapd]
\* ns-slapd binary bit version		 [32]
\* Checking path [-a] to binary name	 [success, path != binary name]
\* Checking path [-a] is a directory	 [success]
\* Locating ns-slapd			 [success]
\* Checking located ns-slapd is 32 bit	 [success]
\* Binary located			 [/opt/sun/dir/ds6/lib/ns-slapd]
\* Adding binary to pkgapp.pldd		 [success]
\* Grabbing libs from /proc		 [success]
\* Grabbing [-i] core/gcore		 [success]
\* Grabbing core/gcore cksum		 [success]
\* Javatools [-j] not set		 [skipped]
\* Grabbing /var/log/messages		 [success]
\* Grabbing uname -a			 [success]
\* Grabbing date/time			 [success]
\* Grabbing rpm -qa			 [success]
\* Grabbing /etc/redhat-release		 [success]
\* Grabbing ulimit			 [success]
\* Grabbing libs				 [success]
\* Making lib paths app/			 [success]
\* Making lib paths libs/		 [success]
\* Linking libraries8 of 68      	 [success]
\* Libraries linked			 [68 ttl]
\* 					
\* Using hostid for naming .tar.gz	 [120ad308]
\* Writing file				 [pkgapp-120ad308-s-6048a-x6250d-vbox1-gmp03.uk.sun.com-091113-205150.tar.gz]
\* 					
\* Done gathering files			
\* Writing dbxrc & opencore.sh files	 [success]
\* Writing coreinfo.sh file		 [success]
\* Writing validate-libs.sh file		 [success]
\* Writing manifest-091113-205150.log	 [success]
\* Writing pkgapp-args-091113-205150	 [success]
\* Creating final tarfile		 [success]
\* Compressing tarfile			 [success]
\* End of runtime logging		
\* Saving history info			 [success]
\* Saving runtime log			 [success]
\* Saving Library list			 [success]
\* Saving Library list README		 [success]
\* Removing [-r] temp area/files		 [left alone]
\* 					
\* Operations Complete			
\* Total runtime				 [0 minutes]
\* ----------------------------------------------------------------------------------
Upload the following file(s) to your supportfiles.sun.com Cores Directory at Sun
					
1) File(s) located in directory /var/tmp/storage/pkgapp-111309-01
					
		[ pkgapp-120ad308-s-6048a-x6250d-vbox1-gmp03.uk.sun.com-091113-205150.tar.gz ]
					


				Thank you.
				Sun Software Technology Service Center (STSC)


NOTES:
1) You can check for updates to this script here:
	BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
	Blog - http://blogs.sun.com/Dirtracer
	Sun Learning Exchange Video's - https://slx.sun.com/search/node/dirtracer

2) Release Notes and Guides located here:
	Docs - http://docs.sun.com/app/docs/doc/820-0437

3) GDD information located here:
	Docs - http://www.sun.com/service/gdd/index.xml
	Video - https://slx.sun.com/search/node/GDD
	Video - https://slx.sun.com/search/node/DIRTRACER

4) Please send all Bugs and RFE\\'s to the following address:
	Subject "pkgapp bug/rfe" - gdd-issue-tracker@sun.com

5) Please send all other questions etc to:
	Subject "pkgapp feedback" - gdd-feedback@sun.com

6) See the following reference files.
	/var/tmp/pkgapp-history/runtime-091113-205150.log
	/var/tmp/pkgapp-history/history-091113-205150.log
	/var/tmp/pkgapp-history/library.list-091113-205150
	/var/tmp/pkgapp-history/library.list-091113-205150.README


Lee

Thursday Jul 30, 2009

DirTracer, Pkgapp and SLX

Alo all,

I received good news from the Big Admin admins today, the new DirTracer and Pkgapp versions are online!

The following versions are now available here on Big Admin. For information on the latest changes in this update, please see my previous blog - DT 6.0.7 Coming soon!

DirTracer 6.0.7 Photo © Lee Trujillo

Pkgapp 3.2

I have also released the SLX 02 DirTracer Quickstart on SLX (Sun Learning Exchange) - "This video is a Quickstart guide to getting DirTracer6 running in a short amount of time using Configurator 6. I also step through a real example of how DirTracer runs using the config file created by Configurator."

Finally, I am in production of the latest DirTracer SLX video "Server Impact".  Using Slamd, Cacti and DirTracer I plan on showing exactly what impact a Directory Server will see from DirTracer runs based on different run/problem types.  More on this later.

Cheers!

Lee

Tuesday Jun 02, 2009

Pkgapp 3.2 for Solaris available on BigAdmin

 

Hey all,

Quick update to say Pkgapp 3.2 is now available externally on the Big Admin scripts site.  You can download it here.

Currently I am working on Pkgapp 3.2 ports to Linux.

Big Admin Scripts: Sun Java System Category

Big Admin MOTD

Cheers!

Lee

Thursday May 21, 2009

Pkgapp 3.2 for Solaris released!

Hi all!

Pkgapp 3.2 is available internally...yeah!

Artistic Bridge

I finally had time to update the Pkgapp documentation and retest it on various application cores/gcores and it is now ready for prime time.  I have also submitted it to BigAdmin and it should be available in the next few days.  If you need it now, please contact your favorite Sun Support representative and request he send it from the internal download site.  I will also update "this" post once Big Admin has uploaded the new version.

Pkgapp 3.2 has 2 big new features...

  1. It includes adm64 libraries that were previously not included making debugging easier on that platform for java issues.
  2. It includes a new core truncation feature developed by a member of the Directory Server Sustaining team.  The great news is, the feature not only works on Directory Server cores but any other core/gcore you may be dealing with!

If the core/gcore is good, the user is given the following message:

\* Checking if corefile is truncated     [CORE gcore.cshttpd.28734 GOOD Core contains 293 segments ]

If the core is truncated, the user is alerted and required to answer yes/no as to whether they understand the data may be totally unusable.

\* Checking if corefile is truncated     [WARNING! core gcore.cshttpd.28734.truncated is truncated (40396864d bytes instead of 40397397d bytes)]
\*                                     
\* ALERT! The core was found to be truncated, it therefore may not be usable
\* pkgapp will continue to gather data and you can still upload the result to Sun
\*                                     
\* Do you understand the core may not be usable? [yes|no] yes
                                        [Answer was -> yes]
\* Continuing...  

If you use/have an RSS Reader such as NetNewsWire or  Google Reader you can see the latest updates to Big Admin using the following links.

Big Admin Scripts: Sun Java System Category

Big Admin MOTD

Next up...I plan to finalize and release DTR and begin production of my first SLX video for Dirtracer users!

Take care,

Lee

Monday Nov 24, 2008

Pkgapp 3.0 for Solaris Available!

Hello all,

I'm please to announce I have released Pkgapp 3.0 for Solaris!

So far Pkgapp 3.0 is only availabe for Solaris but I will begin porting this code to Linux and HPUX.  I wanted to get Solaris first instead of waiting for all to be done...this way at least one unix flavor is available for use.

You can obtain Pkgapp 3 from Big Admin: Pkgapp 3.0 Solaris

Sun Employees can also find information on Sunspace.

Included files:

  • Pkgapp version 3 Script
  • Pkgapp Reference Guide: SUN-GDD_Pkgapp_Reference_Guide3.0.0.pdf
  • Pkgapp Readme
  • GDD Readme
  • GDD License

For information on what changed, what the usage looks like and a runtime example (using mysql) see my last post - Pkgapp 3.0 code has been finalized!

Enjoy all!

Lee

Wednesday Nov 19, 2008

Pkgapp 3.0 code has been finalized!

Pkgapp3 is the latest update to the Sun GDD tool set. These tools are designed to help customers gather the data needed for the best possible support on their issues.

Pkgapp is a shell tool built by Sun Support Engineering to help gather all OS and application libraries used by Sun Java Systems (and others) servers at runtime.  Pkgapp can be used to gather these libs, etc, from cores produced by crashes or gcores captured manually by administrators.

Pkgapp is part of the GDD (Gathering Debug Data) suite of tools and has been used for years to help Sun Support debug tricky crashes, as well as performance and memory leak problems where cores or gcores are available.

New major features in Pkgapp 3.0:

1) Now handles Java Cores and grabs jstack and jinfo.
 - See the -j and -J options

2) Includes all functionality that was available in both pkg_app 2.7 and pkgcore.

3) Can allow for the temporary files to be retained in /tmp
 - See the -r option

4) Allows pkgapp3 to be placed in a $PATH location; can now be run from any location.

5) Can use relative paths for the following options:
 -c <core file OR pid of a running process>
 -p <full path to, but not including the process binary> (ns-slapd, imapd, httpd etc.)
 -s <Storage; path to store the final tar file in>

6) Retains all lib/binary paths (pkgcore) as well as the pkg_app app/ location for older pkg_app aware applications.

7) Names the final storage path as pkgapp-<date>-<index> to help administrators locate the final tar file faster.
 - Ex: pkgapp3-111708-01/

8) Gathers the following Solaris only proctool commands from the core.
 - pldd
 - pstack
 - pmap
 - pcred
 - pflags
 - pargs

9) Checks to see if the core file is truncated and checks the core files cksum to ensure the file size match.

10) Gathers the following core and pkgapp3 specific data
 - core file cksum
 - core file truncation (good/bad)
 - core file size (ls -l)
 - library list (from the pldd)
 - manifest log (all files included int he final tar)
 - pkgapp3 arguments used.

11) Gathers the following OS specific data
 - coreadm info
 - date/time
 - messages (from /var/adm)
 - pkginfo
 - showrev -p
 - /etc/release (Solaris)
 - ulimit
 - uname -a

12) Creates a coreinfo.sh script to allow Sun Engineers to quickly run "coreinfo" for ns-slapd processes (only).

13) Alerts administrators they must upload 2 files (pkgapp tar & core) separately when they do not use the -i (include core) switch.

14) Will exit with a fatal error if a pldd cannot be properly retrieved from the core and alerts the administrator to run pkgapp3 using a pid as the -c option.

15) Saves the pkgapp3 arguments and runtime log into /var/tmp/pkgapp-history/.  Helps administrators and pkgapp see previous runs.

16) Updated usage and alert messages.

 Here are a couple previews of how the new code works.

 Example: Usage output

\* ----------------------------------------------------------------------------------
\* Sun Microsystems RSD pkgapp 3.0 Solaris                               [11/24/2008]
\* ----------------------------------------------------------------------------------
pkgapp 3.0, a Sun Microsystems data gathering utility.
Usage:
  pkgapp [options] -c <core file | pid> -p <full path (path only) to process binary> -s [path to write tar file]

Required parameters:
 -c <core file OR pid of a running process>
 -p <full path to, but not including the process binary> (ns-slapd, imapd, httpd etc.)

Optional parameters:
 -i (Include a previously generated core file with the final tar.gz)
 -j (Javacore; process a java core)
 -r (Remove all temp files)
 -q (Quiet)
 -d (Debug)
 -J <JstackPath; path to the jstack (jdk) commands>
        defaults to /usr/jdk/instances/...
 -s <Storage; path to store the final tar file in>

usage:  pkgapp -c <name of the core file> -p <path to process binary>
usage:  pkgapp -c <pid of the running app> -p <path to process binary>

Examples: these are examples mixing various parameters

Directory Server
pkgapp -i -r -c ./core.14740 -p /var/mps/ds52p4/bin/slapd/server/64

Messaging Server
pkgapp -c ./core.3496 -p /opt/SUNWmsgsr/lib

Web Server
pkgapp -c ./core.1092 -p /space/iws70/lib -s /var/crash

Calendar Server
pkgapp -r -c ./core -p /opt/SUNWics5/cal/lib

Sendmail
pkgapp -i -c 512 -p /usr/lib

Mysqld
pkgapp -i -r -c ./core -p /support/mysql-5.0.41/bin

Example: Runtime output

\* ----------------------------------------------------------------------------------
\* Sun Microsystems RSD pkgapp 3.0 Solaris                               [11/24/2008]
\* ----------------------------------------------------------------------------------
\* OS release                            [5.10]
\* Platform                              [SUNW,Sun-Blade-2500]
\* Checking [-c] is a core or pid        [using pid 1409]
\* Process Root                          [/support/mysql-5.0.41/data]
\* Databin parameter [-s] checks         [reset to /var/tmp/dev]
\* Databin found                         [/var/tmp/dev/test/cores/storage]
\* Databin writable check                [success]
\* Databin used/created is               [/var/tmp/dev/test/cores/storage/pkgapp-112408-05]
\* Creating temp area                    [/tmp/pkgapp.2928/]
\* Pid used, no corefile to check       
\* Process binary                        [mysqld]
\* Checking usage history                [not recently run]
\* mysqld binary bit version             [32]
\* Checking path [-p] to binary name     [success, path != binary name]
\* Checking path [-p] is a directory     [success]
\* Locating mysqld                       [success]
\* Checking located mysqld is 32 bit     [success]
\* Binary located                        [/support/mysql-5.0.41/bin/mysqld]
\* Adding binary to pkgapp.pldd          [success]
\* Grabbing pldd                         [success]
\* Grabbing pstack                       [success]
\* Grabbing pmap                         [success]
\* Grabbing pcred                        [success]
\* Grabbing pflags                       [success]
\* Grabbing pargs                        [success]
\* Provide the full path and name to the core file
\* If you do not have a core, enter "none"
\* Example - /data/cores/core.1445:      /var/tmp/dev/cores/core.1409
                                        [Answer was -> /var/tmp/dev/cores/core.1409]
\* Grabbing [-i] core/gcore              [success]
\* Javatools [-j] not set                [skipped]
\* Grabbing /var/adm/messages            [success]
\* Grabbing uname -a                     [success]
\* Grabbing date/time                    [success]
\* Grabbing showrev -p                   [success]
\* Grabbing pkginfo -l                   [success]
\* Grabbing /etc/release                 [success]
\* Grabbing coreadm                      [success]
\* Grabbing ulimit                       [success]
\* Grabbing libs                         [success]
\* Making lib paths app/                 [success]
\* Making lib paths libs/                [success]
\* Linking libraries                     [success]
\* Libraries linked                      [62 ttl]
\*                                      
\* Using hostid for naming .tar.gz       [837872d0]
\* Writing file                          [pkgapp-837872d0-s4u-2500a-brm04-081124-095809.tar.gz]
\*                                      
\* Done gathering files                 
\* Writing dbxrc & opencore.sh files     [success]
\* Writing manifest-081124-095809.log    [success]
\* Writing pkgapp-args-081124-095809     [success]
\* Creating final tarfile                [success]
\* Compressing tarfile                   [success]
\* End of runtime logging               
\* Saving history info                   [/var/tmp/pkgapp-history/history-081124-095809.log]
\* Saving runtime log                    [/var/tmp/pkgapp-history/runtime-081124-095809.log]
\* Removing [-r] temp area/files         [removed]
\*                                      
\* Operations Complete                  
----------------------------------------------------------------------------------
Upload the following file(s) to your supportfiles.sun.com Cores Directory at Sun

1) File(s) located in directory /var/tmp/dev/test/cores/storage/pkgapp-112408-05

                [ pkgapp-837872d0-s4u-2500a-brm04-081124-095809.tar.gz ]



                                Thank you.
                                Sun Software Technology Service Center (STSC)


NOTES:
1) You can check for updates to this script here:
        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
2) Release Notes and Guides located here:
        Docs - http://docs.sun.com/app/docs/doc/820-0437
3) GDD information located here:
        Docs - http://www.sun.com/service/gdd/index.xml

4) Please send all Bugs and RFE's to the following address:
        Subject "pkgapp bug/rfe" - gdd-issue-tracker@sun.com

5) Please send all other questions etc to:
        Subject "pkgapp feedback" - gdd-feedback@sun.com


I will announce Pkgapp 3.0 for Solaris's availability in the next few days...stay tuned!

Lee

About

A Tech Blog about the Sun Java Systems Dirtracer Toolkit. Dirtracer and this blog written and maintained by Lee Trujillo an Oracle Senior Principal Support Engineer.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today