Wednesday Dec 16, 2009

The New DirTracer (DT7) and Pkgapp (3.4) are here!

Just in time for the holidays!

I am pleased to finally announce new versions of not only Pkgapp, but DirTracer as well.Photo © Lee Trujillo

Updated: March 03, 2011 with the Oracle Support Doc ID's for these.

  • DirTracer7 (DT7) - Oracle Support Doc ID 1291443.1
  • Pkgapp 3.4 - Oracle Support Doc ID 1274584.1

Both products have gone through some serious revisions to make them not only easier on the processes being tracked, but also to make their captured data slimmer. Here is a complete list of the changes to each product.  Please see the included Reference Guide for more detail.

DT7 (DirTracer 7.0)


  1. Directory Server 7 compatible. DT7 can work with directory server versions 5.1, 5.2.x, 6.x as well as 7.0.
  2. Data capture size reduced by ½. Disk usage with a Basic Capture in DT6.0.7 was 4169 kb.  The same capture size is now 2191 kb.
  3. Internal tar files removed.
  4. Safer when used against ds processes...removed pflags, pldd, pmap, pfiles as they can create locking issues with the ds process on some os versions.  Currently the only use of these proctools are against gcores or cores if provided.
  5. Checks available databin space when a gcore is requested.  Will disable GRAB_GCORE if space is needed.  Helps customers from inadvertently filling disk space to 100% with a gcore.
  6. New validate data script provided.  Checks the manifest against the captured data.
  7. Now uses Pkgapp 3.3.
  8. Pkgapp captures are now included with final tar.  Customers no longer have to upload a second file for data analysis. 
  9. DT7 now has an epoch check feature alerting the user if the DT7 version being used is more than 120 days old.   A simple alert to have the customer check Big Admin for a newer version.
  10. Renamed the databin from a numbered path to include the name dirtracer.
  11. Changed the name of some config parameters.
  12. Changed how the high cpu tracking is seen.
  13. New data captured.
  14. Added an overall timer which is displayed at the end of the DT7 run.
  15. Renamed the README.dirtracer to README.changelog
  16. Created a text based README.FIRST.  This is a Quickstart Guide if you don't have time to read the full Reference Guide.
  17. Renamed the command file from "dirtracer" to dt7.  Also renamed the config file from "dirtracer.config" to "dt.conf" to make commands easier to type.


  1. Fixed the nsds50ruv search to be a scope sub search as opposed to one.
  2. Fixed (enhanced) the Control C feature so users can Control C once to exit the Main Loop; this advances DT7 to the Post Loop data gathering phase.  Control C a second time allows the user to exit even the Post Loop.
  3. Fixed a minor issue with how the values were store in the dt.conf.last file for the GATHER_N_XXXXX_LOGS parameters.
  4. Fixed the paths to check “carpet” type for Directory Server 7.
  5. Fixed an issue in Configurator which wouldn't return the stty echo if you used Control C.
  6. Fixed an issue where DT7 couldn't derive the correct bit mode (32 or 64bits) when SERVER_DOWN=1.  Both DT7 and Configurator ask the user for this information if it is not available.
  7. Fixed an issue where Pkgapp isn't gathered if DT7 is set to capture 2 gcores and is exited before the second gcore is gathered.

Pkgapp 3.3


  1. Added a new parameter -l.  This takes a known good pldd text file to be used in place of either a pldd against the pid or the core.  This allows admins to use a previously generated pldd text file, instead of attaching to the live pid or a possibly bad core file.
  2. Changed the overall usage. Both can be used on Solaris but the -p <pid> is now required on Linux.  This is due to a limitation of the ldd command and lack of pldd on Redhat and SuSE linux.  ldd misses most libs that a real process runs with. -p is now used for the <pid> and -c is now sed for the <core>.
  3. Added an Epoch (date of the first run) check to alert users when their pkgapp version is 120 days old.   Just a simple reminder to check if a newer version exists.
  4. Uses the /proc file system to quickly gather library names instead of using pldd etc (when a pid is used).  Solaris - uses pldd on the core if available. Linux - requires a pid to use the /proc fs as ldd is not accurate.
  5. Created a file which can be used to validate all the libraries gathered (as seen in the manifest) in the capture paths.
  6. Now copies the library-list and a README to the /var/tmp/pkgapp-history/
  7. Updated the post-run NOTES: section to include new links and information.


  1. Fixed bug 6881865: 6881865 Pkgapp 3.2 fails to exec gb when pkgapp is run from outside its install path
  2. Fixed bug 6882174: 6882174 Pkgapp 3.2 fails to gather jstack/jinfo

What is next?

Due to time constraints (my regular job, etc.) I had to cut development of DT7 off as DS7 was released.  I had a bigger list of the enhancements I wanted to add but ran out of time.  Here are a few of them I want to explore for DT7.1.

  1. Anonymizer. Data anonymizer for sensitive customer data.
  2. ReptopCk inclusion
  3. Search Debug Feature. Set debug: 4 on, play SRCH/MOD etc, set debug off. Great for debugging problem searches/mods.
  4. Add precise timing for all DT executions. Would get the date before each command ever done as well as adding it to a you can see each command executed in a linear list.
  5. For DS6 and 7 do an ls -laR on the CARPETPATH and INSTANCEPATH
  6. Rewrite the config file processor
  7. Add a new command processor. This would allow for more flexibility to run a specific search or use a Dtrace script.

More on these later.

I hope the new versions help you out, and remember you can always send me feedback for new improvements or for issues encountered.

 Happy Holidays and Regards,


Tuesday May 26, 2009

DirTracer now DS7 compatible!

 Yes it's true!  DirTracer now works with DS7!It's the Tubes

I was fortunate to get access to DS7 during the Directory Masters event in New Jersey.  While there I was able to quickly install DS7 and migrate an old dead DS6.2 instance.  I ran DT 6.0.6 against this new instance and was ecstatic to see it ran with 0 errors.  In reality DS Development had moved two commands but DT had used its defaults to guess things like install type (zip/native) and where the dsrk's ldapsearch and ldapmodify were located.

Today I decided to revisit these two items and found it was easy as pie to change; only took a few minutes.  I had to copy/paste and change approximately seven lines of code (mostly path based) and that was it. 

To mirror the DT version I used when DS6 was released, I will now re-version this as DirTracer 7.0.0 or DT7.  While DT7 works perfectly with DS7,  I'm not saying DT7 won't change.  In fact I'm planning many new features for DT7 for its first

Sun Microsystems dirtracer 7.0.0 Solaris Sparc                        05/26/2009
\* Checking Config File                  [success 1211182548 != 3006111464]
\* Using config file                     [./dirtracer.config]
\* OS Type and Version                   [SunOS 10 sparc]
\* Dirtracer mainloop will run for       [1 sec.]
\* Databin parameter check 1             [success]
\* Databin parameter check 2             [success]
\* Databin Found                         [/var/tmp/dtcapture]
\* Databin used is                       [/var/tmp/dtcapture/052609-02]
\* Relocating runtime log                [success]
\* Case Number                           [NOT AVAILABLE]
\* User is root. Continuing...          
\* PID Check     (18925)                 [success]
\* Process State                         [not hung]
\* High CPU tracking                     [off]
\* Mem Leak tracking                     [off]
\* Config Only Capture                   [on]
\* Pstack Force (-F)                     [off]
\* Instance Name is                      [/opt/dsee62/ldapinst3]
\* Using host name                       [s-6000a-t6300i-zone8-sca11]
\* Using host id                         [84aaaac8]
\* Ldap Tools Path                       [/opt/dsee7/dsrk/bin]
\* DS Version                            [7.0 - 64 bit - zip install]
\* Backends Found                        [sun]
\* Transaction log location              [/opt/dsee62/ldapinst3/db]
\*   Solaris tar found                   [/usr/sbin/tar]
\*   gzip found                          [/bin/gzip]
\*   elfdump found                       [/usr/ccs/bin/elfdump]
\*   pstack found                        [/bin/pstack]
\*   prstat found                        [/bin/prstat]
\*   netstat found                       [/bin/netstat]
\*   iostat found                        [/bin/iostat]
\*   vmstat found                        [/bin/vmstat]
\*   mpstat found                        [/bin/mpstat]
\*   ldapmodify found                    [/opt/dsee7/dsrk/bin/ldapmodify]
\*   ldapsearch found                    [/opt/dsee7/dsrk/bin/ldapsearch]
\* found                        [/opt/dirtracer/dirtracertools/]
\* Access/Error Logging                  [left as is]
\* Audit Logging                         [left as is]
\* Iostat available.     Executing...    [success]
\* Vmstat available.     Executing...    [success]
\* Mpstat available.     Executing...    [success]
\* (pmonitor) executed            [success]
\* interval(1) x checks(1)      [ run time (1 sec.)]
\* Entering Main Performance Gathering Loop
\* Loop 0 - 090526-132724                [pstack/prstat override on]
\* Exiting Main Performance Gathering Loop
\* Done gathering stats                  [ iostats vmstats etc]
\* Access/Error Logging                  [left as is]
\* Audit Logging                         [left as is]
\* Logging level                         [left as is]
\* Gathering needed customer defined configuration
\*   ls -laR of slapd Instance           [success]
\*   nsslapd-db-home-directory           [success]
\*   nsslapd-db-logdirectory             [success]
\*   nsslapd-directory db                [success]
\*   nsslapd-directory sun               [success]
\* Gathering Other Environment Information
\*   ns-slapd version                    [success]
\*   nsds50ruv dc=sun,dc=com             [success]
\*   cn=config search                    [success]
\*   rootdse search                      [success]
\*   idsktune                            [not found]
\*   dirtracer7.0.0                      [success]
\*   /etc/release                        [success]
\*   /etc/system                         cp: cannot access /etc/system
\*   /var/adm/messages                   [success]
\*   uname -a                            [success]
\*   uptime -a                           [success]
\*   date/time                           [success]
\*   ifconfig -a                         [success]
\*   mount -p                            [success]
\*   ps -aef                             [success]
\*   pflags                              [failed]
\*   pldd                                [failed]
\*   pmap                                [failed]
\*   pfiles                              [failed]
\*   prtconf -v                          prtconf: devinfo facility not available
\*   showrev -p                          [success]
\*   swap -s                             [success]
\*   pkginfo -l                          [success]
\*   df -k                               [success]
\*   logs - access: gathering 1 files   
\*       access                          [success]
\*       access rotation                 [success]
\*   logs - error: gathering 1 files    
\*       errors                          [success]
\*       error rotation                  [success]
\*   logs - audit: gathering 1 files    
\*       audit                           [success]
\*       audit rotation                  [success]
\*   config dse.ldif's                   [success]
\*   schema                              [success]
\*   shared mem files                    [skipped]
\* [ALERT 1] Shared Mem files were not included in the dirtracer tar file and were NOT copied/saved to the databin.
\*   ds lib files cksum                  [skipped]
\*   ds lib files elfdump                [skipped]
\* Writing used dirtracer.config.last    [success]
\* Copying dirtracer.config.last         [success]
\* Writing manifest-090526-132724.log    [success]
\* Packaging files                      
\*   Preparing files - pstack            [0 files]
\*   Preparing files - prstat            [0 files]
\*   Preparing files - netstat           [success]
\*   Preparing files - ds-config-data    [success]
\*   Preparing files - ds-logs           [success]
\*   Preparing files - ds-mmap-files     [success]
\*   Preparing files - ds-srch-monitor   [success]
\*   Preparing files - ds-srch-cache     [success]
\*   Preparing files - ds-txnlogs-data   [success]
\*   Preparing files - os-info           [success]
\*   Preparing files - gcore             [0 files]
\*   Preparing files - crash             [0 files]
\*   Preparing Main file                 [success]
\* Leaving my temp files intact         

Operations Complete
1) Dirtracer capture data located in directory [ /var/tmp/dtcapture/052609-02 ]

Upload "only" this file to your cores directory at Sun

        [ dirtracer-84aaaac8-s-6000a-t6300i-zone8-sca11-090526-132724.tar.gz ]

                                Thank you.
                                Sun Software Technology Service Center (STSC)

1) You can check for updates to this script here:
        BigAdmin -

2) Release Notes and Guides located here:
        Docs -

3) GDD information located here:
        Docs -

4) Please send all Bugs and RFE's to the following address:
        Subject "dirtracer bug/rfe" -

5) Please send all other questions etc to:
        Subject "dirtracer feedback" -


ALERTS ENCOUNTERED: See /var/tmp/dtcapture/052609-02/ALERTS-090526-132718.log

1) Shared Mem files were not included in the dirtracer tar file and were NOT copied/saved to the databin.

As I mentioned two blogs ago I was preparing to start releasing DT Video (pocasts?) on the Sun Learning Exchange (SLX).  The first draft is complete for my first video titled "DirTracer Overview".  I will start production of the video later today or tomorrow and hope to release it to SLX quite soon.




A Tech Blog about the Sun Java Systems Dirtracer Toolkit. Dirtracer and this blog written and maintained by Lee Trujillo an Oracle Senior Principal Support Engineer.


« August 2016