Monday Apr 12, 2010

Pkgapp 3.4 in the hopper!

Releasing soon

It has been a long time since my last update but a lot has transpired in the past few months.

Sun was bought by case you've been hiding in a cave waiting for the latest :)  As you can see I am still employed with Oracle. Although my company has changed, my day to day duties have remained the same: Sun Directory Server support, as well as DirTracer and Pkgapp maintenance.

With version 3.3, I changed how I gathered libraries from cores and processes. As a result, some international users had issues with localized versions of Sun web server.  Because of this bug and a few others, as well as needing a re-branding of the Pkgapp product from Sun to Oracle, I've been working these past few months on Pkgapp 3.4.

The code looks tight and the Pkgapp Reference Guide (pdf) is complete.  Once I get the okay from Oracle Legal on licensing, I can release this new version.  I'm hoping this will happen in the next few days, so stay tuned.


Thursday Jul 30, 2009

DirTracer, Pkgapp and SLX

Alo all,

I received good news from the Big Admin admins today, the new DirTracer and Pkgapp versions are online!

The following versions are now available here on Big Admin. For information on the latest changes in this update, please see my previous blog - DT 6.0.7 Coming soon!

DirTracer 6.0.7 Photo © Lee Trujillo

Pkgapp 3.2

I have also released the SLX 02 DirTracer Quickstart on SLX (Sun Learning Exchange) - "This video is a Quickstart guide to getting DirTracer6 running in a short amount of time using Configurator 6. I also step through a real example of how DirTracer runs using the config file created by Configurator."

Finally, I am in production of the latest DirTracer SLX video "Server Impact".  Using Slamd, Cacti and DirTracer I plan on showing exactly what impact a Directory Server will see from DirTracer runs based on different run/problem types.  More on this later.



Thursday May 07, 2009

DTR (Dirtracer Reporter) is coming and other news!

Hi all,

Thought I would drop in a quick blog and let you know what I've been up to.

Directory Masters Event in Somerset New Jersey Lee Trujillo talking DirTracer at the Directory Master event in Somerset NJ

As mentioned in my last blog I was asked to present Dirtracer at the DM event.  The event, hosted by Michael Melore, Wajih Ahmed and Ludovic Poitou was great!  I was able to meet people I've only talked to on the phone or emailed over the past 5 years.  I was able to meet with Sun Partners and Sun Development and Support Engineers...

I presented an overview of Dirtracer and its components as well as my latest project, DTR (Dirtracer Reporter); more on this below.  I have to admit I was very nervous.  In past lives I was a pre-sales engineer and presentations like this were easy.  Needless to say I haven't presented to a large group since joining Sun; just not something I do in my Support role.

In the end I forgot about being nervous and soon was on a roll.

It was great meting you all, I had a blast.  See Ludo's Sketches blog for more info on this event.

Dirtracer Reporter (DTR)

Over the past couple of months I have been ratifying a few data reporting tools I and others have created into one nice package called DTR.  Dirtracer Reporter is a bundle of existing Directory Server data reporting scripts rolled into a single usable interface.  DTR shines when run against Dirtracer captured data, it however can run against basic Directory Server data such as dse.ldif and log files.

I am in the final test phase and making sure a slight glitch here and there are fixed and hope to release to Big Admin in the next coming weeks.  More to come on DTR soon.

Sun Learning Exchange (SLX)

One of my other sub projects which will kick off soon is the creation of SLX videos for Dirtracer.  These will be video podcast in nature and will show how Dirtracer is used for various problem types.

 Here is a list of topics I hope to highlight.

  • Dirtracer overview
  • Process Hangs
  • High CPU
  • Replication
  • Crashes
  • Memory Leaks
  • Server Down issues
  • Basic Captures
  • Configuration only captures
  • Configurator for Dirtracer usage

These videos will supplement my blog content as well as giving you audio and visual queues of how Dirtracer looks when running for a specific problem type.

Until my next blog...ciao!


Tuesday May 06, 2008

Tracing ns-slapd process id's dynamically.

I was asked on the monthly Directory Collaborators call something to the effect of the following...

"It looks like Dirtracer is bound to one process id (pid) in the config file, but what if the server is restarted, does it know the new pid number dynamically"?

Currently, no.  Dirtracer was built to trace one pid at a time and must be changed in the dirtracer.config file before running.

I thought more on this and came to the conclusion it is possible in a future version of Dirtracer.  If you have a dirtracer.config or dirtracer.config.last (the config from the last run dirtracer capture) but the process id has changed, I could back track using the pid file located in the <slapd instance name>/logs location to use the current pid.

This would be based on the dirtracer.config file parameter INSTANCENAME.

If the INSTANCENAME parameter was not set (would be in a dirtracer.config.last or dirtracer.config.x) then I would have to abort the run.

While possible, it would however have limitations (gotchyas) that could prevent data capture.  If the directory server was hung/deadlocked and an old dirtracer.config file was using with BORKED="0" (Not Hung) then when launched, Dirtracer would attempt an ldapsearch and itself hang waiting on the search return.

Note: because Dirtracer is a shell script and relies on ldapsearch as opposed to an internal search mechanism it is limited to the ldapsearch capabilities therefor does not have an option (currently) to timeout a connection.  I have been thinking recently of using my timer function for this issue.

This dynamic pid check would mostly be useful for a couple scenarios I can think of.

1) Dirtracer run's to gather historical/config info using a cron without having to check/recheck the running pid vs config file setting (in the event the ds was restarted).

2) To set up Problem Type config files in advance without the current ns-slapd process's pid number embedded in the config file.  In this case the admin could just run ./dirtracer -f ./dirtracer.config.hang with very short notice and capture data quicker than having to edit the config file and change the pid parameter.

I will think this idea over some more but it has merit so it could make its way into the next release.


Thursday May 01, 2008

The DATABIN parameter

I had a question from a Front Line Engineer recently where they did not understand how to select a proper location for the DATABIN parameter.

DATABIN="<DATA OUTPUT PATH>"    # Databin main path.
                                # Sub dirs will be created beneath this path.

The DATABIN is the path where you want Dirtracer to store the data it captures.  Special care should be taken when selecting the right path based on the project size of the data you need to gather.

Sun GDD Directory Dirtracer Reference Guide: Page 17

Disk Usage

Disk space used is almost entirely dependent on the following.

1. How Dirtracer is configured; i.e. what it is asked to gather.
2. How many loops Dirtracer is configured to complete.
  •     cache, monitor searches
  •     netstats, iostats, pstacks, prstats
  •     transaction log ls -l captures.
3. How many access/error and audit logs are captured.  
  •     configured from the GATHER_N_XXXXX_LOGS="N" parameters.
4. How big each of those logs are.  (var/adm/messages logs included)
5. Shared Memory (MMAP) files
  •     how big the ns-slapd process size is.
6. Cores
  •     how big the ns-slapd process size is.
7. Gcores
  •     how big the ns-slapd process size is.
8. If Dirtracer has REMOVE_TEMP_DATA=0.
  •     saves all temp files in addition to the final tar file.
9. If Dirtracer has SKIP_TAR_GZIP=1.
  •     Skips the final tar/gz saving 1⁄2 the space it normally uses; i.e. duplication of files  occurrs as files are tarred and gzipped.

The Engineer was also requested to setup two directory.config files to trace two separate slapd instances at the same time, on the same system.  Would the DATABIN parameters need to be different? No.

Early on I saw a problem with Stracer (the old Dirtracer) when customers would use the same DATABIN to store data and the previously capture files would be overwritten or you would have multiples of the same files.

I solved this issue by having Dirtracer create a unique time/date based directory under the defined DATABIN path.  Even if Dirtracer is run multiple times on the same system a new sub databin is created to segregate the data.


1) Set the DATABIN as follows.


2) Run Dirtracer 3 times and observe the directories created in /var/tmp/data/

root[/var/tmp/data]#ls -l
total 12
drwxr-xr-x  11 root     other       1536 Apr 21 15:38 042108-01
drwxr-xr-x  11 root     other       1536 Apr 21 15:42 042108-02
drwxr-xr-x  11 root     other       1536 Apr 21 15:43 042108-03

root[/var/tmp/data]#find . -name "dirtracer-\*gz" -print

You can clearly see how the data is separated and should not collide.


Configurator, the dirtracer.config.template and their uses.

I was recently asked what the differences are between the dirtracer.config.template and the Configurator script and how they are used.

The previous version of my script Stracer used both a config file as well as a full range of command line switches.  The command line switches confused many and the config file then was not well documented.  As a result we had many Dirtracer's configured to capture the wrong type of data for the problem type.

Shortly after I decided to create the "Configurator", and released it with Stracer 1.9.3.  Configurator took the Problem Type encountered by the Customer and translated it into a working dirtracer.config file.  Originally Configurator contained 7 problem type options.  With Configurator 6.0.6 I have added Option 8 for a Configuration Only Capture.

Sun Microsystems Configurator 6.0.6                                       
Please choose the type of problem you are experiencing

Process Hung                            [1]
High CPU                                [2]
Replication                             [3]
Crashing                                [4]
Memory Leak                             [5]
Server Down                             [\*]     DISABLED - (SLAPDPID is set)
Basic Capture                           [7]
Config Only Capture                     [8]

NOTE: Now that the Document for Dirtracer has progressed to this point I may have to add a full section for Configurator; even though it's interactive and self explaining.

Configurator takes you through the following sections in which to create a dirtracer.config file

1) Case Number (if available)
2) Slapd Instance selection.
3) Directory Manager Password entry
4) Data Storage location.  This is the location of the DATABIN parameter where all captured data will be stored.
5) Skip Tar Gzip question
6) Problem Type selection.
    a) Process Hung. Hang detection, Gcore selection
    b) High CPU. CPU % thrshold level, Gcore selection
    c) Replication.  Sets replication debug logging (8192)
    d) Crashing.
    e) Memory Leak.
    f) Server Down. DS version [5x|6x], Instance path entry.
    g) Basic Capture
    h) Config Only Capture
7) DS Log capture selection; access, error and audit logs.
8) Dirtracer Runtime selection.
9) Pmonitor ( Runtime selection.
10) Configuration Summary
11) Data Capture Size guesstimation.
12) Config file (dirtracer.config) creation.

The Configurator is a good way to for those new to Dirtracer to quickly setup a dirtracer.config file for an event.

So what is the difference between the Configurator and the dirtracer.config.file template?  Well, Configurator asks questions to setup a ready to use dirtracer.config.  The dirtracer.config.template is just that...a template.  The dirtracer.config.template does contain all parameters available that would be set when creating a new dirtracer.config using the Configurator.  The dirtracer.config.template does however "have" to be edited in order to be used with Dirtracer and does not have Presets for Problem Types.

Without the following parameters properly set/changed, Dirtracer will exit and alert the admin the file needs to be changed.  Likewise the template contains some default settings.

SLAPDPID="<SLAPD PID>"          # Slapd pid number
MGRPW="<PASSWORD>"              # Mgr password
DATABIN="<DATA OUTPUT PATH>"    # Databin main path.

The template can be copied, renamed and edited to contain different parameter settings for the same problem types as seen above.  The dirtracer.config.template is completely self documented so administrators can quickly look at a parameter and select its use (or not).

Hope this was useful.


Presenting at the Directory Collaboration Meeting

Good news!

I will be one of the speakers in this months Directory Collaboration Meeting this coming Monday May 02, 12pm ET.  Partners, if you normally attend this, it will be an opportunity to ask me questions on Dirtracer usage and the future of my product.

I will be giving a mini preso on Dirtracer for those who are unfamiliar with it; is this even possible? :)  In addition to the new features you can take advantage of in the new version 6.0.6.

Directory Collaboration:
"Regularly scheduled monthly Directory Services collaborative meetings providing information in field experiences, deployment and configuration strategies, knowledge sharing, and questions and answers."

Hope to see you there!


Wednesday Apr 30, 2008

Dirtracer 6.0.6 Unleashed!

Dirtracer 6.0.6 Unleashed!

The latest version of Dirtracer is now available for Customer and Partner download on the BigAdmin System Administration Portal.

The major changes included in 6.0.6 are as follows:

1) Added SKIP_TAR_GZIP Code.  Skips all tar and gzip functions.
 - \*   Preparing files - pstack            [skipped - skip tar gzip enabled]

2) Added Carpet Version Type Check. Gets a better DS Type from 6.x
 - \* DS Version                            [6.1 - 64 bit - zip install]

 - 6658311: Dirtracer (GDD): 6.0.5 can report the wrong install type - zipinstall vs pkginstall (jes).

3) Reset the ldapsearch path ..uses locateLdapTools function
 - 5.1 

\* Ldap Tools Path                       [/data/sunone/ds51sp4/shared/bin]

 - 5.2 

\* Ldap Tools Path                       [/opt/ds52p6/shared/bin]

 - 6.x 

\* Ldap Tools Path                       [/opt/dsee62/dsrk6/bin]

4) Added a secondary check for backends. Highlights if DS6 & No Backends exist.

 \* Backends Found                        [none configured]

5) ps -ae changed to include ps -aef
 \* ps -aef                             [success]

6) Add a checkPatch for Sol 8 on 108995-08
 - 108995-08
 - SunOS 5.8: /usr/lib/ patch

 - Sol 5.9

7) Added CONFIG_ONLY runtime option.  See Ref. Guide for Config Only Capture.

8) Added an ls -laR of the slapd instance
 \* Gathering needed customer defined configuration
 \*   ls -laR of slapd Instance           [success]

Stay tuned for explanations of the new features and how they can benefit you.


Friday Apr 25, 2008

The Dirtracer Blog is here!

Welcome to the new Dirtracer Blog!

I am Lee Trujillo, an Engineer supporting the Sun Java Directory Server since December 2003.  I am also the creator of Dirtracer, Sun's number one tool for tracing issues for our Sun Directory Server.

Early in 2004, I saw that the support organization had no formalized standard for asking for and obtaining data related to issues surrounding the Directory Server, so I created Stracer (Stack Tracer), the predecessor to Dirtracer (Directory Tracer).  Stracer 1.0 consisted of 169 lines of shell script that basically gathered pstacks, prstats and top output from a running ns-slapd process.  Conversely, Dirtracer 6x (2008) is a complex 2,590 line script of functions that can be combined in many ways to gather Directory data based on problem type.

Dirtracer is a troubleshooting tool designed to help reduce resolution time on complex Directory Server problems and to ease the data-gathering process for Sun's customers.

Dirtracer is part of the GDD (Gathering Debug Data) suite of tools and has already been used for years to tackle some of the most persistent, difficult Sun Java Systems Directory Server problems faced in the field.  For problems such as server hangs, crashes, and high cpu utilization, Dirtracer simplifies the sampling of system resources and crash data in order to help identify trends.

Save you and your customers time and aggravation -- discover the power of Dirtracer.

I expect Dirtracer version 6.0.6 to be available later today on the external Big Admin Administration Portal site for Customer and partner download.


A Tech Blog about the Sun Java Systems Dirtracer Toolkit. Dirtracer and this blog written and maintained by Lee Trujillo an Oracle Senior Principal Support Engineer.


« July 2016