Monday May 05, 2008

Bork Bork Bork

The BORKED parameter.

I realize most of our international users of Dirtracer may not get the reference but the BORKED parameter got its name in part from the US TV show called The Muppet Show; something I watched as a child.  A show made up entirely of Puppets (Muppets) who's main character is called Kermit the Frog.

The BORKED parameter is a reference to the Swedish Chef special Bork-speak...a parody of a wacky Chef who speaks in unintelligible Swedish.  In certain tech circles "Borked" is synonymous with "Broken", and when developing this settings purpose the name just stuck.

Think of Borked as "Hung" when it comes to its use with Dirtracer.  If a Directory Server process is thought to be Hung, then use set the BORKED parameter to 1.


Note:  I plan to rename the parameter to HUNG (Config File only) in the next release.

What does BORKED="1" do?

Normally Dirtracer will run a set of the following searches and or modiify's, if Borked is set to 1 (on) then these searches etc. are skipped.  Setting Borked to 1 helps make sure Dirtracer itself doesn't hang waiting on these ldapsearches to return.  If Borked is not set to 1 when the Directory Server is suspected of being Hung then Dirtracer will not complete its data gathering.

    Backend Suffix names; naming contexts
    Backend Database names
    cache info searches
    nsds50ruv; replica ruv's
    cn=config info

Modifies; only completed if Dirtracer is configured to do so.

    PTS_CONFIG_LOGGING can set the server Logging level to the parameter value configured.  This sets the nsslapd-infolog-area or nsslapd-errorlog-level

        nsslapd-infolog-area: 4 sets Heavy Trace Logging
        nsslapd-infolog-area: 128 sets ACI debugging
        nsslapd-infolog-area: 8192  sets Replication Debug Logging

    See the following link for more info on Logging Levels

Dirtracer can also set (rarely used) Logging On or Off.

    TURN_LOGGING_ON="0"             # Turn On access/error logs
    TURN_LOGGING_OFF="0"            # Turn Off access/error logs
    AUDIT_LOGGING_ON="0"            # Turn On audit logs
    AUDIT_LOGGING_OFF="0"           # Turn Off audit logs

As mentioned Dirtracer only completes the above ldapmodify's if configured to do so.


Thursday May 01, 2008

The DATABIN parameter

I had a question from a Front Line Engineer recently where they did not understand how to select a proper location for the DATABIN parameter.

DATABIN="<DATA OUTPUT PATH>"    # Databin main path.
                                # Sub dirs will be created beneath this path.

The DATABIN is the path where you want Dirtracer to store the data it captures.  Special care should be taken when selecting the right path based on the project size of the data you need to gather.

Sun GDD Directory Dirtracer Reference Guide: Page 17

Disk Usage

Disk space used is almost entirely dependent on the following.

1. How Dirtracer is configured; i.e. what it is asked to gather.
2. How many loops Dirtracer is configured to complete.
  •     cache, monitor searches
  •     netstats, iostats, pstacks, prstats
  •     transaction log ls -l captures.
3. How many access/error and audit logs are captured.  
  •     configured from the GATHER_N_XXXXX_LOGS="N" parameters.
4. How big each of those logs are.  (var/adm/messages logs included)
5. Shared Memory (MMAP) files
  •     how big the ns-slapd process size is.
6. Cores
  •     how big the ns-slapd process size is.
7. Gcores
  •     how big the ns-slapd process size is.
8. If Dirtracer has REMOVE_TEMP_DATA=0.
  •     saves all temp files in addition to the final tar file.
9. If Dirtracer has SKIP_TAR_GZIP=1.
  •     Skips the final tar/gz saving 1⁄2 the space it normally uses; i.e. duplication of files  occurrs as files are tarred and gzipped.

The Engineer was also requested to setup two directory.config files to trace two separate slapd instances at the same time, on the same system.  Would the DATABIN parameters need to be different? No.

Early on I saw a problem with Stracer (the old Dirtracer) when customers would use the same DATABIN to store data and the previously capture files would be overwritten or you would have multiples of the same files.

I solved this issue by having Dirtracer create a unique time/date based directory under the defined DATABIN path.  Even if Dirtracer is run multiple times on the same system a new sub databin is created to segregate the data.


1) Set the DATABIN as follows.


2) Run Dirtracer 3 times and observe the directories created in /var/tmp/data/

root[/var/tmp/data]#ls -l
total 12
drwxr-xr-x  11 root     other       1536 Apr 21 15:38 042108-01
drwxr-xr-x  11 root     other       1536 Apr 21 15:42 042108-02
drwxr-xr-x  11 root     other       1536 Apr 21 15:43 042108-03

root[/var/tmp/data]#find . -name "dirtracer-\*gz" -print

You can clearly see how the data is separated and should not collide.


Configurator, the dirtracer.config.template and their uses.

I was recently asked what the differences are between the dirtracer.config.template and the Configurator script and how they are used.

The previous version of my script Stracer used both a config file as well as a full range of command line switches.  The command line switches confused many and the config file then was not well documented.  As a result we had many Dirtracer's configured to capture the wrong type of data for the problem type.

Shortly after I decided to create the "Configurator", and released it with Stracer 1.9.3.  Configurator took the Problem Type encountered by the Customer and translated it into a working dirtracer.config file.  Originally Configurator contained 7 problem type options.  With Configurator 6.0.6 I have added Option 8 for a Configuration Only Capture.

Sun Microsystems Configurator 6.0.6                                       
Please choose the type of problem you are experiencing

Process Hung                            [1]
High CPU                                [2]
Replication                             [3]
Crashing                                [4]
Memory Leak                             [5]
Server Down                             [\*]     DISABLED - (SLAPDPID is set)
Basic Capture                           [7]
Config Only Capture                     [8]

NOTE: Now that the Document for Dirtracer has progressed to this point I may have to add a full section for Configurator; even though it's interactive and self explaining.

Configurator takes you through the following sections in which to create a dirtracer.config file

1) Case Number (if available)
2) Slapd Instance selection.
3) Directory Manager Password entry
4) Data Storage location.  This is the location of the DATABIN parameter where all captured data will be stored.
5) Skip Tar Gzip question
6) Problem Type selection.
    a) Process Hung. Hang detection, Gcore selection
    b) High CPU. CPU % thrshold level, Gcore selection
    c) Replication.  Sets replication debug logging (8192)
    d) Crashing.
    e) Memory Leak.
    f) Server Down. DS version [5x|6x], Instance path entry.
    g) Basic Capture
    h) Config Only Capture
7) DS Log capture selection; access, error and audit logs.
8) Dirtracer Runtime selection.
9) Pmonitor ( Runtime selection.
10) Configuration Summary
11) Data Capture Size guesstimation.
12) Config file (dirtracer.config) creation.

The Configurator is a good way to for those new to Dirtracer to quickly setup a dirtracer.config file for an event.

So what is the difference between the Configurator and the dirtracer.config.file template?  Well, Configurator asks questions to setup a ready to use dirtracer.config.  The dirtracer.config.template is just that...a template.  The dirtracer.config.template does contain all parameters available that would be set when creating a new dirtracer.config using the Configurator.  The dirtracer.config.template does however "have" to be edited in order to be used with Dirtracer and does not have Presets for Problem Types.

Without the following parameters properly set/changed, Dirtracer will exit and alert the admin the file needs to be changed.  Likewise the template contains some default settings.

SLAPDPID="<SLAPD PID>"          # Slapd pid number
MGRPW="<PASSWORD>"              # Mgr password
DATABIN="<DATA OUTPUT PATH>"    # Databin main path.

The template can be copied, renamed and edited to contain different parameter settings for the same problem types as seen above.  The dirtracer.config.template is completely self documented so administrators can quickly look at a parameter and select its use (or not).

Hope this was useful.


Presenting at the Directory Collaboration Meeting

Good news!

I will be one of the speakers in this months Directory Collaboration Meeting this coming Monday May 02, 12pm ET.  Partners, if you normally attend this, it will be an opportunity to ask me questions on Dirtracer usage and the future of my product.

I will be giving a mini preso on Dirtracer for those who are unfamiliar with it; is this even possible? :)  In addition to the new features you can take advantage of in the new version 6.0.6.

Directory Collaboration:
"Regularly scheduled monthly Directory Services collaborative meetings providing information in field experiences, deployment and configuration strategies, knowledge sharing, and questions and answers."

Hope to see you there!


Wednesday Apr 30, 2008

Dirtracer 6.0.6 Unleashed!

Dirtracer 6.0.6 Unleashed!

The latest version of Dirtracer is now available for Customer and Partner download on the BigAdmin System Administration Portal.

The major changes included in 6.0.6 are as follows:

1) Added SKIP_TAR_GZIP Code.  Skips all tar and gzip functions.
 - \*   Preparing files - pstack            [skipped - skip tar gzip enabled]

2) Added Carpet Version Type Check. Gets a better DS Type from 6.x
 - \* DS Version                            [6.1 - 64 bit - zip install]

 - 6658311: Dirtracer (GDD): 6.0.5 can report the wrong install type - zipinstall vs pkginstall (jes).

3) Reset the ldapsearch path ..uses locateLdapTools function
 - 5.1 

\* Ldap Tools Path                       [/data/sunone/ds51sp4/shared/bin]

 - 5.2 

\* Ldap Tools Path                       [/opt/ds52p6/shared/bin]

 - 6.x 

\* Ldap Tools Path                       [/opt/dsee62/dsrk6/bin]

4) Added a secondary check for backends. Highlights if DS6 & No Backends exist.

 \* Backends Found                        [none configured]

5) ps -ae changed to include ps -aef
 \* ps -aef                             [success]

6) Add a checkPatch for Sol 8 on 108995-08
 - 108995-08
 - SunOS 5.8: /usr/lib/ patch

 - Sol 5.9

7) Added CONFIG_ONLY runtime option.  See Ref. Guide for Config Only Capture.

8) Added an ls -laR of the slapd instance
 \* Gathering needed customer defined configuration
 \*   ls -laR of slapd Instance           [success]

Stay tuned for explanations of the new features and how they can benefit you.


Friday Apr 25, 2008

The Dirtracer Blog is here!

Welcome to the new Dirtracer Blog!

I am Lee Trujillo, an Engineer supporting the Sun Java Directory Server since December 2003.  I am also the creator of Dirtracer, Sun's number one tool for tracing issues for our Sun Directory Server.

Early in 2004, I saw that the support organization had no formalized standard for asking for and obtaining data related to issues surrounding the Directory Server, so I created Stracer (Stack Tracer), the predecessor to Dirtracer (Directory Tracer).  Stracer 1.0 consisted of 169 lines of shell script that basically gathered pstacks, prstats and top output from a running ns-slapd process.  Conversely, Dirtracer 6x (2008) is a complex 2,590 line script of functions that can be combined in many ways to gather Directory data based on problem type.

Dirtracer is a troubleshooting tool designed to help reduce resolution time on complex Directory Server problems and to ease the data-gathering process for Sun's customers.

Dirtracer is part of the GDD (Gathering Debug Data) suite of tools and has already been used for years to tackle some of the most persistent, difficult Sun Java Systems Directory Server problems faced in the field.  For problems such as server hangs, crashes, and high cpu utilization, Dirtracer simplifies the sampling of system resources and crash data in order to help identify trends.

Save you and your customers time and aggravation -- discover the power of Dirtracer.

I expect Dirtracer version 6.0.6 to be available later today on the external Big Admin Administration Portal site for Customer and partner download.


A Tech Blog about the Sun Java Systems Dirtracer Toolkit. Dirtracer and this blog written and maintained by Lee Trujillo an Oracle Senior Principal Support Engineer.


« July 2016