Monday Aug 18, 2008

Reference tuning architecture - Directory Server running on small systems

Following is an effective reference tuning architecture for servers with insufficient physical memory to cache the entire Directory Server database in database cache and entry cache(s).

Directory Server uses a single database cache for all entries in the database, that is, db/\*/\*.db3 files, and one entry cache per backend. The database cache is a copy in pages of the .db3 files, the entry cache is essentially a hashmap of entries formatted to return to the LDAP client. The best possible performance occurs when the entry being searched for is in the entry cache and can be returned immediately to the client. The worst possible performance occurs when Directory Server must read a page containing an entry from disk and the page is not in the file system buffer cache.

Referring to the diagram below, when an LDAP client issues a search, Directory Server examines the entry cache for the entry and returns the entry if found. If the entry is not in the entry cache, Directory Server examines the pages in the database cache for the entry. If the entry is found in the database cache, the entry is formatted, placed in the entry cache, and returned to the client. If the entry is neither in the entry cache or the database cache, then Directory Server must issue a disk read to read the page containing the entry - assuming it exists - into the database cache, then format the entry for the entry cache, then return the entry to the client. If the page containing the entry is not in the filesystem buffer cache, then that page must be read from disk.


In some large databases, all entries cannot be cached in database cache and entry cache. When all entries cannot be cached, tuning efforts should concentrate on reducing disk I/O on search operations. This is accomplished by caching the database file pages in the file system buffer cache as follows: set segmap_percent in /etc/system to accommodate the database, reboot the server, and prime the file system cache with dd, set database cache and entry cache sizes to something reasonable, perhaps 2GB each - the database cache size will help with updates. Directory Server response times should be excellent in this scenario. To determine the amount of memory needed for the file system cache, sum the number of bytes in all the db3 files in the database and multiply the result by 1.2 (20%). This is easily done with ls and awk:

ls -l db/\*/\*.db3 | awk '{ total += $5; } END { print total \* 1.2; }'

I/O caused by Directory Server reading data from disk can have a negative impact on Directory Server response times. Not only can response times be elevated, the response times are subject to a greater correlation of variation, i.e., response times are "spiky". The following graph shows the results of a 40 minute test run against Directory Server 5.2 patch 4 under extreme load, with the nsslapd-db-home-directory set to a local filesystem:

Baseline 40Min Spikes

in the diagram above, disk service times on the filesystem containing the mmap backing files averaged above 500 ms.

Change the nsslapd-db-home-directory to a tempfs filesystem, restart Directory Server, and run a new test. After restarting directory server, there will be a short "caching" period as the database cache and entry cache are filled, resulting in:


The gradual increase in average response times up to about 1000 seconds into the test is the caching period. Immediately afterwards run the same job again and Directory Server begins to respond with very even average response times: this is because there is no I/O on search operations:


Technorati Tags: , , , , ,

Friday May 02, 2008

The following is a perl script that I use to have a quick look at some values in the ldbm database tree. Example usage and output is:
 # ./ --root /var/opt/sun/ds6 --pwdFile ~/ds6pwd  --bindDN 'cn=dmanager'
localhost 22:23:29 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
localhost 22:23:30 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
localhost 22:23:31 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
localhost 22:23:32 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
The script is very basic and easily extendable.
=== snip ===
#! /usr/bin/perl -w

# 08-FEB-2008

# Parameter and variables

my ( $lines, $linesPerPage, $loadPerConnection );

# set up some default values
$port      = 389;
$root      = "/ldap/";
$instance  = "slapd-native";
$delay     = 1;
$hostname  = "localhost";
$pidFile   = $root . "/logs/pid";
$bindDN    = "cn=directory manager";
$pwdFile   = "ds6pwd";
$ldapsearch = "/opt/sun/dsrk6/bin/ldapsearch";
# parameters that cannot be modified by
# command line options
$dbCacheDN="cn=monitor,cn=ldbm database,cn=plugins,cn=config";

sub displayUsageHints() {
  print " prints a tabular display of some directory server\\n";
  print "        operational information.\\n";
  print "\\n";
  print "Usage:\\n";
  print " [OPTIONS]\\n";
  print "\\n";
  print "[--hostname hostname]           LDAP server hostname, default: $hostname\\n";
  print "[--port port]                   LDAP server port, default: $port\\n";
  print "[--instance instance]           LDAP server instance, default: $instance\\n";
  print "[--delay delay]                 delay in seconds between samples, default: $delay\\n";
  print "[--pwdFile filename]            file containing the root DN password, default: $pwdFile";
  print "[--bindDN bindDN]               root DN, default $bindDN";
  print "[--ldapsearch ldapsearchBinary] ldapsearch binary, default $ldapsearch";
  print "[--help]                        display command invocation hints\\n";

# process command line options
$sn = 0;
for (@ARGV) {
  if (/--port/) {
    $port = $ARGV[++$sn];
  } elsif (/--hostname/) {
    $hostname = $ARGV[++$sn];
  } elsif (/--root/) {
    $root = $ARGV[++$sn];
  } elsif (/--instance/) {
    $instance = $ARGV[++$sn];
  } elsif (/--delay/) {
    $delay = $ARGV[++$sn];
  } elsif (/--pidFile/) {
    $pidFile = $ARGV[++$sn];
  } elsif (/--pwdFile/) {
    $pwdFile = $ARGV[++$sn];
  } elsif (/--bindDN/) {
    $bindDN = $ARGV[++$sn];
  } elsif (/--ldapsearch/) {
    $ldapsearch = $ARGV[++$sn];
  } elsif (/--help/) {

$scratchFile = "/tmp/" . "cache-" . `date +%H%m%S`;

$lines = 0;
$linesPerPage = 30;

for (;;) {
  # database cache
  $searchBase = $dbCacheDN;
  $cmd = "$ldapsearch -D '$bindDN' -j '$pwdFile' -s base -h $hostname -p $port -b '$searchBase' '(objectClass=\*)'  > $scratchFile";

  open(scratchFile,$scratchFile) || die "Failed to open $scratchFile: $!";

  while ()


    ($attribute,$value) = split(" ",$_,2);

    if (/dbcachehitratio/) {
      $dbcachehitratio = $value;
    } elsif (/dbcachepagein/) {
      $dbcachepagein = $value;
    } elsif (/dbcachepageout/) {
      $dbcachepageout = $value;
    } elsif (/dbcacheroevict/) {
      $dbcacheroevict = $value;
    } elsif (/dbcacherwevict/) {
      $dbcacherwevict = $value;

    $attribute = "";

  # Get the current time
  ($sec,$min,$hour,$junk,$junk,$junk,$junk,$junk,$junk) = localtime(time);

  printf "%9.9s %2.2d:%2.2d:%2.2d hitratio:%d pagein:%d pageout:%d roevict:%d rwevict:%d\\n",

  sleep $delay;

=== snip ===

Sunday Nov 25, 2007

Directory Server FAQ - Wiki

There is new content in Directory Server FAQ.

Technorati Tags:

Friday Oct 19, 2007

LDAP as a naming service articles on Sun wiki

There are some excellent new articles on the Sun Wiki about using DSEE as a naming service.

Technorati Tags: , , , ,

db_stat article at Sun wiki

There is a new article about db_stat on the eternal wiki. Follow the link.

Technorati Tags: , , , , , , ,


Sun, LDAP, SLAMD, DSLA, java, Struts, networking, chess, books, cooking, wine, and many other things.


« July 2016