A long awaited feature...

Directory Server Enterprise Edition 6.x is built on SleepyCat Berkeley DB for the storage of the LDAP entries and indexes...

For many years, customers have requested that we provide a way to shrink the database files, reclaiming unused pages. Well, the version of the SleepyCat BDB that we are using with DS 6.1 now has a public API that offer this capability and we've added the feature in DSEE 6.1. There is now a new subcommand for dsadm the offline DS management CLI: repack.
The usage is the following:

ludo:dsee63 > ds6/bin/dsadm repack --help                         

Usage: dsadm repack [ -b ] INSTANCE_PATH SUFFIX_DN [SUFFIX_DN ...]

Repacks existing suffix

The accepted value for OPTIONS is:

-b, --backend
                Enables to specify backend name instead of SUFFIX_DN

For global options, use dsadm --help.

INSTANCE_PATH  Path of the Directory Server instance
SUFFIX_DN      Suffix DN (Distinguished Name) to repack

For more information, see dsadm(1M).

It operates on a Suffix and all DB files for this suffix are compacted. This includes the main data file (id2entry), all index files but also the replication changelog file if the suffix is Replicated and configured as a Master or Hub Replica. The compaction will process up to 8 files in parallel, each in its own thread.

Because database compaction is very IO intensive and requires exclusive access to the database, the Directory Server must be stopped to run this.

The time to do the compaction varies a lot depending on the overall size of the database, the number of updates done on the data since the creation of the suffix or last compaction, and mostly the performance of the disk subsystem. Or course, the larger the DB and the more changes, the longer it'll take but the more it'll regain disk space. In our experience with a database of approximately 10 million entries and many changes done since creation, the compaction process took approximately a couple of hours (on a v20 z).

Definitely, database compaction is not something that you want to run in your weekly maintenance routine. The downtime of Directory Server may be too long. But if the available disk's space is getting close to the low watermark, it is worth stopping the server and reclaiming some of the space, before thinking of expanding the disk partition.

Below is an example of the dc=example,dc=com DB files before and after compaction, and the commands used to do the compaction: stop the server, compact, start the server.

ludo:dsee63 > ll instances/ds1/db/example/
total 3640
drwx------   2 lpoitou  icnc        1536 Jul  3 10:45 .
drwxr-xr-x   3 lpoitou  icnc         512 Jul  9 17:39 ..
-rw-------   1 lpoitou  icnc          38 Jul  3 10:45 DBVERSION
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_aci.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 17:39 example_ancestorid.db3
-rw-------   1 lpoitou  icnc      147456 Jul  9 17:39 example_cn.db3
-rw-------   1 lpoitou  icnc      122880 Jul  9 17:39 example_entrydn.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_givenName.db3
-rw-------   1 lpoitou  icnc      835584 Jul  9 17:39 example_id2entry.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_mail.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_mailAlternateAddress.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_mailHost.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_member.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsCalXItemId.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nscpEntryDN.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsds5ReplConflict.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsLIProfileName.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsRoleDN.db3
-rw-------   1 lpoitou  icnc      131072 Jul  9 17:39 example_nsuniqueid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nswcalCALID.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_numsubordinates.db3
-rw-------   1 lpoitou  icnc       32768 Jul  9 17:39 example_objectclass.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_owner.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 17:39 example_parentid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pipstatus.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pipuid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdaccountlockedtime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdfailuretime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdgraceusetime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_seeAlso.db3
-rw-------   1 lpoitou  icnc      139264 Jul  9 17:39 example_sn.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_telephoneNumber.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_uid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_uniquemember.db3

ludo:dsee63 > ds6/bin/dsadm stop /local/demo/dsee63/instances/ds1
Server stopped

ludo:dsee63 > ds6/bin/dsadm repack /local/demo/dsee63/instances/ds1 dc=example,dc=com
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example'
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', LDAP entries, size 827392 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index aci, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index ancestorid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index ancestorid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index aci finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index cn, size 139264 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', LDAP entries finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index cn finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailHost, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailHost finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index member, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index member finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsCalXItemId, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsCalXItemId finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nscpEntryDN, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nscpEntryDN finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsds5ReplConflict, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsds5ReplConflict finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsLIProfileName, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsLIProfileName finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsRoleDN, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsRoleDN finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsuniqueid, size 122880 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nswcalCALID, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nswcalCALID finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsuniqueid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index entrydn, size 122880 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index numsubordinates, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index numsubordinates finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index entrydn finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index givenName, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index givenName finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index parentid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index parentid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipstatus, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipstatus finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipuid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipuid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdaccountlockedtime, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdaccountlockedtime finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdfailuretime, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdfailuretime finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdgraceusetime, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdgraceusetime finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index seeAlso, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index seeAlso finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index sn, size 131072 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index sn finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index owner, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index owner finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index telephoneNumber, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index telephoneNumber finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uniquemember, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uniquemember finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mail, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mail finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailAlternateAddress, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailAlternateAddress finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index objectclass, size 32768 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index objectclass finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example' ended.
[10/Jul/2007:14:11:46 +0200] - Repack finished.
[10/Jul/2007:14:11:46 +0200] - Waiting for 6 database threads to stop
[10/Jul/2007:14:11:47 +0200] - All database threads now stopped

ludo:dsee63 > ll instances/ds1/db/example/
total 1000
drwx------   2 lpoitou  icnc        1536 Jul  3 10:45 .
drwxr-xr-x   3 lpoitou  icnc         512 Jul 10 14:11 ..
-rw-------   1 lpoitou  icnc          38 Jul  3 10:45 DBVERSION
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_aci.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_ancestorid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_cn.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_entrydn.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_givenName.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_id2entry.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_mail.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_mailAlternateAddress.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_mailHost.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_member.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsCalXItemId.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nscpEntryDN.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsds5ReplConflict.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsLIProfileName.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsRoleDN.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsuniqueid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nswcalCALID.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_numsubordinates.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_objectclass.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_owner.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_parentid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pipstatus.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pipuid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pwdaccountlockedtime.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pwdfailuretime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdgraceusetime.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_seeAlso.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_sn.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_telephoneNumber.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_uid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_uniquemember.db3

ludo:dsee63 > ds6/bin/dsadm start /local/demo/dsee63/instances/ds1
Server started: pid=15983

ludo:dsee63 >

Technorati Tags: , ,

Comments:

Post a Comment:
Comments are closed for this entry.
About

This is the blog of a senior software engineer, specialized in LDAP, Directory Server and OpenDS. Ludovic Poitou works in France at the Grenoble Engineering Center, in the Directory Services Engineering team. Outside work, I love skiing and taking photo

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today