LDAP Paged Results - More

To demonstrate what VLV control provides what is missing in 
Simple Paged Results specification, let us examine the data that 
is loaded in OpenDS. 

		
I have loaded OpenDS with 10001 entries of sample data.
Loading sample data into OpenDS is very simple.
Assume that OpenDS is installed under "/opt/OpenDS" directory. 
cd /opt/OpenDS/bin
./stop-ds
./import-ldif -n userRoot -A /opt/OpenDS/config/MakeLDIF/example.template
./start-ds
The tool that I am using here is the "ldapsearch" that is 
shipped with OpenDS which is capable of running LDAP 
searches with both Simple Paged Results control and VLV 
Control.

For demonstration purposes, let us add an administrator account.
$ ./ldapmodify -p 1389 -D "cn=directory manager" -w password -a 
dn: uid=admin,ou=people,dc=example,dc=com 
objectClass: top 
objectClass: person 
objectClass: organizationalPerson 
objectClass: inetOrgPerson 
sn: Administrator 
cn: Directory Administrator 
userPassword: password 
Add an ACI to grant full access to "admin" user 
$ ./ldapmodify -p 1389 -D "cn=directory manager" -w password
dn: dc=example,dc=com 
changetype: modify 
add: aci 
aci: (targetattr = "\*")(version 3.0; acl "Admin Access"; allow(all) 
userdn = "ldap:///uid=admin,ou=people,dc=example,dc=com";) 
Grant read permission on Simple Paged Results Control for 
user "admin". 

The OID for Simple Paged Results control is 
1.2.840.113556.1.4.319.
$ ./dsconfig -h localhost -p 1389 -D "cn=Directory Manager" -w password set-access-control-
handler-prop \\
--add global-aci:"(targetcontrol=\\"1.2.840.113556.1.4.319\\")(version 3.0; acl \\"Allow Simple Paged 
Results Access\\"; allow(read) userdn = \\"ldap:///uid=admin,ou=people,dc=example,dc=com\\";)" -n 
Let us run the ldapsearch with Simple Paged Results control
./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D 
"uid=admin,ou=people,dc=example,dc=com" -w password --simplePageSize 1000 
"objectclass=inetOrgPerson" 
SEARCH operation failed 
Result Code: 50 (Insufficient Access Rights) 
Additional Information: You do not have sufficient privileges to perform an unindexed search 
By default, the OpenDS does not allow search with more than 
4000 candidates unless you are using cn=directory manger
One way to fix this is to change the index entry limit to a desired 
value and another way of accomplishing the same is to create an 
administrative account and grant unindexed-search privilege.

I chose the former approach. Not because of any particular 
merit though.

So, let us change the default index limit of the index 
objectCalss to 20,000.
$ ./dsconfig -h localhost -p 1389 -D "cn=directory manager" -w password 
-n set-local-db-index-prop \\ --backend-name userRoot --index-name objectclass 
--set index-entry-limit:20000 
$ ./stop-ds 
$ ./rebuild-index -i objectclass -b dc=example,dc=com 
$ ./start-ds 
Also, grant unlimited lookthrough limit to "admin" user. Here is the 
more information on lookthrough limit.
$ ./ldapmodify -p 1389 -D "cn=directory manager" -w password 
dn: uid=admin,ou=people,dc=example,dc=com 
changetype: modify 
add: ds-rlim-lookthrough-limit 
ds-rlim-lookthrough-limit: -1 
Let us try the ldapsearch again.
./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D 
"uid=admin,ou=people,dc=example,dc=com" -w password --simplePageSize 2000 
"objectclass=inetOrgPerson" dn 
dn: uid=user.0,ou=People,dc=example,dc=com 
dn: uid=user.1,ou=People,dc=example,dc=com 
dn: uid=user.2,ou=People,dc=example,dc=com 
dn: uid=user.3,ou=People,dc=example,dc=com 
. 
. 
. 
SEARCH operation failed 
Result Code: 4 (Size Limit Exceeded) 
Additional Information: This search operation has sent the maximum of 1000 entries to the client 
The above command requests the OpenDS to return all entries in 
pages with 2000 entries per page. However, the client receives LDAP 
error code 4 (Size Limit Exceeded). Which means that the 
server can't send send 2000 entries back to the client as requested. 
The size limit is the constraint that the OpenDS server imposes with 
default value set to 1000. More information on size limit is here.
So, let us tune our ldapsearch and decrease the page size to 1000.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D 
"uid=admin,ou=people,dc=example,dc=com" -w password --simplePageSize 1000 
"objectclass=inetOrgPerson" dn 
dn: uid=user.0,ou=People,dc=example,dc=com
dn: uid=user.1,ou=People,dc=example,dc=com
dn: uid=user.2,ou=People,dc=example,dc=com
dn: uid=user.3,ou=People,dc=example,dc=com
dn: uid=user.4,ou=People,dc=example,dc=com
dn: uid=user.5,ou=People,dc=example,dc=com
dn: uid=user.6,ou=People,dc=example,dc=com
dn: uid=user.7,ou=People,dc=example,dc=com
.
.
.
We got our paged results. All of 10002 entries are returned with 
1000 entries at a time. I have noticed that some LDAP server 
administrators use Simple Paged Control to return all
the entries to generate reports, perform other operations on the data.
Now, let us examine VLV control and see how it can provide us 
with a better paged results where a client can request the server to
"sort" on an attribute for the results and where to "start" in the results set.


The VLV Control allows a client to request the server in a

manageable list of entries.


The Control provides following options to the client.

1. before:after:index:count 
before: Specify the number of entries before the target entry 
after: Specify the number of entries after the target entry 
index: Offset to the target entry. Number 1 means first entry 
count: Number of the size of result set for a search filter. 

2. before:after:index 
before: Specify the number of entries before the target entry 
after: Specify the number of entries after the target entry 
index: string within the result set (index=worrell) 

Again, the ldapsearch is the tool to use VLV control . In the following 
command the ldapsearch sends a VLV request to the server with 
following options (-G 0:1:1:0)

before: 0 - Zero entries before the target
after: 1 - One entry after the target
index: 1 - First Entry is the target.
Count: 0 - Unknown. Let the server determine the size.

The server returns two entries two entries one before the target and 
the target itself. Most importantly, the result set is sorted on 
"sn" attribute.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D 
"uid=admin,ou=people,dc=example,dc=com" -w password -G 0:1:1:0 --sortOrder sn 
"objectclass=inetOrgPerson" 
dn: uid=user.4,ou=People,dc=example,dc=com 
objectClass: person 
objectClass: organizationalperson 
objectClass: inetorgperson 
objectClass: top 
postalAddress: Aartjan Aalders$60403 South Street$Salem, MS 22403 
postalCode: 22403 
uid: user.4 
description: This is the description for Aartjan Aalders. 
userPassword: {SSHA}HBLw+/8K4Tf1fjzCyDyyCuhEmAO4+lL2Cmz5rw== 
employeeNumber: 4 
initials: AVA 
givenName: Aartjan 
pager: +1 165 230 1073 
mobile: +1 096 889 4907 
cn: Aartjan Aalders 
telephoneNumber: +1 804 202 6921 
sn: Aalders 
street: 60403 South Street 
homePhone: +1 326 592 0306 
mail: user.4@example.com 
l: Salem 
st: MS 

dn: uid=user.5,ou=People,dc=example,dc=com 
objectClass: person 
objectClass: organizationalperson 
objectClass: inetorgperson 
objectClass: top 
postalAddress: Abagael Aasen$33591 Main Street$Lansing, NV 51577 
postalCode: 51577 
uid: user.5 
description: This is the description for Abagael Aasen. 
userPassword: {SSHA}3OoD2VVJOou9klYCX2i6ayO2LbY2tUF8x5Z3nw== 
employeeNumber: 5 
initials: AQA 
givenName: Abagael 
pager: +1 331 917 5915 
mobile: +1 258 717 0685 
cn: Abagael Aasen 
telephoneNumber: +1 039 625 4980 
sn: Aasen 
street: 33591 Main Street 
homePhone: +1 052 066 4526 
mail: user.5@example.com 
l: Lansing 
st: NV 
# VLV Target Offset: 1 
# VLV Content Count: 10002 
Let us look at another example. Request the server to return the target
entry that whose "sn" attribute begins with "jensen" and one 
followed by the target.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D 
"uid=admin,ou=people,dc=example,dc=com" -w password -G 0:1:jensen --sortOrder sn 
"objectclass=inetOrgPerson" 
dn: uid=user.5814,ou=People,dc=example,dc=com
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: top
postalAddress: Mollie Jensen$10019 First Street$Hattiesburg, SD 58443
postalCode: 58443
uid: user.5814
description: This is the description for Mollie Jensen.
userPassword: {SSHA}yAKGDBMBn0qTPIksRITI+CxaGem+AH0BzkLYrQ==
employeeNumber: 5814
initials: MHJ
givenName: Mollie
pager: +1 891 013 7026
mobile: +1 139 787 4408
cn: Mollie Jensen
telephoneNumber: +1 880 751 8601
sn: Jensen
street: 10019 First Street
homePhone: +1 016 249 2054
mail: user.5814@example.com
l: Hattiesburg
st: SD

dn: uid=user.5815,ou=People,dc=example,dc=com
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: top
postalAddress: Molly Jensenworth$31942 Fifteenth Street$Portland, NM 00430
postalCode: 00430
uid: user.5815
description: This is the description for Molly Jensenworth.
userPassword: {SSHA}2Bi8fv6syMvpWYqtO+o7sbe67zdereieAaT/gQ==
employeeNumber: 5815
initials: MTJ
givenName: Molly
pager: +1 327 826 0387
mobile: +1 896 260 6042
cn: Molly Jensenworth
telephoneNumber: +1 004 038 6469
sn: Jensenworth
street: 31942 Fifteenth Street
homePhone: +1 044 105 6790
mail: user.5815@example.com
l: Portland
st: NM
# VLV Target Offset: 5816
# VLV Content Count: 10002
It is now clear that with VLV control the client can determine the 
result set, server side sort attribute and the page size. If you are writing 
a LDAP application with a large amount of data the VLV control is the
best option. 

For example, for an Address Book application you can efficiently
provide paged results based on what user inputs in the "Last Name" 
field instead of going through multiple pages with potentially unwanted 
data in it.

In order to get the above example to work, OpenDS needs some 
configuration. 

We need to tell the server how to handle VLV requests before start 
using VLV control in your application(s). First provide VLV control
access to "admin" user VLV Control OID: 2.16.840.1.113730.3.4.9 
./dsconfig -h localhost -p 1389 -D "cn=Directory Manager" -w password 
set-access-control-handler-prop --add global-aci:"(targetcontrol=\\"2.16.840.1.113730.3.4.9\\")(version 3.0; 
acl \\"Allow Simple Paged Results Access\\"; allow(read) 
userdn = \\"ldap:///uid=admin,ou=people,dc=example,dc=com\\";)" -n 
Create VLV control configuration for the search operation. Please note 
that the "Base DN", "Search Scope", "Search Filter" and the 
"Sort Attribute" should match the configuration in order to make 
VLV to work.

Following dsconfig command adds the following VLV configuration.
Search Base: ou=people,dc=example,dc=com
Search Scope: One Level
Search Filter: "objectclass=inetOrgPerson"
Sort Attribute: sn
$ ./dsconfig -h localhost -p 1389 -D "cn=Directory Manager" -w password 
create-local-db-vlv-index --backend-name userRoot --index-name People_Search 
--set sort-order:sn --set scope:single-level --set base-dn:ou=people,dc=example,dc=com 
--set filter:objectclass=inetOrgPerson -n 
After creating the VLV configuration, the index should be generated 
before using the VLV control.
$ ./stop-ds
$ ./rebuild-index -i vlv.People_Search -b dc=example,dc=com 
$ ./start-ds 
If you are reviewing the DSEE (Directory Server Enterprise
 Edition 6.3) and would like to get the above VLV configuration 
workingon the Sun Directory Server 6.3 perform the following 
configuration.

Create the VLV configuration LDIF file "vlv.ldif" with following entry.
dn: cn=getsn,cn=example,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: vlvSearch
cn: getsn
vlvBase: ou=People,dc=example,dc=com
vlvScope: 1
vlvfilter: (objectclass=inetorgperson))
aci: (target="ldap:///cn=getsn,cn=example,cn=ldbm database,cn=plugins,cn=config")(targetattr="\*")
 (version 3.0;acl "Config";allow(read,search,compare)(userdn="ldap:///anyone");)
dn: cn=sortsn,cn=getsn,cn=example,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: vlvIndex
vlvsort: sn
cn: sortsn
Use ldapmodify to add the configuration to the Server.
$ ./ldapmodify -h localhost -p 1389 -D "cn=Directory Manager" -w password -a -f vlv.ldif
Now, generate the VLV index. Assuming that the Directory Server 
is installed under /opt/DSEE63 and the actual Directory Server instance 
is running under /opt/ds directory.
$ cd /opt/DSEE63/ds6/bin
$ ./dsadm stop /opt/ds
$ ./dsadm reindex -l -t sortsn /opt/ds dc=example,dc=com
$ ./dsadm start /opt/ds
All set. Now, you can run the ldapsearch (shipped with OpenDS 
and also that is shipped with DSEE). I am going to provide an 
example using the "ldapsearch" that is shipped with DSEE. You 
can find the "ldapsearch" under /opt/DSEE63/dsrk6/bin directory 
if you have installed using zip distribution.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D 
"uid=admin,ou=people,dc=example,dc=com" -w password -G 0:5:jensen -S sn -x 
"objectclass=inetOrgPerson" dn
version: 1
dn: uid=kjensen, ou=People, dc=example,dc=com
dn: uid=bjensen, ou=People, dc=example,dc=com
dn: uid=gjensen, ou=People, dc=example,dc=com
dn: uid=jjensen, ou=People, dc=example,dc=com
dn: uid=ajensen, ou=People, dc=example,dc=com
dn: uid=bjense2, ou=People, dc=example,dc=com
The Directory Editor which is part of DSEE an 
example of application that uses VLV control for searching 
the Directory Server.

One difference that I have noticed when using Simple Paged 
Results control with ldapsearch, all the entries (10002 in this 
case) are returned for the search filter "objectclass=inetorgperson" 
with 1000 entries per page. However, when the VLV control 
is used in the ldpasearch, I received only first page. I have noticed 
that for some administrators it could be desirable to fetch all the 
entries page wise just like ldpasearch has pulled all entries when 
used with Simple Paged Results. If you are a developer/administrator
 who likes to build tools around LDAP for administration 
purposes to accomplish certain tasks then you can use 
standard LDAP SDK for Java to write any LDAP applications 
in Java.

I have written a Java class using LDAP SDK for Java 
which is part of the DSEE (Directory Server Enterprise Edition), 
can be downloaded from here.

The LDAP SDK for Java can be downloaded separately here.

I have used NetBeans IDE 6.1 to build the jar file.

VLVFetchAll.java - Performs a LDAP search using VLV Control 
and fetches all the entries page wise.

You will need LDAP SDK for Java to compile. Get it from here. 
If you have installed the DSEE then it will be part of DSRK.

VLVFetchAll.jar - Binary. Built using NetBeans IDE 6.1.
Usage: java VLVFetchAll [-h] [-p] [-b] [-D ] [-w ] [-S] [-s] [-P] [-f] 
-h - LDAP Server host. Default: localhost 
-p - LDAP Server port. Default: 389 
-b - Base DN for search. Default: rootDSE 
-D - Bind DN. Default: anonymous
-w - Bind Password. Defualt: blank
-s - Scope fo the search. Default: one
 Allowed values: base,one,sub
-S - Attribute(s) to sort. Default: sn
-f - Search filter. Mandatory option
-A - Space delimited list of attributes. Default: all attributes
-P - VLV pages size. Default: 2000
Example:
java -jar VLVFetchAll -h sundir.example.com -b "ou=people,dc=example,dc=com" -D 
"uid=admin,ou=people,dc=example,dc=com" -w password -f 
"(objectclass=posixAccount)"-S "cn uid" -P 1000
In the above example all the entries are fetched that match the filter (objectClass=posixAccount)
Please note that the VLV configuration should already exist. If the Directory Server is configured 
for LDAP naming services for Unix then the above example would work without any issues
If you are a System Administrator with lots of Perl background, 
then you can use Net::LDAP to write a Perl script that exactly 
does the same as the Java class above.

Comments:

One thing that I cannot figure out yet is how to use the Virtual List Control to get a total record count without doing the iteration.

Is it possible? I can get the total number of records in ldap from it, but if I do a search, I cannot figure out how to get the total number of records in the result set.

Advice?

Posted by John Bailo on October 03, 2010 at 05:33 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Srikanth Konjarla

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today