Saturday Dec 19, 2009

DPS Coherence Plug-In: New Features

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

After having conversations on this plug-in both internally and with a handful of you out there, I sat down and added what I felt was the main missing features in my current implementation. This mainly covers security but also some flexibility. 

Bird's Eye View

On the security front, you now have the ability to filter out attributes so they don't make it in the cache (e.g. userPassword). What's more, you can filter out what entries you don't want in the cache in order to avoid the cache to be polluted by the occasional non-production (e.g. administration) hits.

On the flexibility front, you can set a time to live for entries that do make it in the cache. This allows to control whether you want to retain a value forever in the cache (default) or you want to make sure that it gets evicted after a certain time. You can also provide a list of regular expressions for the Bind DN (the identity) you grant access to the cache. And of course, you can decide to inlcude (default) or exclude unauthenticated clients to access the cache as well.

The Meat 

  • attributeNotToCache: userPassword
  • This attribute in the plugin's configuration entry can be multivalued and is not a regular expression but a plain string. Case, as always in LDAP, matters not. Any attribute name matching on of the provided values will be stripped from the entry before storing in Coherence.

  • dnNotToCache: .\*,ou=secret,o=o
This attribute can multivalued and allows to prevent DNs matching the regular expression to be stored in Coherence.
  • cacheForBindDN: cn=[2-3],o=o
This is attribute can be multivalued. It is a regular expression. Any authenticated clients' Bind DN must match one of the provided regular expressions to be granted access to the contents stored in Coherence.
  • cacheForAnonymous: false
This attribute is single valued. It is a boolean, either true or false. When false, unauthenticated clients will not be granted access to the contents stored in Coeherence and will therefore always hit the back-end.
  • cacheExpirationInMS: 30000

This attribute is single valued. It is a long and represents the length of time in milliseconds that an entry should be kept in the cache after the last time it has been accessed.

So, in the end, here is an example configuration entry:

dn: cn=CoherencePlugin,cn=Plugins,cn=config
objectClass: top
objectClass: configEntry
objectClass: plugin
objectClass: extensibleObject
cn: CoherencePlugin
description: Oracle Coherence Cache Plugin
enabled: true
pluginClassName: com.sun.directory.proxy.extensions.CoherencePlugin
pluginType: preoperation search
pluginType: search result entry
pluginType: postoperation delete
pluginType: postoperation modify
pluginType: postoperation modify dn
cacheName: LDAPCache
attributeNotToCache: userpassword
attributeNotToCache: aci
dnNotToCache: .\*,ou=secret,o=o
dnNotToCache: .\*,cn=config
cacheForBindDN: cn=[2-3],o=o
cacheForBindDN: uid=user.[0-9]+,ou=People,o=o
cacheForAnonymous: false
cacheExpirationInMS: 30000

That's it for my Friday night, let me know if there is more than DPS+Coherence can do for you!

As always, if you want to try this DPS plug-in, ping me: arnaud -at- sun -dot- com

Wednesday Dec 16, 2009

Oracle Coherence Support Right In DPS!!!

Rationale

Why do caching in DPS ?

The Directory Server back-ends are not able to "absorb" as many updates when they're stressed with a large proportion of searches. After all there's already caching on the back-end Directory Server itself. It helps a lot performance, since reads return to the client faster, it relieves some of the stress and frees up resources to take care of the writes that lock resources for longer atomic stretches of time. But as long as searches hit the back-end, even with cache, there's some weight lifting to be done: open the connection, parse the request, put the request in the work queue, lookup entries in the cache, return the entry, close the connection...

That's why caching right in DPS started to look appealing to me.

Why Coherence ?

Well, as much as one may think it is because Sun is about to be gobbled up by Oracle that I made this choice but the answer is no. Coherence is simply a compelling choice, these guys seem to have made all the technical choices I would have ... and then some. For one, you download the bits, then just start it and it works. It may sound like a marketing pitch but see for yourself. Excellent 0 to 60 in my book. Once you have it working, you get acquainted with it, the protocol is dead simple, the API is clean, robust and pretty lean. After that, you check the docs out (that's right, I read the docs after the fact) and start to realize how powerful a platform it is, how expandable it is, how far you can push deployments to accommodate growing performance or reliability needs. 

Bird's Eye View

The integration with Coherence is done by way of DPS (7+) plug-in that will -asynchronously- populate a Coherence cache with entries being returned by your regular traffic. When requests come in, a lookup is done to check if the entry is present in the cache and returned immediately if it is the case, otherwise the request is routed to the back-end as usual.

Note that I'm not making any claims on the performance aspect from the client's perspective of this caching approach because our Directory Server back-end is already pretty darn fast. This for sure relieves it from a bit of "frequent" traffic and will certainly benefit the overall performance of the topology. The relief will most certainly result in improved write response times but nothing speaks to the performance of the Coherence cache lookup. I just haven't collected enough data so far.

The Meat

nitty gritty anyone ?

Suppose we have a setup like this ... 

 the first read would be processed as described below:


but the second one would be processed without hitting the back-end.

Understand the tremendous impact this will have on your back-end ?

It frees it up to process writes or some heavy weight searches...

How to get the plug-in? ask nicely. 

Wednesday Dec 02, 2009

A Script To Rule Them All ... clients and backends that is

<script type="text/freezescript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/freezescript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

With Directory Proxy Server, regardless of the version, investigating traffic can get:

  • a) real tricky
  • b) time consuming
  • c) confusing
  • d) all of the above

and the answer is ... rolling drum ... d) !

So here is a script that you can feed your DPS access log to. It will output a CSV file that you can then load in your favorite spreadsheet software or just graph with tools like gnuplot and the like... it just will make your life easy...er.

Bird's Eye View

Disclaimer: It's not as anywhere as clever as logconv.pl for Directory Server, it only munches the data so that YOU can more easily spot issues or identify patterns. So what does this script produce in the end ?

It will take your DPS 6.x/7.0 access log in and output three csv files, one with the transaction volumes (suffixed "tx"), one with the average response times (suffixed "avg") and finally one with the maximum response time over a minute (suffixed "max"). Why not all in one file? I did initially but in a csv it turned out to really not be practical. So at least when you open up one of these files you know what you're looking at.

The Meat

Pre-requisites

Since I really started this initially to simply be able to "grep" a file on a windows system, I really had no plan and no idea it would end up in a tool like this. All that to say that I wrote in python instead of our customary Java tools. At least it has the merit of existing so you don't have to start from scratch. So you'll need python, at least 2.4. If you're on Solaris or Linux, you're covered. If on windows, simply download your favorite python, I have installed the 2.6.4 windows version from here.

You will also need to download the script. You may as well get the netbeans project if you'd like to change it to adapt it to your specific needs or help improve on it.

How Does It Work

0 To 60 In No Time

python dps-log-cruncher.py access 

The Rest Of The Way

-c      : break up statistics per client
-s      : break up statistics per back-end server
-f hh:mm: start parsing at a given point in time
-t hh:mm: stop parsing after a given point in time
-h      : print this help message
-v      : print tool version

Some examples:

split the output per client for all clients:

python dps-log-cruncher.py -c \* access 

 split the output per back-end server for client 192.168.0.17:

python dps-log-cruncher.py -c 192.168.0.17 -s \* access 

 split the output for all clients, all servers:

python dps-log-cruncher.py -c \* -s \* access 

 only output results from 6pm (18:00) to 6:10pm (18:10):

python dps-log-cruncher.py -f 18:00 -t 18:10 access 

 output results between 6:00pm (18:00) to 6:10pm (18:10) and split results for all clients and back-end servers:

python dps-log-cruncher.py -f 18:00 -t 18:10 -c \* -s \* access 

Enhancements

This is a list to manage expectations as much as it is one for me to remember to implement:

  1. Selectable time granularity resolution. Currently, all data is aggregated per minute. In some case, it would be useful to be able to see what happens per second
  2. Improve error handling for parameters on the CLI.
  3. Add a built-in graphing capability to avoid having to resort to using a spreadsheet. Spreadsheets do however give a lot of flexibility
  4. Add the ability to filter / split results per bind DN
  5. Output the response time distribution

Support

Best effort is how I will label it for now, you can send your questions and support requests to arnaud -- at -- sun -- dot -- com.

Enjoy!

Tuesday Aug 18, 2009

Dissipating Performance Misconceptions About DPS

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

Recently engaged on a project where a Directory Server farm capable of  delivering 100,000 searches per second , I was faced with skepticism about the ability for DPS to keep up in such a demanding environment. I had to prove ten times over that our proposed architecture would fit the bill and quantify how much headroom it would provide. Here is some of the things we observed along the way.

Bird's Eye View

 In this article I will quickly talk about a theoretical performance of DPS under ideal circumstances, you'll see that the setup is quite far from a real deployment from the hardware to the way the stack is set up.

The Meat

The results

I won't make you read through the whole thing if you do not want to so here goes: 

Throughput

DPS Throughput

Response Time

DPS Response Time

Setup

The box  I'm developing on and running tests these days is somewhat unusual and the proper disclaimer is necessary: it is a corei7 (975 EE) very slightly overclocked to 3.9GHz fitted with 12GB of 1600MHz DDR3 RAM. It runs OpenSolaris 2009.06 with ZFS. There is a 10,000 rpm 320GB drive and an Intel Extreme 32GB SSD. All in all a nice rig but not a server either. The closest server if I had to give a comparable would be a beefed up Sun Fire X2270. Keep in mind that my desktop is single socket but about twice faster on the CPU clock rate.

To load this, I use SLAMD. One server, 7 clients (2 Sun Fire T5120, 4 Sun Fire X2100, 1 Sun Fire X4150). For the record, this client set has generated north of 300,000 operations per second in other tests I ran in the past. Clients in this test campaign were always loaded at less than 15% CPU, so results are not skewed by CPU pressure on the client side.

On the software side (middleware I should say) we have an Sun Ultra 27 running OpenDS with a null back-end  and my rig running running my workspace of DPS 7 plus my little sauce...

Why a null back-end on OpenDS? that is not representative of real life!

Well, quite true, but the whole point in case here is to push DPS to the limit, not the backend, not the network or any other component of the system. So a null back-end is quite helpful here as it simulates a perfect back end that responds immediately to any requests DPS sends. Quite handy if you come to think of it because what you have as a result is an overview of the overhead of introducing DPS between your clients and your servers under heavy load. The load is actually all the hardware I had could take, the CPU is almost completely used, with idle time varying frantically between 1% and 7%. Keep in mind as well that DPS runs in a JVM and at these rate, garbage collections are almost constant. 

Here is how the set up looks like:

That's all I had for you guys tonight! 

Friday Aug 07, 2009

Taking Throttling To The Next Level

Rationale

After releasing the first version of the throughput throttling, most customers seemed interested in at least kicking the tires and wanted to evaluate it but as it turned out, the fact that throttling was choking traffic across the board, it would have actually required to deploy an extra instance of DPS in the topology for the sole purpose of choking traffic to an acceptable level to the business. While some felt it was simple enough to do, some found it to be a show stopper and therefore, I wrote this new plug-in, leveraging the distribution algorithm API to allow to narrow the scope of traffic throttling per data view, bringing a whole new dimension of flexibility to this feature.

Bird's Eye View

This new wad not only provides a new, more flexible throttling facility to your LDAP Proxy Server, it also comes with a CLI that makes it trivial to configure and change your settings on the fly. The README has some instructions to get you going in no time, but I will provide a quick overview here as well.

The Meat

First things first, you need to unzip the package, which will give your the following files:

$ find Throttling
Throttling
Throttling/throttleadm
Throttling/Throttling.jar
Throttling/README

As you can see, pretty trim.

The CLI will ease mainly 3 things:

  1. Setting up data views to be able to throttle traffic (throttleadm configure)
  2. Configuring the throughput limits to enforce (per data view and operation type, e.g.: throttleadm throttle root\\ data\\ view mod 200 )
  3. Checking the current configuration (throttleadm list)

Here is the complete usage for this little tool

$ ./throttleadm --help
Checking for binaries presence in path
.
.
.
.
throttleadm usage:
  throttleadm list
  throttleadm configure <dataViewName>
  throttleadm unconfigure <dataViewName>
  throttleadm choke <dataViewName>
  throttleadm unchoke <dataViewName>
  throttleadm throttle <dataViewName> <operation> <rate>
for example:
  To list the data views configured for throttling
  throttleadm list

  To set up a data view to use throttling
  throttleadm configure root\\ data\\ view

  To set up a data view back to its original state
  throttleadm unconfigure root\\ data\\ view

  To enable throttling on the default data view (provided the data view has been properly configured)
  throttleadm choke root\\ data\\ view

  To disable throttling on the default data view
  throttleadm unchoke root\\ data\\ view

  To change or set the maximum search throughput on the default data view to 20
  throttleadm throttle root\\ data\\ view search 20

Finally, when you change the settings, you can see them be enforced right away. In the example below, I initially have set the bind throughput limit to 200 per second. The left window has an authrate running and in the right window, while the authrate is running, I lower the throughput limit to 100 for 4 seconds and then set it back to 200. See how that works below:

Finally, here is a quick snap of the output of the CLI for the throttling status.

 $ ./throttleadm -D cn=dpsadmin -w /path/to/password-file list
Checking for binaries presence in path
Will now take action list with the following environmentals:
-host=localhost -port=7777 -user=cn=dpsadmin -password file=/path/to/password-file
VIEW NAME          - THROTTLED -  ADD - BIND -  CMP -  DEL -  MOD - MDN - SRCH
ds-view            -      true -   12 -  200 -   13 -   14 -    1 -  16 -  112

 That's it !

As usual, don't hesitate to come forth with questions, remarks, requests for enhancements.


<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Monday Jun 15, 2009

Operation Throttling - Protect Your LDAP Servers

Rationale

    Many times - for various maintenance and operational reasons - we need to run batches of updates to an Identity repository. Whether it is a new application that was introduced requiring new attributes or a broad sweep cleanup for a retired application, the net result is a same: an additional write load is inflicted to the LDAP farm with the ever undesirable performance impact on the "regular" traffic. As a work around, this used to be done during maintenance windows, at night or over a quiet week end... this usually leads to stressful early Monday mornings if you had overestimated the absorption capacity of the infrastructure.

Bird's Eye View

    The idea is to allow DPS to throttle traffic in order to be able to "choke" traffic coming from a particular user or host. This would allow to leave the regular traffic alone and only apply the limitation on writes coming from the user running the batch job for example.

The Meat

 The principle is pretty straightforward, traffic fills a queue until the queue is full. When it is, DPS delays the next requests until the next slot becomes available in the queue. This is effective as it does not disrupt traffic. It only makes the LDAP infrastructure appear slower to clients. Most throttling solutions I have seen out there would return "Server Busy" or something along those lines, which may cause errors on the client side and defeats the purpose of throttling altogether from a client's perspective. It works only from the server's perspective, which indeed see their traffic decreased.
With this plug-in, all the requests sent by the client will be honored, it'll just take longer.

One of the added benefits is that the throughput limit can be changed on the fly without disturbing regular "unthrottled" traffic.

So you for example could leave the batch job completely unleashed and flood your LDAP farm over the week-end and then strangle the traffic Monday at 4:00am to an acceptable trickle. Since the configuration of DPS can be altered over LDAP, all there is to it is an entry in your cron, and you have yourself a nicely controlled environment...

This plug-in for DPS is available through Directory Integration Team via Sun Professional Services (or shoot me an email, arnaud@sun.com)

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Performance Impact Of Toggling Access Log

Rationale

    Many a customer need to have consistent high performance over traceability but most want both. As part of our current effort to rewrite the logging subsystem, we took a couple of quick metrics to assess how much headroom we had. This may answer some of the questions you asked yourself in the past.

Bird's Eye View

    I'll make this quick:

Throughput comparison

    This was just a crude test on a Sun Fire X4150. This is in no way a benchmark of DPS for maximum throughput or optimal response time. The only intent of this article is to illustrate the performance difference the access log makes for DPS.

    How do I know if I hit the access log bottleneck ?

Mainly, your CPU utilization is less than 100% (sometimes far less) and you have reached a maximum of throughput, throwing any more load to DPS only results in longer response times... then there is a strong likelihood that turning off the access log will allow to squeeze the extra drops of performance.

The Meat

    I won't go into details about the ins and outs of logging but simply try to articulate the challenge it poses under stress. Particularly, if you think about it, as you have seen in the first graph above, we may process about 10,000 operations per second. For each operation, we may need to log a number of steps, among which:

  • the time the connection was established
  • the time the connection was processed by the default connection handler
  • the time the operation was handed off to the adequate connection handler
  • the time the operation was sent to the back end server
  • the time the operation came back from the back end server
  • the time the operation was sent back to the client

For some back-ends or if custom plug-ins are configured, we may have more.

 We may easily have 5 times the number of writes to the log compared to the number of LDAP operations we actually process. Now imagine that we take a lock anywhere in the process and you will immediately understand how this can become a bottleneck under heavy LDAP load. We are currently in the process of removing this lock and we inspired ourselves from the approach we used in OpenDS. OpenDS has a very efficient implementation, costing only  between 6 and 8% of decrease compared to having no access log. The main goal of this reimplementation is to remove the performance bottleneck (some may call it a glass ceiling...) and introduce as little jitter in response time as possible.

Here's another snapshot of the state of affairs with DPS 6.x ...

 I hope this helped shed some light.


<script type="text/freezescript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/freezescript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Friday Apr 03, 2009

Setting DPS As Replication Hub - Part 2: Replication to SQL and LDAP

Rationale

So you need to maintain some data in both LDAP and an SQL database?

As we've seen in part 1 of this tutorial, the replication distribution algorithm allows to duplicate write traffic between data views. Thanks to DPS support of both LDAP data views and JDBC data views, we can do the same as in  part 1 but use one SQL data base in place of "store B". In this example, I will use MySQL but this works on IBM DB2, Oracle, hsql or any other data base with a jdbc driver.

Bird's Eye View

The Meat

  1. Configure the LDAP Store A. See here
  2. Configure the SQL Store B. See here
  3. Configure the Replication Distribution Algorithm Between Them
If like me you are lazy or unwilling to jump to another page, the whole procedure is also described below:

Store A Back End Setup: Directory Server

$ echo password > /tmp/pwd
$ dsadm create -p 1389 -P 1636 -D cn=dsadmin -w /tmp/pwd ds
Use 'dsadm start 'ds'' to start the instance
$ dsadm start ds
Directory Server instance '/path/to/sun/dsee/6.3/replication2/ds' started: pid=1836
$ dsconf create-suffix -D cn=dsadmin dc=example,dc=com
$ cat admin.ldif
dn: cn=admin,dc=example,dc=com
objectClass: person
cn: admin
sn: administrator
userPassword: password
$ ldapadd -p 1389 -D cn=dsadmin -w password  < admin.ldif
adding new entry cn=admin,dc=example,dc=com
arnaud@nullpointerexception:/path/to/sun/dsee/6.3/replication2$  ldapmodify -p 1389 -D cn=dsadmin -w password
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr=\*) (version 3.0; acl "allow all";allow(all) userdn="ldap:///anyone";)

modifying entry dc=example,dc=com

\^C

 Store A Configuration In DPS


$ dpconf create-ldap-data-source sourceA localhost:1389
$ dpconf create-ldap-data-source-pool poolA
$ dpconf attach-ldap-data-source poolA sourceA
$ dpconf set-attached-ldap-data-source-prop poolA sourceA add-weight:1 bind-weight:1 delete-weight:1
$ dpconf set-attached-ldap-data-source-prop poolA sourceA add-weight:1 bind-weight:1 delete-weight:1 modify-weight:1 search-weight:1
$ dpconf create-ldap-data-view viewA poolA dc=example,dc=com

 Store B Configuration In DPS: MySQL

For this example I have assumed that we already had a running instance of DPS with a data base named "replication" that contains a single table "users" with a single row of data. This row is the admin user entry.

$ dpconf create-jdbc-data-source -b replication -B jdbc:mysql:/// -J file:/path/to/apps/mysql-connector-java-5.1.6/mysql-connector-java-5.1.6-bin.jar -S com.mysql.jdbc.Driver sourceB
$ dpconf set-jdbc-data-source-prop sourceB db-user:root db-pwd-file:/tmp/pwd
The proxy server will need to be restarted in order for the changes to take effect
$ dpadm restart dps
Directory Proxy Server instance '/path/to/sun/dsee/6.3/replication2/dps' stopped
Directory Proxy Server instance '/path/to/sun/dsee/6.3/replication2/dps' started: pid=2020
$ dpconf create-jdbc-data-source-pool poolB
$ dpconf attach-jdbc-data-source poolB sourceB
$ dpconf create-jdbc-data-view viewB poolB dc=example,dc=com
$ dpconf create-jdbc-table dpsUsersTable users
$ dpconf add-jdbc-attr dpsUsersTable sn id
$ dpconf add-jdbc-attr dpsUsersTable cn name
$ dpconf add-jdbc-attr dpsUsersTable userPassword password
$ dpconf create-jdbc-object-class viewB person dpsUsersTable cn
$ dpconf set-jdbc-attr-prop dpsUsersTable sn sql-syntax:INT
$ ldapmodify -p 7777 -D cn=dpsadmin -w password
dn: cn=permissive_aci,cn=virtual access controls
changetype: add
objectClass: aciSource
dpsAci: (targetAttr="\*") (version 3.0;acl "Be lenient";allow(all) userdn="ldap:///anyone";)
cn: permissive_aci

adding new entry cn=permissive_aci,cn=virtual access controls

$ dpconf set-connection-handler-prop "default connection handler" aci-source:permissive_aci

Replication Configuration Between Directory Server And MySQL 

$ dpconf set-ldap-data-view-prop viewA distribution-algorithm:replication replication-role:master
The proxy server will need to be restarted in order for the changes to take effect
$ dpconf set-jdbc-data-view-prop viewB distribution-algorithm:replication replication-role:master
The proxy server will need to be restarted in order for the changes to take effect
$ ldapmodify -p 7777 -D cn=dpsadmin -w password
dn: cn=viewA,cn=data views,cn=config
changetype: modify
add: distributionDataViewType
distributionDataViewType: read

modifying entry cn=viewA,cn=data views,cn=config

\^C

$ dpadm restart dps
Directory Proxy Server instance '/path/to/sun/dsee/6.3/replication2/dps' stopped
Directory Proxy Server instance '/path/to/sun/dsee/6.3/replication2/dps' started: pid=2258

Testing Replication To Both Data Stores 

$ cat add.ldif
dn: cn=user,dc=example,dc=com
objectClass: person
cn: user
sn: 1
userPassword: password

$ ldapadd -p 7777 -D cn=admin,dc=example,dc=com -w password < add.ldif
adding new entry cn=user,dc=example,dc=com

$ ldapsearch -p 1389 -b dc=example,dc=com "(cn=user)"
version: 1
dn: cn=user,dc=example,dc=com
objectClass: person
objectClass: top
cn: user
sn: 1
userPassword: {SSHA}6knZSKvWHj5LKwZ5jUmyYVqxQAQKFRd0rziYXA==

$ /usr/mysql/bin/mysql
Welcome to the MySQL monitor.  Commands end with ; or \\g.
Your MySQL connection id is 57
Server version: 5.0.45 Source distribution

Type 'help;' or '\\h' for help. Type '\\c' to clear the buffer.

mysql> use replication;
Database changed
mysql> select \* from users;
+------+-------+----------+
| id   | name  | password |
+------+-------+----------+
|    0 | admin | password |
|    1 | user  | password |
+------+-------+----------+
2 rows in set (0.00 sec)

mysql> 

Caveats

    You need to remember a couple of important things:

  • Authenticate as a user present in both data stores. "cn=Directory Manager" is not going to work for multiple reasons that I won't describe in detail here but heterogeneous data stores come with some constraints.
  • Make sure the user has the proper rights to manipulate data on both data stores.
<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

The Stupid Simple DPS MySQL Example

Rationale

    Even though I use DPS every day, I find myself looking for tips quite frequently.
Here is just a REALLY simple example of how to get started with MySQL as a data store.

There are very good, detailed examples in the documentation but none really dead simple. And that's precisely what this entry aims at.

Bird's Eye View

Below is a graph trying to depict how we map in DPS from SQL to LDAP. To be honest, SQL is a quite radically different model and therefore, even in this "stupid simple" example, there are a number of things that DPS cannot guess, namely:

  1. The data source is configured to point to a specific database (through the jdbc url)
  2. The data view is configured to represent an LDAP objectClass from an SQL table
  3. Each column of the SQL table need to be mapped to an LDAP attribute

 Here's how all this looks from a DPS configuration stand point:


The Meat

   In this example, we have a database engine containing a single database named "DEADSIMPLE". The DEADSIMPLE database has a single table "USERS" with two columns, "NAME" and "PASSWORD". The "USERS" table content is a single row as described in the above figure. This is all to make it as small and as easy as possible.

    We will here want to expose this data from the MySQL database as a proper "person" object, containing a "cn" (common name) attribute, "sn" (surname) attribute and a "userPassword" attribute in order for us to be able to authenticate as user cn=admin,dc=example,dc=com with password "password". Eventually, we want the entry to look as follows:

dn: cn=admin,dc=example,dc=com
objectclass: top
objectclass: person
userpassword: password
sn: 0
cn: admin

And here the log of my session. I'll upate this article later with more details.

$ echo password > /tmp/pwd
$ dpadm create -p 7777 -P 7778 -D cn=dpsadmin -w /tmp/pwd dps
Use 'dpadm start /path/to/sun/dsee/6.3/dps' to start the instance
$ dpadm start dps
Directory Proxy Server instance '/path/to/sun/dsee/6.3/dps' started: pid=966
$ dpconf create-jdbc-data-source -b replication -B jdbc:mysql:/// -J file:/path/to/mysql-connector-java-5.1.6-bin.jar -S com.mysql.jdbc.Driver sourceA
$ dpconf set-jdbc-data-source-prop sourceA db-user:root db-pwd-file:/tmp/pwd
The proxy server will need to be restarted in order for the changes to take effect
$ dpadm restart dps
Directory Proxy Server instance '/path/to/sun/dsee/6.3/dps' stopped
Directory Proxy Server instance '/path/to/sun/dsee/6.3/dps' started: pid=1065

$ dpconf create-jdbc-data-source-pool poolA
$ dpconf attach-jdbc-data-source poolA sourceA
$ dpconf create-jdbc-data-view viewA poolA dc=example,dc=com
$ dpconf create-jdbc-table dpsUsersTable users
$ dpconf add-jdbc-attr dpsUsersTable sn id
$ dpconf add-jdbc-attr dpsUsersTable cn name
$ dpconf add-jdbc-attr dpsUsersTable userPassword password
$ dpconf create-jdbc-object-class viewA person dpsUsersTable cn
$ldapsearch -p 7777 -D cn=admin,dc=example,dc=com -w password -b dc=example,dc=com "(objectClass=\*)"
version: 1
dn: dc=example,dc=com
objectclass: top
objectclass: extensibleObject
description: Glue entry automatically generated
dc: example

dn: cn=admin,dc=example,dc=com
objectclass: top
objectclass: person
userpassword: password
sn: 0
cn: admin


$ dpconf set-jdbc-attr-prop dpsUsersTable sn sql-syntax:INT

$ cat add.ldif
dn: cn=user,dc=example,dc=com
objectClass: person
cn: user
sn: 1
userPassword: password

$ ldapadd -p 7777 -D cn=admin,dc=example,dc=com -w password < add.ldif
adding new entry cn=user,dc=example,dc=com
ldap_add: Insufficient access
ldap_add: additional info: No aciSource setup in connection handler "default connection handler"


$ ldapmodify -p 7777 -D cn=dpsadmin -w password
dn: cn=mysql_aci,cn=virtual access controls
changetype: add
objectClass: aciSource
dpsAci: (targetAttr="\*") (version 3.0; acl "Allow everything for MySQL"; allow(all) userdn="ldap:///anyone";)
cn: mysql_aci

adding new entry cn=mysql_aci,cn=virtual access controls

$ dpconf set-connection-handler-prop "default connection handler" aci-source:mysql_aci

$ ldapadd -p 7777 -D cn=admin,dc=example,dc=com -w password < add.ldif
adding new entry cn=user,dc=example,dc=com

$ ldapsearch -p 7777 -D cn=admin,dc=example,dc=com -w password -b dc=example,dc=com "(objectClass=\*)"
version: 1
dn: dc=example,dc=com
objectclass: top
objectclass: extensibleObject
description: Glue entry automatically generated
dc: example

dn: cn=admin,dc=example,dc=com
objectclass: top
objectclass: person
userpassword: password
sn: 0
cn: admin

dn: cn=user,dc=example,dc=com
objectclass: top
objectclass: person
userpassword: password
sn: 1
cn: user

$ ldapmodify -p 7777 -D cn=admin,dc=example,dc=com -w password
dn: cn=user,dc=example,dc=com
changetype: modify
replace: userPassword
userPassword: newPassword

modifying entry cn=user,dc=example,dc=com

\^C
$ ldapsearch -p 7777 -D cn=admin,dc=example,dc=com -w password -b dc=example,dc=com "(cn=user)"version: 1
dn: cn=user,dc=example,dc=com
objectclass: top
objectclass: person
userpassword: newPassword
sn: 1
cn: user

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Wednesday Apr 01, 2009

Setting DPS As Replication Hub - Part 1: a simple tut'

Rationale

    There may be cases where you would like to keep two environments up to date with the same data but there is no replication or synchronization solution that fit your particular needs. One example that comes to mind is to migrate away from a legacy LDAP (RACF, OiD, Sun DS 5...) to  OpenDS. After having initialized your new data store with the former data store contents, without synchronization mechanism, you would have to switch to the new data store right away. That would not quite be acceptable in production because for one thing, importing the data might take longer than the maintenance window, and more importantly, may something unexpected happen, all real-life deployments want to preserve the option of rolling back to the legacy system (that has proved to work in the past -even if performance or functionality could use a dust-off- ).

Enters DPS "replication" distribution algorithm. The idea is quite simple: route reads to a single data store, duplicate writes across all data stores. I use the term data store here because it needs not be LDAP only but any SQL data base that has a JDBC driver can be replicated to as well. For this tutorial though, I will use two LDAP stores. We will see a MySQL example in Part 2.

Bird's Eye View

    Unlike load balancing and fail over algorithm, which work across sources in a same pool, distribution algorithms work across data views. A distribution algorithm is a way to pick the appropriate data view among eligible data views to process a given client request. In this tutorial, I will show how the "replication" distribution algorithm allows to duplicate write traffic across two distinct data sources.

In the graph below, you can see how this is structured in DPS configuration.

The Meat

We will assume here that we have two existing LDAP servers running locally and serving the same suffix dc=example,dc=com:

  1. Store A: dsA on port 1389
  2. Store B: dsB on port 2389

Let's first go about the mundane task of setting up both stores in DPS:
    For Store A:

#dpconf create-ldap-data-source dsA localhost:1389
#dpconf create-ldap-data-source-pool poolA
#dpconf attach-ldap-data-source poolA dsA
#dpconf set-attached-ldap-data-source-prop poolA dsA add-weight:1 bind-weight:1 delete-weight:1 modify-weight:1 search-weight:1
#dpconf create-ldap-data-view viewA poolA dc=example,dc=com

    For Store B:

#dpconf create-ldap-data-source dsB localhost:2389
#dpconf create-ldap-data-source-pool poolB
#dpconf attach-ldap-data-source poolB dsB
#dpconf set-attached-ldap-data-source-prop poolB dsB add-weight:1 bind-weight:1 delete-weight:1 modify-weight:1 search-weight:1
#dpconf create-ldap-data-view viewB poolB dc=example,dc=com

    Now, the distribution algorithm must be set to replication on both data views:

#dpconf set-ldap-data-view-prop viewA distribution-algorithm:replication replication-role:master
#dpconf set-ldap-data-view-prop viewB distribution-algorithm:replication replication-role:master

  And finally, the catch:

    When using dpconf to set the replication-role property to master, it effectively writes distributionDataViewType as a single valued attribute in the data view configuration entry when in reality the schema allows it to be multi-valued. To see that for yourself, simply do:

#ldapsearch -p <your DPS port> -D "cn=proxy manager" -w password "(cn=viewA)"
version: 1
dn: cn=viewA,cn=data views,cn=config
dataSourcePool: poolA
viewBase: dc=example,dc=com
objectClass: top
objectClass: configEntry
objectClass: dataView
objectClass: ldapDataView
cn: viewA
viewAlternateSearchBase: ""
viewAlternateSearchBase: "dc=com"
distributionDataViewType: write
distributionAlgorithm: com.sun.directory.proxy.extensions.ReplicationDistributionAlgoritm


and then try to issue the following command:

#dpconf set-ldap-data-view-prop viewA replication-role+:consumer
The property "replication-role" cannot contain multiple values.
XXX exception-syntax-prop-add-val-invalid

...

...or just take my word for it. 

The issue is that in order for DPS to process read traffic (bind, search, etc...), one data view needs to be consumer but for the replication to work across data views, all of them must be master as well. That is why you will need to issue the following command on one (and one only) data view:

#ldapmodify -p <your DPS port> -D "cn=proxy manager" -w password
dn: cn=viewA,cn=data views,cn=config
changetype: modify
add: distributionDataViewType
distributionDataViewType: read

That's it!
Wasn't all that hard except it took some insider's knowledge, and now you have it.
Your search traffic will always go to Store A and all write traffic will get duplicated across Store A and B.

Caveats

Note that while this is very useful in a number of situations where nothing else will work, this should only be used for transitions as there are a number of caveats.
DPS does not store any historical information about traffic and therefore, in case of an outage of one of the underlying stores, contents may diverge on data stores. Especially when this mode is used where no synchronization solution can catch up after an outage.

Store A and Store B will end up out of synch if:

  • either store comes to be off-line
  • either store is unwilling to perform because the machine is  outpaced by traffic
<script type="text/freezescript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/freezescript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Wednesday Aug 06, 2008

From An Old Suffix To A New One Live With No Down Time

You currently have your entries an "old" suffix that we will here on call dc=old and you would like to be able to move your entries to a new suffix we will refer to as dc=new for the purpose of this article. The catch is that you cannot stop your server and migrate your data off line. On top of this, during the transition period when your client applications get reconfigured to use dc=new, you need entries to appear to be in both dc=new and  dc=old.

To make this a little simpler to picture in our minds, let's look at the data life cycle:

  1. Before the migration starts, all data resides under dc=old, requests on dc=new would fail but no application is "aware" of dc=new
    • e.g. cn=user.1,dc=old works but cn=user.1,dc=new fails. At this point the DIT looks like this:
      • dc=old
        • cn=user.1
        • cn=user.3
      • dc=new
  2. When the migration starts, all data resides under dc=old, both requests on dc=old and dc=new are honored
    • e.g. cn=user.1,dc=old and cn=user.1,dc=new will work but the entry is actually only stored physically under dc=old. The DIT is unchanged compared to 1. but we have shoehorned DPS in the topology and added a data view to take care of the transformation.
  3. While the migration is ongoing, to be migrated data resides under dc=old but migrated data resides on dc=new
    • this is when it will start to get a little complicated. Here is what the DIT might look like:
      • dc=old
        • cn=user.1
      • dc=new
        • cn=user.3
    • At this point, a request on cn=user.1,dc=old will work along with a request for cn=user.1,dc=new. But both requests for cn=user.3,dc=new and cn=user.3,dc=old will work as well. Basically, to the client application there is a true virtualization of where the data actually resides. This is crucial when you have a heterogeneous environment of applications being reconfigured to use the new suffix while some older applications might take too much work to reconfigure and are simply waiting to be discontinued.
  4. When data migration is complete, all data resides under dc=new but both requests for entries under both suffixes will be served. At this point our DIT would look like this:
    • dc=old
    • dc=new
      • cn=user.1
      • cn=user.3
  5. Once we are confident no application requests entries under dc=old then we can take the virtualization data views down. Only requests to dc=new will be served.

If you want to try it out for yourself, here are the steps to follow to get such a setup with DPS 6 but first let us agree on the environment:

  • Directory Server bits are installed in/path/to/sun/dsee/6.3/bits/ds6
  • Directory Proxy Server bits installed in /path/to/sun/dsee/6.3/bits/dps6
  • I won't use "cn=Directory Manager" but uid=admin instead with a password file containing the "password" string in /path/to/pwd
  • the data for dc=new and dc=old in our example is as follows

    dn: dc=old
    dc: old
    objectClass: top
    objectClass: domain
    aci: (targetattr=\*) ( version 3.0; acl "allow all anonymous"; allow (all) userdn="ldap:///anyone";)

    dn: cn=user.1,dc=old
    objectClass: person
    objectClass: top
    cn: user.1
    sn: 1
    userPassword: {SSHA}PzAK73RDZIikdI8qRqD7MYubasZ5JyJa/BToMw==

    dn: dc=new
    dc: new

    objectClass: top
    objectClass: domain
    aci: (targetattr=\*) ( version 3.0; acl "allow all anonymous"; allow (all) userdn="ldap:///anyone";)

    dn: cn=user.3,dc=new
    objectClass: person
    objectClass: top
    cn: user.3
    sn: 3
    userPassword: {SSHA}PzAK73RDZIikdI8qRqD7MYubasZ5JyJa/BToMw==

Before we begin, it is very convenient to set the following environment variables to make subsequent calls the CLIs much easier to read:

export PATH=${PATH}:/path/to/sun/dsee/6.3/bits/ds6/bin:/path/to/sun/dsee/6.3/bits/dps6/bin:/path/to/sun/dsee/6.3/bits/dsrk6/bin
export LDAP_ADMIN_PWF=/path/to/pwd
export LDAP_ADMIN_USER=uid=admin
export DIRSERV_PORT=1389
export DIRSERV_HOST=localhost
export DIRSERV_UNSECURED=TRUE
export DIR_PROXY_HOST=localhost
export DIR_PROXY_PORT=7777
export DIR_PROXY_UNSECURED=TRUE

First off we need to create an instance of Directory Server to store our entries. This is a two step process:

  1. Create and start the instance that we will name master
    >dsadm create -D uid=admin -w /path/to/pwd -p 1389 -P 1636 master
    Use 'dsadm start 'master'' to start the instance
    >dsadm start master
    Directory Server instance '/path/to/sun/dsee/6.3/live-migration/master' started: pid=2968
  2. On to creating a suffix and populating it with data
    >dsconf create-suffix dc=old
    >dsconf import /path/to/sun/dsee/6.3/instances/dc\\=old.ldif dc=old
    New data will override existing data of the suffix "dc=old".
    Initialization will have to be performed on replicated suffixes.
    Do you want to continue [y/n] ?  y
    ## Index buffering enabled with bucket size 40
    ## Beginning import job...
    ## Processing file "/path/to/sun/dsee/6.3/instances/dc=old.ldif"
    ## Finished scanning file "/path/to/sun/dsee/6.3/instances/dc=old.ldif" (3 entries)
    ## Workers finished; cleaning up...
    ## Workers cleaned up.
    ## Cleaning up producer thread...
    ## Indexing complete.
    ## Starting numsubordinates attribute generation. This may take a while, please wait for further activity reports.
    ## Numsubordinates attribute generation complete. Flushing caches...
    ## Closing files...
    ## Import complete.  Processed 3 entries in 4 seconds. (0.75 entries/sec)
    Task completed (slapd exit code: 0).
  3. We can now check the data was successfully loaded with a quick broad sweep search:
    >ldapsearch -p 1389 -b "dc=old" "(objectClass=\*)"
    version: 1
    dn: dc=old
    dc: old
    objectClass: top
    objectClass: domain

    dn: cn=user.1,dc=old
    objectClass: person
    objectClass: top
    cn: user.1
    sn: 1
    userPassword: {SSHA}PzAK73RDZIikdI8qRqD7MYubasZ5JyJa/BToMw==
  4. Repeat these last 3 steps for dc=new

Directory Proxy Server configuration

  1. Create and start an instance of the proxy

    >dpadm create -p 7777 -P7778 -D uid=admin -w /path/to/pwd proxy
    Use 'dpadm start /path/to/sun/dsee/6.3/live-migration/proxy' to start the instance

    >dpadm start proxy
    Directory Proxy Server instance '/path/to/sun/dsee/6.3/live-migration/proxy' started: pid=3061

  2. Connect the proxy to the Directory Server instance

    >dpconf create-ldap-data-source master localhost:1389

    >dpconf create-ldap-data-source-pool master-pool

    >dpconf attach-ldap-data-source master-pool master

    >dpconf set-attached-ldap-data-source-prop master-pool master add-weight:1 bind-weight:1 compare-weight:1 delete-weight:1 modify-dn-weight:1 modify-weight:1 search-weight:1

  3. Create a straight data view to dc=old and verify we can get through to the source

    >dpconf create-ldap-data-view actual-old master-pool dc=old

    >ldapsearch -p 7777 -b dc=old "(objectClass=\*)"
    version: 1
    dn: dc=old
    dc: old
    objectClass: top
    objectClass: domain

    dn: cn=user.1,dc=old
    objectClass: person
    objectClass: top
    cn: user.1
    sn: 1
    userPassword: {SSHA}PzAK73RDZIikdI8qRqD7MYubasZ5JyJa/BToMw==

    >ldapsearch -p 7777 -b dc=old "(cn=user.1)"
    version: 1
    dn: cn=user.1,dc=old
    objectClass: person
    objectClass: top
    cn: user.1
    sn: 1
    userPassword: {SSHA}PzAK73RDZIikdI8qRqD7MYubasZ5JyJa/BToMw==

  4. Create a virtual data view representing the physical entries under dc=old as it were under dc=new

    >dpconf create-ldap-data-view virtual-new master-pool dc=new

    >dpconf set-ldap-data-view-prop virtual-new dn-mapping-source-base-dn:dc=old

    >ldapsearch -p 7777 -b dc=new "(cn=user.1)"
    version: 1
    dn: cn=user.1,dc=new
    objectClass: person
    objectClass: top
    cn: user.1
    sn: 1
    userPassword: {SSHA}PzAK73RDZIikdI8qRqD7MYubasZ5JyJa/BToMw==


  5. Repeat 3 and 4 for the physical dc=new suffix and we now have totally virtualized back end.

    >dpconf create-ldap-data-view actual-new master-pool dc=new

    >dpconf create-ldap-data-view virtual-old master-pool dc=old

    >dpconf set-ldap-data-view-prop virtual-old dn-mapping-source-base-dn:dc=new

    >ldapsearch -p 7777 -b dc=old "(cn=user.3)"
    version: 1
    dn: cn=user.3,dc=old
    objectClass: person
    objectClass: top
    cn: user.3
    sn: 3
    userPassword: {SSHA}5LFqYHLashsY7PFAvFV9pM+C2oedPTdV/AIADQ==

    >ldapsearch -p 7777 -b dc=new "(cn=user.3)"
    version: 1
    dn: cn=user.3,dc=new
    objectClass: person
    objectClass: top
    cn: user.3
    sn: 3
    userPassword: {SSHA}5LFqYHLashsY7PFAvFV9pM+C2oedPTdV/AIADQ==

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>
About

Directory Services Tutorials, Utilities, Tips and Tricks

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today