By arnaud on Dec 07, 2009
Ever had this seemingly insurmountable wall of incomprehension between two teams that it was pretty obvious communication would never quite work ? Last year I had an interaction with a customer where one team manages all the storage infrastructure for the company and another manages the LDAP infrastructure. The LDAP team had to "purchase" storage from the storage team. Pricing was based on guaranteed IOPS. The LDAP folks provide authentication to the business and they are given performance requirements to meet and need to architect or expand to meet demand. The gap is that the LDAP team had no real tool to translate their LDAP performance numbers in terms of IOPS in order to purchase the right storage package.
This morning, I was on a call with a prospect who wants to run everything on VMWare ESX and swear their NAS can serve the load. Our LDAP experts told them that there just was no way to meet their very stringent performance requirements. The customer shot back the usual "show me the IOPS. So I thought I'd dust off the old DSIO DTrace script and share here since it could be reused in other instances that I do not know about.
Bird's Eye View
The DTrace script hooks up to your slapd process and intercepts calls the processing functions to be able to count the number of LDAP operations. It also intercepts lower level I/O operating system calls in order to be able to associate the LDAP operation to its subsequent I/O calls. Simple, isn't it?
- DTrace (obviously)
- KSH (at least 88 but that's not much to ask)
- the dsio script
0 to 60 in no time
First, please note that you need to have administrative rights to run DTrace.
Second, by default dsio will look for the first slapd process it finds running, so only use this default behavior when you know for a fact that you have a single instance of Directory Server running.
$ pfexec ./dsio
The Rest Of The Way
- P: specify the PID of the running directory to trace (in case you'd have more than one running)
- l: print detailed information with respect to the LDAP traffic
-r: print detailed information with respect to the read I/O activity
-w: print detailed information with respect to the write I/O activity
As you can see in this case I only apply a modify load to the instance so as to make the point more explicit. Printing out the details is very useful to compare the actual counts of operations and the breakdown, the times, etc...
Note that in this case the IOPs are about half the LDAP throughput. How is that possible? By exploiting one of ZFS best features, the ZIL. I initially thought that my script was wrong and went to double check things with the ZFS folks. But that story is for another article.
Special thanks to Brendan Gregg for his invaluable scripts and tutorials on DTrace which helped me tremendously. The content this dude puts out turns DTrace black art into nothing more than shell scripting. Simply mind boggling.
- Allow looping indefinitely
- Implement a CSV output
Best effort again as I'm not dedicated to this, you can send your
questions and support requests to arnaud -- at -- sun -- dot -- com, or post your rants as comments here and I'll do my best to take care of you.