Monday Dec 21, 2009

Twice Faster: Import Tips And Tricks

Rationale

We understand. Really, we do.

You want to import your data as fast possible. Not only nobody likes to wait but there are time constraints on everyone of us. In particular, we're constrained by maintenance windows.

So, what does give you the best return on investment? Higher CPU Clock speed? More CPUs? More memory? We're here to discover just that together.

Irrespective of the litterature on the topic, you must make sure that import will not have to be done in multiple passes or you'll get killed on index merging time. To do so, shoot high and dial down gradually to find the right cache size for your data set.

Bird's Eye View

In essence, what I did was to start an import on a vanilla system and Directory Server with a customer's actual data, 7.9M entries. The system I am testing with isn't a server. The system doesn't matter, it is a constant. The variable are the amount of import cache, the number of CPUs active (1-8) and the CPU clock speed (from 3.3 to 3.7GHz). In short, memory matters most.

The Meat

The Setup

The instance I am doing this with is an actual telco setup, with 7.9M entries in the LDIF file. The LDIF weighs in at 12GiB. There are 10 custom indexes configured for equality only.The 8 system indexes are there as well.

On the physical side of things, the machine is my desktop, an Intel Corei7 975EE @ 3.3GHz. It has a 64GB Intel X25 and a 10,000 rpm SATA drive. The disk layout is described in more detail here.

Sensivity To Import Cache 

Despite what the documentation says, there are huge gains to be reaped from increasing the import cache size, and depending on your data set, this may make  a world of difference.

This is the first thing that I tweaked during this test first phase and bumping import cache from 2GiB to 4GiB chopped import time in half. Basically, if your import has to occur in more than a single pass, then your import cache isn't big enough, try to increase it if your system can take it.

Sensivity To Clock Speed

Ever wondered if a system with CPUs twice faster would buy you time on import? Not really. Why? Well, if the CPUs are waiting on the disks or locks then higher clock speeds isn't going to move the needle at all. That's what's going on here. Check this out...

The reason the 3.7GHz import isn't as fast as the 3.3GHz is because my overclocking might have thrown off the balance between the core clock and the bus clock, so the CPU is spinning its wheels ,waiting to access memory and IO...

I officially declare this one moot. I'll try again later with underclocking.

Sensivity To Number Of CPUs

Scalability is an interesting challenge. Ideally, you'd want half the import time given twice the resources. In reality, import is very lock intensive to avoid collisions/corruptions so it isn't quite that linear. Here's what I got on my system, all other things being equal.

So even though the scalability isn't linear, the good thing is the more CPUs the better your performance is going to be.

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Tuesday Dec 15, 2009

DSEE 7.0 On Windows Gotcha

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

When you just want it to work quick and easy, and since the zip distribution lends itself particularly well to that with only extracting the archive in the way to being productive with your Directory Server, you only skim through the install guide. Well, at least if you're as impatient as I am. Make sure to read this. 

Bird's Eye View

After the quick and painless install, you add the single bin folder to your PATH environment and get crackin'. Directory Server instances can be created. As can Directory Proxy Server instances. But the usual command line in the DSRK, the indispensable ldapsearch, doesn't. What? You missed one tiny component that still needs to be "installed" on your machine. 

The Meat 

If when you try ldapsearch from DSEE 7.0 on your vanilla windows install you get the message below: 

 I'll make a quick text note here so it's searchable...

The application was unable to start correctly (0xc0150002)

Then it only means that you forgot to install the Visual C++ Redistributable Package, as stated in a tiny Note- block in the install guide which had completely escaped me during my quick skimming. Note that the doc points you to downloading the bits from microsoft but since we're nice guys, the installer is included in the DSEE ZIP archive, at the root of the zip file and is called vcredist_x86.exe. Be sure to install that if you need the ldap\* utilities.

happy LDAP'ing

Monday Dec 14, 2009

Import: Intel's HyperThreading Performance Gain

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

 As I am experimenting with more and more of the new-ish Intel Nehalem-based servers, a question kept bugging me. How much am I  getting out of these "hyper" threads that appear as CPUs in the operating system? I took some time to compare a Directory Server import on my corei7 workstation with an actual customer's data set of 7.9 Million entries.

Bird's Eye View

Don't bet the house on HyperThreading.

The Meat

To conduct this test in a simple fashion, all I did was an import of these entries first with all the CPUs enabled and then by disabling all the odd CPUs. Doesn't get any simpler. All other things can be considered equal.

The result is quite stunning really.



 Import Time
 8 CPUs (with HyperThreading)
 30:56.62
 4 CPUs (without HyperThreading)
 35:04.60

Which makes a difference of about 13%...

Just thought someone out there would find this as puzzling as I. Quite honestly, I  believe this is shared between the Directory Server not scaling too well, the IO actually being a contention point and HyperThreading being somewhat ineffective. More on that later.

Tuesday Dec 08, 2009

Using Wireshark Do Dissect LDAP Traffic

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

Every once in a while, an elusive issue may drive you bonkers because you can't quite identify what is going awry, which of course is crucial to solving said issue. In such cases, it's good to go back to the basics and actually analyze what is flowing on the wire. To do so, snoop is the tool. But more important than capturing the data, which is fairly well documented on the web, analyzing properly what the capture contains may put you on track to solving the issue.

Bird's Eye View

In this entry, I'll introduce how wireshark can be used to help investigate LDAP issues or even simply to double check your LDAP product is true in its performance metrics... A word of advice, never trust a single metric at face value, check yourself. The LDAP server (or web or app server for that matter) may report a response time of X, but that really is no good to your client applications if the network card or the TCP layer has trouble getting the message through the wire. Do use snoop to collect data and wireshark to check that everything is fine or dig in the snoop files.  

The Meat

First things first, installing Wireshark.

On OpenSolaris:

1- get pkgutil

2- install wireshark

Here's how:

pfexec pkgadd -d http://blastwave.network.com/csw/pkgutil_`uname  -p`.pkg
yes|pfexec /opt/csw/bin/pkgutil --install CSWwireshark

3- start wireshark, it is installed in /opt/sfw/bin/wireshark, for example do:

nohup /opt/csw/bin/wireshark 2>/dev/null &

 4- Now that wireshark is started up, you can open up you snoop file and what you get is basic Ethernet+IP+TCP decoding., like so:

5- So we will quite simply have to tell wireshark to decode the TCP packets content as being LDAP (wherever applicable). Go to Analyze->Deocde As. A window pops up to allow you to select which decoder to use. Select the "Transport" tab, and then click on LDAP in the list. Parsing through the whole snoop file again may take a while, but once it's done, it will be worth the wait.

6- Once data is properly interpreted as LDAP, we can see that those TCP packets with an LDAP payload will now be advertised as LDAP (highlighted in green) right in the wireshark interface:

Now that you have followed these few simple steps, you can dig in the snoop data and graph statistics.

For example, you can very easily graph the ratio of LDAP operations to LDAP operation taking more than one second with the statistics graphing tool:

 Of course, there are dozens of other invaluable tools in wireshark that are not only of the greatest quality but immensely useful as well to spot problems in your traffic that many higher level tools won't be able to help with, what comes to mind is stuff like:

  • issues with lower-level than like IP problems (for e.g.  CRC errors) or TCP retransmits
  • hitting the maximum bandwith on the interface (use the statistics->IO Graphs tools and select YAxis to Bytes/Tick)
  • LDAP level SLA not being met: you can check wehther a particular LDAP Time was met or not over your snoop file as shown above
  • Check for particular LDAP return codes
  • Check for particular LDAP filters
  • Filter your results on anything contained in an LDAP packet

 As I said earlier, this tool is truly fantastic.

Hope this ends up helping someone. Enjoy! 

Monday Dec 07, 2009

DSIO: A Storage Sizing Tool For LDAP

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

 Ever had this seemingly insurmountable wall of incomprehension between two teams that it was pretty obvious communication would never quite work ? Last year I had an interaction with a customer where one team manages all the storage infrastructure for the company and another manages the LDAP infrastructure. The LDAP team had to "purchase" storage from the storage team. Pricing was based on guaranteed IOPS. The LDAP folks provide authentication to the business and they are given performance requirements to meet and need to architect or expand to meet demand. The gap is that the LDAP team had no real tool to translate their LDAP performance numbers in terms of IOPS in order to purchase the right storage package.

This morning, I was on a call with a prospect who wants to run everything on VMWare ESX and swear their NAS can serve the load. Our LDAP experts told them that there just was no way to meet their very stringent performance requirements. The customer shot back the usual "show me the IOPS. So I thought I'd dust off the old DSIO DTrace script and share here since it could be reused in other instances that I do not know about.

Bird's Eye View

The DTrace script hooks up to your slapd process and intercepts calls the processing functions to be able to count the number of LDAP operations. It also intercepts lower level I/O operating system calls in order to be able to associate the LDAP operation to its subsequent I/O calls. Simple, isn't it?

The Meat

Pre-requisites

  • DTrace (obviously)
  • KSH (at least 88 but that's not much to ask) 
  • the dsio script

0 to 60 in no time

First, please note that you need to have administrative rights to run DTrace.

Second, by default dsio will look for the first slapd process it finds running, so only use this default behavior when you know for a fact that you have a single instance of Directory Server running.

# ./dsio

On Solaris:

$ pfexec ./dsio

The Rest Of The Way

- P: specify the PID of the running directory to trace (in case you'd have more than one running)

- l: print detailed information with respect to the LDAP traffic

-r: print detailed information with respect to the read I/O activity

-w: print detailed information with respect to the write I/O activity

As you can see in this case I only apply a modify load to the instance so as to make the point more explicit. Printing out the details is very useful to compare the actual counts of operations and the breakdown, the times, etc...

Note that in this case the IOPs are about half the LDAP throughput. How is that possible? By exploiting one of ZFS best features, the ZIL. I initially thought that my script was wrong and went to double check things with the ZFS folks. But that story is for another article.

Special thanks to Brendan Gregg for his invaluable scripts and tutorials on DTrace which helped me tremendously. The content this dude puts out turns DTrace black art into nothing more than shell scripting. Simply mind boggling.

Enhancements

  • Allow looping indefinitely
  • Implement a CSV output

Support

Best effort again as I'm not dedicated to this, you can send your questions and support requests to arnaud -- at -- sun -- dot -- com, or post your rants as comments here and I'll do my best to take care of you.

Enjoy!

About

Directory Services Tutorials, Utilities, Tips and Tricks

Search

Categories
Archives
« December 2009
SunMonTueWedThuFriSat
  
1
3
4
5
6
9
10
11
12
13
17
20
22
23
24
25
26
27
28
29
30
31
  
       
Today