Wednesday Dec 02, 2009

A Script To Rule Them All ... clients and backends that is

<script type="text/freezescript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/freezescript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

With Directory Proxy Server, regardless of the version, investigating traffic can get:

  • a) real tricky
  • b) time consuming
  • c) confusing
  • d) all of the above

and the answer is ... rolling drum ... d) !

So here is a script that you can feed your DPS access log to. It will output a CSV file that you can then load in your favorite spreadsheet software or just graph with tools like gnuplot and the like... it just will make your life easy...er.

Bird's Eye View

Disclaimer: It's not as anywhere as clever as logconv.pl for Directory Server, it only munches the data so that YOU can more easily spot issues or identify patterns. So what does this script produce in the end ?

It will take your DPS 6.x/7.0 access log in and output three csv files, one with the transaction volumes (suffixed "tx"), one with the average response times (suffixed "avg") and finally one with the maximum response time over a minute (suffixed "max"). Why not all in one file? I did initially but in a csv it turned out to really not be practical. So at least when you open up one of these files you know what you're looking at.

The Meat

Pre-requisites

Since I really started this initially to simply be able to "grep" a file on a windows system, I really had no plan and no idea it would end up in a tool like this. All that to say that I wrote in python instead of our customary Java tools. At least it has the merit of existing so you don't have to start from scratch. So you'll need python, at least 2.4. If you're on Solaris or Linux, you're covered. If on windows, simply download your favorite python, I have installed the 2.6.4 windows version from here.

You will also need to download the script. You may as well get the netbeans project if you'd like to change it to adapt it to your specific needs or help improve on it.

How Does It Work

0 To 60 In No Time

python dps-log-cruncher.py access 

The Rest Of The Way

-c      : break up statistics per client
-s      : break up statistics per back-end server
-f hh:mm: start parsing at a given point in time
-t hh:mm: stop parsing after a given point in time
-h      : print this help message
-v      : print tool version

Some examples:

split the output per client for all clients:

python dps-log-cruncher.py -c \* access 

 split the output per back-end server for client 192.168.0.17:

python dps-log-cruncher.py -c 192.168.0.17 -s \* access 

 split the output for all clients, all servers:

python dps-log-cruncher.py -c \* -s \* access 

 only output results from 6pm (18:00) to 6:10pm (18:10):

python dps-log-cruncher.py -f 18:00 -t 18:10 access 

 output results between 6:00pm (18:00) to 6:10pm (18:10) and split results for all clients and back-end servers:

python dps-log-cruncher.py -f 18:00 -t 18:10 -c \* -s \* access 

Enhancements

This is a list to manage expectations as much as it is one for me to remember to implement:

  1. Selectable time granularity resolution. Currently, all data is aggregated per minute. In some case, it would be useful to be able to see what happens per second
  2. Improve error handling for parameters on the CLI.
  3. Add a built-in graphing capability to avoid having to resort to using a spreadsheet. Spreadsheets do however give a lot of flexibility
  4. Add the ability to filter / split results per bind DN
  5. Output the response time distribution

Support

Best effort is how I will label it for now, you can send your questions and support requests to arnaud -- at -- sun -- dot -- com.

Enjoy!

About

Directory Services Tutorials, Utilities, Tips and Tricks

Search

Archives
« December 2009
SunMonTueWedThuFriSat
  
1
3
4
5
6
9
10
11
12
13
17
20
22
23
24
25
26
27
28
29
30
31
  
       
Today