A Script To Rule Them All ... clients and backends that is

<script type="text/freezescript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/freezescript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>


With Directory Proxy Server, regardless of the version, investigating traffic can get:

  • a) real tricky
  • b) time consuming
  • c) confusing
  • d) all of the above

and the answer is ... rolling drum ... d) !

So here is a script that you can feed your DPS access log to. It will output a CSV file that you can then load in your favorite spreadsheet software or just graph with tools like gnuplot and the like... it just will make your life easy...er.

Bird's Eye View

Disclaimer: It's not as anywhere as clever as logconv.pl for Directory Server, it only munches the data so that YOU can more easily spot issues or identify patterns. So what does this script produce in the end ?

It will take your DPS 6.x/7.0 access log in and output three csv files, one with the transaction volumes (suffixed "tx"), one with the average response times (suffixed "avg") and finally one with the maximum response time over a minute (suffixed "max"). Why not all in one file? I did initially but in a csv it turned out to really not be practical. So at least when you open up one of these files you know what you're looking at.

The Meat


Since I really started this initially to simply be able to "grep" a file on a windows system, I really had no plan and no idea it would end up in a tool like this. All that to say that I wrote in python instead of our customary Java tools. At least it has the merit of existing so you don't have to start from scratch. So you'll need python, at least 2.4. If you're on Solaris or Linux, you're covered. If on windows, simply download your favorite python, I have installed the 2.6.4 windows version from here.

You will also need to download the script. You may as well get the netbeans project if you'd like to change it to adapt it to your specific needs or help improve on it.

How Does It Work

0 To 60 In No Time

python dps-log-cruncher.py access 

The Rest Of The Way

-c      : break up statistics per client
-s      : break up statistics per back-end server
-f hh:mm: start parsing at a given point in time
-t hh:mm: stop parsing after a given point in time
-h      : print this help message
-v      : print tool version

Some examples:

split the output per client for all clients:

python dps-log-cruncher.py -c \* access 

 split the output per back-end server for client

python dps-log-cruncher.py -c -s \* access 

 split the output for all clients, all servers:

python dps-log-cruncher.py -c \* -s \* access 

 only output results from 6pm (18:00) to 6:10pm (18:10):

python dps-log-cruncher.py -f 18:00 -t 18:10 access 

 output results between 6:00pm (18:00) to 6:10pm (18:10) and split results for all clients and back-end servers:

python dps-log-cruncher.py -f 18:00 -t 18:10 -c \* -s \* access 


This is a list to manage expectations as much as it is one for me to remember to implement:

  1. Selectable time granularity resolution. Currently, all data is aggregated per minute. In some case, it would be useful to be able to see what happens per second
  2. Improve error handling for parameters on the CLI.
  3. Add a built-in graphing capability to avoid having to resort to using a spreadsheet. Spreadsheets do however give a lot of flexibility
  4. Add the ability to filter / split results per bind DN
  5. Output the response time distribution


Best effort is how I will label it for now, you can send your questions and support requests to arnaud -- at -- sun -- dot -- com.



I tried this on a solaris10 server with DPS 6.3.1, and I am getting an error, would you mind to help me on whats wrong (I don't know python language).
# /usr/local/bin/python dps-log-cruncher.py -c \* access.12
[('-c', 'access')]
['access.1', 'access.1-avg.csv', 'access.1-max.csv', 'access.1-tx.csv', 'access.10', 'access.11', 'access.12', 'access.12-avg.csv', 'access.12-max.csv', 'access.12-tx.csv', 'access.13', 'access.14', 'access.15', 'access.2', 'access.3', 'access.4', 'access.5', 'access.6', 'access.7', 'access.8', 'access.9', 'access.rotationinfo', 'access.rotationinfo.next', 'dps-log-cruncher.py', 'errors', 'errors.rotationinfo', 'errors.rotationinfo.next', 'access.12']
Traceback (most recent call last):
File "dps-log-cruncher.py", line 229, in <module>
NameError: name 'srvPattern' is not defined

Posted by Eli Kleinman on December 03, 2009 at 12:00 AM MST #

Yeah, my bad, I uploaded a new version with this fixed. Same URL to download the script: http://blogs.sun.com/ds/resource/dps-log-cruncher/dps-log-cruncher.py

let me know how it goes.

Posted by arnaud on December 03, 2009 at 03:59 AM MST #

Thanks a million its it works great.

Posted by Eli Kleinman on December 03, 2009 at 04:33 AM MST #

glad to hear that! And please don't hesitate with suggestions!

Posted by arnaud on December 03, 2009 at 06:00 AM MST #

Does this script work on 5.2 DPS logs? thx

Posted by Tad Perillo on December 04, 2009 at 09:29 AM MST #

I'm sorry, this script does not parse DPS 5 access logs. At least I've never tried. I believe the access log format is fairly different between the 5 and 6 series.

What's more, the 5 series is not only End-Of-Life'd at the end of this month but End-Of-Support-Life'd as well, so I am not sure if there is a need to implement support for 5 since most supported customers have already migrated up to 6, some to 7 already.
Those still on 5 are out-of-support and therefore I believe it is reasonable to expect to get exactly that.

But if there is a burning pressing need, I'll implement DPS 5 parsing, I don't believe it should be drastically different from what is already there.

Posted by arnaud on December 07, 2009 at 02:06 AM MST #

Post a Comment:
  • HTML Syntax: NOT allowed

Directory Services Tutorials, Utilities, Tips and Tricks


« December 2016