Performance Impact Of Toggling Access Log
By arnaud on Jun 15, 2009
Many a customer need to have consistent high performance over traceability but most want both. As part of our current effort to rewrite the logging subsystem, we took a couple of quick metrics to assess how much headroom we had. This may answer some of the questions you asked yourself in the past.
Bird's Eye View
I'll make this quick:
This was just a crude test on a Sun Fire X4150. This is in no way a benchmark of DPS for maximum throughput or optimal response time. The only intent of this article is to illustrate the performance difference the access log makes for DPS.
How do I know if I hit the access log bottleneck ?Mainly, your CPU utilization is less than 100% (sometimes far less) and you have reached a maximum of throughput, throwing any more load to DPS only results in longer response times... then there is a strong likelihood that turning off the access log will allow to squeeze the extra drops of performance.
I won't go into details about the ins and outs of logging but simply try to articulate the challenge it poses under stress. Particularly, if you think about it, as you have seen in the first graph above, we may process about 10,000 operations per second. For each operation, we may need to log a number of steps, among which:
- the time the connection was established
- the time the connection was processed by the default connection handler
- the time the operation was handed off to the adequate connection handler
- the time the operation was sent to the back end server
- the time the operation came back from the back end server
- the time the operation was sent back to the client
For some back-ends or if custom plug-ins are configured, we may have more.
We may easily have 5 times the number of writes to the log compared to the number of LDAP operations we actually process. Now imagine that we take a lock anywhere in the process and you will immediately understand how this can become a bottleneck under heavy LDAP load. We are currently in the process of removing this lock and we inspired ourselves from the approach we used in OpenDS. OpenDS has a very efficient implementation, costing only between 6 and 8% of decrease compared to having no access log. The main goal of this reimplementation is to remove the performance bottleneck (some may call it a glass ceiling...) and introduce as little jitter in response time as possible.
Here's another snapshot of the state of affairs with DPS 6.x ...
I hope this helped shed some light.