Improving DS performances on Sun CoolThreads servers (T1, T2, T2 Plus)
By marcos on Nov 20, 2008
By default, Directory Servers 5.x and 6.x will not take full advantage of the multiple-core multiple-thread architecture provided by Sun CoolThreads servers based on UltraSPARC T1, T2, or T2 Plus processors, as DS processes real-time execution will be limited to a very reduced set of them. This is because DS runs by default with one single polling thread handling all network received activity, which soon becomes the bottleneck for the execution of parallel tasks inside the process (like handling simultaneously multiple read and update requests received.
These initial problems with the T2 performance for Directory were pretty much fixed with a change inside the DS polling thread code to make it configurable and multi-threaded instead of having a single hardcoded thread for such task. This allows for having multiple polling threads running in parallel on different cores, hence splitting the load succesfully between different cores. This bug fix is identified as BugId 6576080. To make it active, one needs to configure a new attribute:
- ds-polling-thread-count under cn=config for 6.x
- nsslapd-max-poll-threads under cn=config for 5.x
Both attributes allows to set up multiple multiple threads (2, 4, 6 depending on the pre-prod tests results are the typical values). This is explained inside the bug itself.
When using the attributes above, the connection handling activity is split and handled by different threads in parallel, allowing for a much more fair repartition of duties and a proper scalability pattern in terms of CPU's core/threads global usage
An aside unexpected undesired side-effect of this very important enhancement was that, when combined with replication, some replication sessions could be impacted by sudden unexpected delays while receiving replicated changes from other replicas. This aside problem was found to be caused by a bug inside the flow control algorithm governing the maximum number of threads which can be used to handle operations incoming via one single connection (mechanism which is governed by the configuration attribute
Indeed, whenever a connection overpasses that threshold (a very common case within a replication session), the flow control algorithm will remove the connection from the polling list. When it goes below the threshold it returns the
connection to the polling
list but ... wake up the thread 0 (hard coded).
In single polling thread, this thread was indeed the responsible of
listening new connection and
opened connections, so that was ok. But with multiple polling threads it is only
responsible of new connections, not existing ones. Hence, waking it up does not help to poll again the replication session.
BugId 6772870 was created to track down this side effect, for which an easy workaround exists:
- A very simple workaround is to increase max-thread-per-connection-count from 5 (default) to 30. In DS 6.x this will be achieved as follows:
dsconf set-server-prop -p <port> -e max-thread-per-connection-count:30
- Because of this setting above, it is also recommended to increase the default nsslapd-threadnumber from 30 to 50 at least.