Improving DS performances on Sun CoolThreads servers (T1, T2, T2 Plus)

By default, Directory Servers 5.x and 6.x will not take full advantage of the multiple-core multiple-thread architecture provided by Sun CoolThreads servers based on UltraSPARC T1, T2, or T2 Plus processors, as DS processes real-time execution will be limited to a very reduced set of them. This is because DS runs by default with one single polling thread handling all network received activity, which soon becomes the bottleneck for the execution of parallel tasks inside the process (like handling simultaneously multiple read and update requests received.

These initial problems with the T2 performance for Directory were pretty much fixed with a change inside the DS polling thread code to make it configurable and multi-threaded instead of having a single hardcoded thread for such task. This allows for having multiple polling threads running in parallel on different cores, hence splitting the load succesfully between different cores. This bug fix is identified as BugId 6576080. To make it active, one needs to configure a new attribute:

  • ds-polling-thread-count under cn=config for 6.x
  • nsslapd-max-poll-threads under cn=config for 5.x

Both attributes allows to set up multiple multiple threads (2, 4, 6 depending on the pre-prod tests results are the typical values). This is explained inside the bug itself.

When using the attributes above, the connection handling activity is split and handled by different threads in parallel, allowing for a much more fair repartition of duties and a proper scalability pattern in terms of CPU's core/threads global usage

An aside unexpected undesired side-effect of this very important enhancement was that, when combined with replication, some replication sessions could be impacted by sudden unexpected delays while receiving replicated changes from other replicas. This aside problem was found to be caused by a bug inside the flow control algorithm governing the maximum number of threads which can be used to handle operations incoming via one single connection (mechanism which is governed by the configuration attribute max-thread-per-connection-count).  Indeed, whenever a connection overpasses that threshold (a very common case within a replication session), the flow control algorithm will remove the connection from the polling list. When it goes below the threshold it returns the connection to the polling list but ... wake up the thread 0 (hard coded). In single polling thread, this thread was indeed the responsible of listening new connection and opened connections, so that was ok. But with multiple polling threads it is only responsible of new connections, not existing ones. Hence, waking it up does not help to poll again the replication session.

BugId 6772870 was created to track down this side effect, for which an easy workaround exists:

 

  • A very simple workaround is to increase max-thread-per-connection-count from 5 (default) to 30. In DS 6.x this will be achieved as follows:
dsconf set-server-prop -p <port> -e max-thread-per-connection-count:30
  • Because of this setting above, it is also recommended to increase the default nsslapd-threadnumber from 30 to 50 at least.
Comments:

I've checked everywhere for nsslapd-max-poll-threads and cannot find it. I've attempted to set that attribute in a 5.2 instance I have running and get the following message:

[21/Nov/2008:08:16:40 -0500] - ERROR<4124> - Config - conn=-1 op=-1 msgId=-1 - Configuration error Unknown attribute nsslapd-max-poll-threads will be ignored.

I'm running 5.2p5 with some hotfixes. Is this only available in p6?

Posted by Frank on November 20, 2008 at 09:36 PM PST #

Inside 5.x, this tuning is only available as a special hotfix, that's right. You need to contact Sun Support to get that. Inside 6.x, the tuning is officially available inside the product without special hotfixes since 6.3.

Posted by Marcos on November 24, 2008 at 03:30 PM PST #

There is a hotfix available for BugId 6772870 mentioned in the above article for 5.X patch6 environment. Received the hotfix couple of weeks ago for my customer.

Posted by Senthilvel Chenniappan on February 18, 2009 at 04:37 AM PST #

Post a Comment:
Comments are closed for this entry.
About

marcos

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today