Scalability and Stability for SysBench on Solaris

My mind is playing "Suffering Succotash..."

I have been working on MySQL performance for a while now, and the team I am in have discovered that SysBench could do with a couple of tweaks for Solaris.

Sidebar - sysbench is a simple "OLTP" benchmark which can test multiple databases, including MySQL. Find out all about it here , but go to the download page to get the latest version.

To simulate multiple users sending requests to a database, sysbench uses multiple threads. This leads to two issues we have identified with SysBench on Solaris, namely:

  • The implementation of random() is explicitly identified as unsafe in multi-threaded applications on Solaris. My team has found this is a real issue, with occasional core-dumps happening to our multi-threaded SysBench runs.
  • SysBench does quite a bit of memory allocation, and could do with a more scalable memory allocator.

Neither of these issues are necessarily relevant only to Solaris, by the way.

Luckily there are simple solutions. We can fix the random() issue by using lrand48() - in effect a drop-in replacement. Then we can fix the memory allocator by simply choosing to link with a better allocator on Solaris.

To help with a decision on memory allocator, I ran a few simple tests to check the performance of the two best-known scalable allocators available in Solaris. Here are the results ("libc" is the default memory allocator):

Throughput

To see the differences more clearly, lets do a relative comparison, using "umem" (A.K.A. libumem) as the reference:

Relative Throughput

So - around 20% less throughput at 16 or 32 threads. Very little difference at 1 thread, too (where the default memory allocator should be the one with the lowest synchronization overhead).

Where you see another big difference is CPU cost per transaction:

CPU Cost

I will just point out two other reasons why I would recommend libumem:

I have logged these two issues as sysbench bugs:

However, if you can't wait for the fixes to be released, try these:

Comments:

Tim, thanks for logging the bugs and providing the patches. Few comments on your observations below.

While it's true that sysbench uses malloc(), most of the calls come from the MySQL client library rather than from sysbench itself. A simple D script and results on a single-threaded OLTP "complex" test:

#!/usr/sbin/dtrace -qs

pid$target::my_\*:entry {client = 1;}
pid$target::my_\*:return {client = 0;}
pid$target::malloc:entry /client == 1/ {client_cnt++;}
pid$target::malloc:entry /client == 0/ {sysbench_cnt++;}
END {printf("Client lib malloc() calls: %i, sysbench malloc() calls: %i", client_cnt, sysbench_cnt);}

# ./count_malloc.d -p 55288
\^C
Client lib malloc() calls: 719458, sysbench malloc() calls: 247280

So about 75% of the calls came from the client library. Although some of them are unavoidable (due to mysql_store_result(), for example) I think at least some can be removed after a brief investigation. In particular, this my_memdup() (which calls malloc() internally) in cli_stmt_execute() was even marked as a candidate for removal:

/\* TODO: Look into avoding the following memdup \*/
if (!(param_data= my_memdup(net->buff, length, MYF(0))))

As to malloc() calls from sysbench itself, they are not really necessary, since all memory can be pre-allocated before the actual test run. This will be fixed in the next version.

Though it would still be useful to be able to build sysbench with alternative allocators, so that allocations occurring in the client library could benefit from better scalability. However the excessive malloc() usage should be investigated further and possibly reported as a MySQL bug.

Regarding the random() issue, it seems like only Linux has a thread-safe random() implementation. I will include the lrand48() patch into the next release.

Posted by Alexey Kopytov on December 15, 2008 at 01:26 AM PST #

Thanks, Alexey.

I will have a look at the issue in the client library you identified.

Can you at least consider having SysBench compile with -lumem (and prior to -lmysqlclient_r). I ask because some who build SysBench may not also be building MySQL, therefore even in the distant future may be using a client library that will continue to have this issue.

Posted by Tim Cook on December 15, 2008 at 03:12 AM PST #

Post a Comment:
Comments are closed for this entry.
About

Tim Cook's Weblog The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today