X
  • MySQL
    December 12, 2008

Scalability and Stability for SysBench on Solaris

My mind is playing "Suffering Succotash..."

I have been working on MySQL performance for a while now, and the team
I am in have discovered that SysBench could do with a couple of tweaks
for Solaris.


Sidebar - sysbench is a simple "OLTP" benchmark which can test
multiple databases, including MySQL. Find out
all about it here
, but

go to the download page
to get the latest version.

To simulate multiple users sending requests to a database, sysbench
uses multiple threads. This leads to two issues we have identified
with SysBench on Solaris, namely:

  • The implementation of random() is explicitly identified as unsafe
    in multi-threaded applications on Solaris. My team has found this is
    a real issue, with occasional core-dumps happening to our multi-threaded
    SysBench runs.
  • SysBench does quite a bit of memory allocation, and could do with
    a more scalable memory allocator.

Neither of these issues are necessarily relevant only to Solaris, by the way.

Luckily there are simple solutions. We can fix the random() issue by
using lrand48() - in effect a drop-in replacement. Then we can fix the
memory allocator by simply choosing to link with a better allocator on
Solaris.

To help with a decision on memory allocator, I ran a few simple tests
to check the performance of the two best-known scalable allocators
available in Solaris. Here are the results ("libc" is the default
memory allocator):


Throughput

To see the differences more clearly, lets do a relative comparison, using "umem" (A.K.A. libumem) as the reference:


Relative Throughput

So - around 20% less throughput at 16 or 32 threads. Very little difference at 1 thread, too (where the default memory allocator should be the one with the lowest synchronization overhead).

Where you see another big difference is CPU cost per transaction:


CPU Cost

I will just point out two other reasons why I would recommend libumem:

I have logged these two issues as sysbench bugs:


However, if you can't wait for the fixes to be released, try these:

Join the discussion

Comments ( 2 )
  • Alexey Kopytov Monday, December 15, 2008

    Tim, thanks for logging the bugs and providing the patches. Few comments on your observations below.

    While it's true that sysbench uses malloc(), most of the calls come from the MySQL client library rather than from sysbench itself. A simple D script and results on a single-threaded OLTP "complex" test:

    #!/usr/sbin/dtrace -qs

    pid$target::my_\*:entry {client = 1;}

    pid$target::my_\*:return {client = 0;}

    pid$target::malloc:entry /client == 1/ {client_cnt++;}

    pid$target::malloc:entry /client == 0/ {sysbench_cnt++;}

    END {printf("Client lib malloc() calls: %i, sysbench malloc() calls: %i", client_cnt, sysbench_cnt);}

    # ./count_malloc.d -p 55288

    \^C

    Client lib malloc() calls: 719458, sysbench malloc() calls: 247280

    So about 75% of the calls came from the client library. Although some of them are unavoidable (due to mysql_store_result(), for example) I think at least some can be removed after a brief investigation. In particular, this my_memdup() (which calls malloc() internally) in cli_stmt_execute() was even marked as a candidate for removal:

    /\* TODO: Look into avoding the following memdup \*/

    if (!(param_data= my_memdup(net->buff, length, MYF(0))))

    As to malloc() calls from sysbench itself, they are not really necessary, since all memory can be pre-allocated before the actual test run. This will be fixed in the next version.

    Though it would still be useful to be able to build sysbench with alternative allocators, so that allocations occurring in the client library could benefit from better scalability. However the excessive malloc() usage should be investigated further and possibly reported as a MySQL bug.

    Regarding the random() issue, it seems like only Linux has a thread-safe random() implementation. I will include the lrand48() patch into the next release.


  • Tim Cook Monday, December 15, 2008

    Thanks, Alexey.

    I will have a look at the issue in the client library you identified.

    Can you at least consider having SysBench compile with -lumem (and prior to -lmysqlclient_r). I ask because some who build SysBench may not also be building MySQL, therefore even in the distant future may be using a client library that will continue to have this issue.


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.