Monday Aug 06, 2007

fhb - Http Load Generator

One of the simplest performance and load tests that can be performed on any web server is to measure the response time of a static HTTP GET request. If you measure how this response time degrades as more parallel connections issue the same request, you can get a basic understanding of the performance and scalability of a web server.

Anyone who's worked with Apache long enough is probably familiar with ab, the  load generator/benchmarking tool which is so wide-spread. This is a sad state of affairs as 'ab' is extremely flawed. I don't want to repeat it's flaws here, as Scott Oaks has done an excellent job of summarizing them. 

Another tool that is also popular is http_load, but this one has its drawbacks as well. Although it is better than ab in that it does maintain multiple parallel connections, the driver is not multi-threaded so it doesn't exactly mimic a multi-user load, leading to inaccuracies in response time measurement. Further it handles connections are handled via select(3c), which has known scalability issues.

Enter the Faban Http Bench (fhb) command line tool. This tool is part of Faban, an open source benchmarking kit that lets you develop and run complex benchmarks. For simple http load testing as described above, Faban now includes a simple tool called fhb.

Fhb generates load by instantiating multiple client threads, each thread runs independently and maintains it's own statistics. The response time data collected is highly accurate. The entire infrastructure is written in Java, allowing it to run on any platform (although the wrapper tool fhb is a shell script).

fhb Usage

1. Download fhb (you only need the client tar) and do the following :
    gzip -cd faban-client-062907.tar.gz | tar xvf -
   You should find fhb in the faban/bin directory.

2. Set JAVA_HOME appropriately (JDK 1.5 is required. /usr/java should work fine but double-check to make sure it's 1.5)

3. Set your PATH to include $JAVA_HOME/bin

4. Start your web server (ideally on another machine) and have a static file of whatever size you want to test. Let's assume that the webserver is running on and a file called test.html exists in its docs directory.

5. Run fhb from your client machine's faban directory as follows :

   # bin/fhb -J -Xmx1500m -J -Xms1500m -s -r 60/180/0 -c 100

   Here is an explanation of some of the args :

   -J :  anything following -J is passed to the JVM as is.

   -s :   Create the summary file and save it in the default output directory. Without this option, fhb will simply delete the files created after displaying the summary statistics. By saving the output, you can go back and look at it later if you want. A separate directory will be created for each run in /var/tmp (changeable via the -D option)

  -r : The rampup/steady-state/rampdown times for the run in seconds.

  -c : The number of connections. (You may want to start with 1, 10 before trying larger numbers)

The final parameter is the URL that is accessed. Note that the URL can also point to a php/perl script if you want to include mod_php/mod_perl in your testing.

fhb Output

When you run the tool, you will get output like the following :

Bytes Read from class :1525
Aug 6, 2007 12:49:43 PM com.sun.faban.driver.core.MasterImpl runBenchmark
INFO: RunID for this run is : 1
Aug 6, 2007 12:49:43 PM com.sun.faban.driver.core.MasterImpl runBenchmark
INFO: Output directory for this run is : /var/tmp//faban_cd/1
Aug 6, 2007 12:49:46 PM com.sun.faban.driver.core.AgentImpl run
INFO: http_driver1Agent[0]: Successfully started 100 driver threads.
Aug 6, 2007 12:49:46 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Started all threads; run commences in 1995 ms
Aug 6, 2007 12:50:48 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Ramp up completed
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Steady state completed
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Ramp down completed
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl getDriverMetrics
INFO: Gathering http_driver1Stats ...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Printing Summary report...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Summary finished. Now printing detail xml ...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Summary finished. Now printing detail ...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Detail finished. Results written to /var/tmp//faban_cd/1.
ops/sec: 7490.989
% errors: 0.0
avg. time: 0.006
max time: 1.001
90th %: 0.05
Saving output from run in /var/tmp//faban_cd

The output files are saved in /var/tmp/faban_cd/1 for this run. Currently, the summary output is only in xml, but it's a simple matter to write an xsl script to convert it to whatever format you want. (The final release will include such a script).

Response Times

A note on response times - I often see reports quoting average response times. In fact, most common benchmarking tools only capture the average response times. If you are concerned about the response experienced by your web customer, then average response times just don't cut it. That is why, most standard benchmarks (like TPC, SPEC) specify a 90th percentile response time criteria - the time by which 90% of the requests should have been completed. This is a reasonable measure - if 90% of your customers see an adequate response, you're site's probably performing okay.

To illustrate my point, look at the response times from the sample output above. The avg. time is 0.006 seconds but the 90% time is 0.05 seconds, nearly ten times the average !

fhb Release

fhb will be part of the impending 1.0 release of Faban. At that time, it should include documentation and a post-processing script for the summary xml output. But please do check it out now. That way, we can incorporate any feedback before it's release.

But don't just stop at using fhb and doing simple GET requests to test your web server. Drive a realistic application to get a better sense of how it performs. You can use Faban's full power to write a load generator for any application or if you already have a full benchmark, drop it into Faban's harness to capitalize on it's many wonderful features such as automated monitoring, graphing and tabular viewing of results, results comparisons etc. Check out all the functionality it offers at the Faban website.

Thursday Apr 12, 2007

SMF support for Cool Stack Apache

Here are instructions on how to use SMF to manage Apache. These instructions assume no knowledge of SMF. All commands should be executed as root.

Create the manifest

 The manifest defines the service and controls the privileges with which the service will execute. We use the service name csk-httpd to distinguish this as a Cool Stack service (and not to cause any confusion with the httpd service that already exists).

Create a file named /var/svc/manifest/network/cskapache2.xml with the following contents :

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
    Copyright 2006-2007 Sun Microsystems, Inc.  All rights reserved.
    CSKapache2 manifest - should reside in /var/svc/manifest/network.

<service_bundle type='manifest' name='CSKamp:apache'>


       Because we may have multiple instances of network/http
       provided by different implementations, we keep dependencies
       and methods within the instance.

    <instance name='CSKapache2' enabled='false'>
       Wait for network interfaces to be initialized.
       <dependency name='network'
           <service_fmri value='svc:/milestone/network:default'/>

          Wait for all local filesystems to be mounted.
       <dependency name='filesystem-local'

           Wait for automounting to be available, as we may be
           serving data from home directories or other remote
       <dependency name='autofs'

           exec='/opt/coolstack/lib/svc/method/svc-cskapache2 start'
                   user='webservd' group='webservd'
                   privileges='basic,!proc_session,!proc_info,!file_link_any,net_privaddr' />

           exec='/opt/coolstack/lib/svc/method/svc-cskapache2 stop'
           <method_context />

           exec='/opt/coolstack/lib/svc/method/svc-cskapache2 refresh'
           <method_context />

        <property_group name='httpd' type='application'>
            <stability value='Evolving' />
            <propval name='ssl' type='boolean' value='false' />

        <property_group name='startd' type='framework'>
            <!-- sub-process core dumps shouldn't restart session -->
            <propval name='ignore_error' type='astring'
                     value='core,signal' />


    <stability value='Evolving' />
            <loctext xml:lang='C'>
                Apache 2 HTTP server
            <manpage title='httpd' section='8'
                manpath='/opt/coolstack/apache2/man' />
            <doc_link name=''
                uri='' />


Create the method

Create the file /opt/coolstack/lib/svc/method/svc-cskapache2 referenced in the manifest with the following contents and make it executable. You will have to create all directories below /opt/coolstack/lib first.This file assumes a certain name and location for the apache configuration and pid file. Edit it if you are not using the default settings from Cool Stack.

# Copyright 2004-2007 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
# ident "@(#)http-apache2       1.2     04/11/11 SMI"
# Modified for apache in CSKamp package of Cool Stack
# This file should reside in /opt/coolstack/lib/svc/method

. /lib/svc/share/


[ ! -f ${CONF_FILE} ] &&  exit $SMF_EXIT_ERR_CONFIG

case "$1" in
        /bin/rm -f ${PIDFILE}
        echo "Usage: $0 {start|stop|refresh}"
        exit 1

exec ${APACHE_HOME}/bin/apachectl $cmd 2>&1

Change file ownership

Since we want to start apache as user webservd (not root), we need to ensure that this user can write to the log directory and the pid file. All these files  reside in /opt/coolstack/apache2/logs by default.

# cd /opt/coolstack/apache2
# chown -R webservd logs
# chgrp -R webservd logs

Disable the Solaris http service

We do not want our service to conflict with the apache2 service that ships with Solaris. Note that by default, this service is disabled. You can check if it is enabled as follows :

# svcs |grep http

If no output is printed, then it is disabled. If you see something like :

maintenance    11:47:11 svc:/network/http:apache2


online    11:47:11 svc:/network/http:apache2

then, the service is up. Disable the service as follows : 

# svcadm -v disable http

svc:/network/http:apache2 disabled.

Start the csk-http service

Import the new service config as follows :

# svccfg -v import /var/svc/manifest/network/cskapache2.xml

We are now ready to start our service. Start it as follows :

# svcadm -v enable csk-http 

If the service starts successfully, you should see httpd processes running. A log of the service startup will be in /var/svc/log/network-csk-http:CSKapache2.log file. You can also get more detailed information for troubleshooting startup failures from the command svcs -x. More information on SMF can be found at





I'm a Senior Staff Engineer in the Performance & Applications Engineering Group (PAE). This blog focuses on tips to build, configure, tune and measure performance of popular open source web applications on Solaris.


« April 2014