Monday Aug 06, 2007

fhb - Http Load Generator

One of the simplest performance and load tests that can be performed on any web server is to measure the response time of a static HTTP GET request. If you measure how this response time degrades as more parallel connections issue the same request, you can get a basic understanding of the performance and scalability of a web server.

Anyone who's worked with Apache long enough is probably familiar with ab, the  load generator/benchmarking tool which is so wide-spread. This is a sad state of affairs as 'ab' is extremely flawed. I don't want to repeat it's flaws here, as Scott Oaks has done an excellent job of summarizing them. 

Another tool that is also popular is http_load, but this one has its drawbacks as well. Although it is better than ab in that it does maintain multiple parallel connections, the driver is not multi-threaded so it doesn't exactly mimic a multi-user load, leading to inaccuracies in response time measurement. Further it handles connections are handled via select(3c), which has known scalability issues.

Enter the Faban Http Bench (fhb) command line tool. This tool is part of Faban, an open source benchmarking kit that lets you develop and run complex benchmarks. For simple http load testing as described above, Faban now includes a simple tool called fhb.

Fhb generates load by instantiating multiple client threads, each thread runs independently and maintains it's own statistics. The response time data collected is highly accurate. The entire infrastructure is written in Java, allowing it to run on any platform (although the wrapper tool fhb is a shell script).

fhb Usage

1. Download fhb (you only need the client tar) and do the following :
    gzip -cd faban-client-062907.tar.gz | tar xvf -
   You should find fhb in the faban/bin directory.

2. Set JAVA_HOME appropriately (JDK 1.5 is required. /usr/java should work fine but double-check to make sure it's 1.5)

3. Set your PATH to include $JAVA_HOME/bin

4. Start your web server (ideally on another machine) and have a static file of whatever size you want to test. Let's assume that the webserver is running on 10.10.18.111:80 and a file called test.html exists in its docs directory.

5. Run fhb from your client machine's faban directory as follows :

   # bin/fhb -J -Xmx1500m -J -Xms1500m -s -r 60/180/0 -c 100 http://10.10.18.111:80/test.html

   Here is an explanation of some of the args :

   -J :  anything following -J is passed to the JVM as is.

   -s :   Create the summary file and save it in the default output directory. Without this option, fhb will simply delete the files created after displaying the summary statistics. By saving the output, you can go back and look at it later if you want. A separate directory will be created for each run in /var/tmp (changeable via the -D option)

  -r : The rampup/steady-state/rampdown times for the run in seconds.

  -c : The number of connections. (You may want to start with 1, 10 before trying larger numbers)

The final parameter is the URL that is accessed. Note that the URL can also point to a php/perl script if you want to include mod_php/mod_perl in your testing.

fhb Output

When you run the tool, you will get output like the following :

Bytes Read from class :1525
Aug 6, 2007 12:49:43 PM com.sun.faban.driver.core.MasterImpl runBenchmark
INFO: RunID for this run is : 1
Aug 6, 2007 12:49:43 PM com.sun.faban.driver.core.MasterImpl runBenchmark
INFO: Output directory for this run is : /var/tmp//faban_cd/1
Aug 6, 2007 12:49:46 PM com.sun.faban.driver.core.AgentImpl run
INFO: http_driver1Agent[0]: Successfully started 100 driver threads.
Aug 6, 2007 12:49:46 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Started all threads; run commences in 1995 ms
Aug 6, 2007 12:50:48 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Ramp up completed
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Steady state completed
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl executeRun
INFO: Ramp down completed
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl getDriverMetrics
INFO: Gathering http_driver1Stats ...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Printing Summary report...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Summary finished. Now printing detail xml ...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Summary finished. Now printing detail ...
Aug 6, 2007 12:53:48 PM com.sun.faban.driver.core.MasterImpl generateReports
INFO: Detail finished. Results written to /var/tmp//faban_cd/1.
ops/sec: 7490.989
% errors: 0.0
avg. time: 0.006
max time: 1.001
90th %: 0.05
Saving output from run in /var/tmp//faban_cd

The output files are saved in /var/tmp/faban_cd/1 for this run. Currently, the summary output is only in xml, but it's a simple matter to write an xsl script to convert it to whatever format you want. (The final release will include such a script).

Response Times

A note on response times - I often see reports quoting average response times. In fact, most common benchmarking tools only capture the average response times. If you are concerned about the response experienced by your web customer, then average response times just don't cut it. That is why, most standard benchmarks (like TPC, SPEC) specify a 90th percentile response time criteria - the time by which 90% of the requests should have been completed. This is a reasonable measure - if 90% of your customers see an adequate response, you're site's probably performing okay.

To illustrate my point, look at the response times from the sample output above. The avg. time is 0.006 seconds but the 90% time is 0.05 seconds, nearly ten times the average !
 

fhb Release

fhb will be part of the impending 1.0 release of Faban. At that time, it should include documentation and a post-processing script for the summary xml output. But please do check it out now. That way, we can incorporate any feedback before it's release.

But don't just stop at using fhb and doing simple GET requests to test your web server. Drive a realistic application to get a better sense of how it performs. You can use Faban's full power to write a load generator for any application or if you already have a full benchmark, drop it into Faban's harness to capitalize on it's many wonderful features such as automated monitoring, graphing and tabular viewing of results, results comparisons etc. Check out all the functionality it offers at the Faban website.
 


About

I'm a Senior Staff Engineer in the Performance & Applications Engineering Group (PAE). This blog focuses on tips to build, configure, tune and measure performance of popular open source web applications on Solaris.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today