Regperf - The followup
By damiencooke on Aug 07, 2007
Tom Daly and I have been looking at open source solutions in the web application framework area. It complemented our shared interest of open source databases. We noticed there was a large amount of interest around Ruby on Rails. It seemed like an obvious place to start researching where the right place, if such a place existed for frameworks like RoR, was and what type of applications were it's sweet spot. On the way we learned about Grails also, a similar purpose project and we decided to include Grails in our investigation.
Regperf is a trivial application to represent many small web applications that individuals and corporations build to do a simple task, in this case keep details about software registrations. The genre of applications that Regperf belongs to is usually used for very simple, one or more table, applications to Create/Read/Update/Delete simple data for a particular purpose. In it's current state it is a single table application that can be simply generated for each of the three platforms we chose to compare. Java EE 5.0, Ruby on Rails (which includes JRuby on Rails) and Grails. We had intended getting this data up on our blogs earlier but I got busy on other stuff. Recently there has been some activity around our JavaOne presentation Some of the blogs I have seen: Joab Jackson and Grails Team So I thought I better spend the time.
The Ruby Performace Tests:
Before we start we should state that Regperf is not nor can be, in it's current state, considered a benchmark. The purpose was to simply test the characteristics of each of the frameworks compared to each other to determine where the framework was best suited and to make recommendations as to where each might be used. It was later we decided that Regperf using FABAN might be useful to others and decided to extend the application from a single table to a multi table application (Regperf 2.0)
We generated the trivial application and tested it using FABAN on each of the platforms. Running the RoR application standalone and with in Netbeans 6 (JRuby). We ran the Java EE 5.0 version on Glassfish and the Grails version was also run on the Glassfish application Server. Our findings were published in the Java One presentation.
Create a regperf user (createuser regperf)
Create a database called regperf ($ createdb regperf)
import the database ($pg_restore -d regperf regperfdb.tar)
To start the application:
In your web browser point to localhost:3000/subscription The application should be running (email as much detail as possible if you are having problems and need some assistance include postgres server log, screen shots etc) Now all we need is the test suite (Faban Driver) which needs to be expanded in $FABAN_HOME/samples directory for pre-installed FABAN. Now you can conduct your own tests. Instructions that need to be followed are:
Setup Faban and the Faban Regperf driver Download and install Faban. Note Faban has a driver and also a harness for automatically queuing and running multiple benchmarks but in these instructions we are going to run just the driver component and again assuming localhost set the FABAN_HOME environment variable and add $FABAN_HOME/bin to your PATH mkdir $FABAN_HOME/output (this is where the tests results will be stored) Download the Faban Regperf driver for Rails (as indicated above) and unjar it into $FABAN_HOME/samples. Edit the $FABAN_HOME/samples/regperf-grails/config/run.xml, change the value for outputDir to match your $FABAN_HOME/output directory. Modify the number of simulated users by changing the value for <scale>1<scale> you will get 10 users for each scale factor e.g. scale 1 will give 10 simulated users. Note the number of loaded subscribers should match the number of rows you load into the database (this is set initially to 100) refer to the Faban documentation for other values to change in the run.xml
Running the performance using the Rails runtime (webbrick)
cd $FABAN_HOME/samples/regperf-rails/sbin, Assuming that the database and the rails application is still running execute ./loader.sh 10 start the rmi registry (this runs in the background) ./registry.sh &
Start and execute the benchmark
./master.sh Starts up all of the simulated users and writes results of the run into $FABAN_HOME/output/xx directory. The results can be conveniently viewed with a web browser.
We would love to hear what you find with either our app or your own. Let us know if you have any problems and stay tuned for Regperf 2.0