Wednesday Jul 25, 2007

Regperf V1 - Running the Grails performance tests (howto)

Regperf (v1) – Running Grails Performance Tests
How to run the Regperf v1 tests using Grails framework
and Glassfish Java EE 5,0 server


Damien Cooke and I have been building some tools to enable performance testing and comparisons of a simple application built in 3 frameworks Grails, Rails and Java EE 5.0 (JSF), this blog entry is one of a number we are posting between us to describe how to run these simple performance tests (which we presented at JavaOne). This blog entry describes how to run the simple Regperf application implemented in Grails , Damien will be posting how to run the Rails version of Regperf in the very near future and when I have finished the JavaServer Faces write up I will post this too.

The instructions and software has been tested on Solaris and Mac OSX but should run anywhere where Faban and Grails , Java and Postgres run

The main steps

    1. Install the database

    2. Install / create the Grails “Regperf” application

    3. Setting up Faban and the Faban Regperf driver

    4. Running the performance tests using the Grails runtime (jetty)

    5. Deploying and testing the Grails application using Glassfish Java EE 5 application server

  1. Install the database.

  • we used postgres 8.4.2 (of course MySQL, JavaDB , Oracle etc are all okay too). For simplicity these instructions assume that Postgres DB is installed and running on the same host as your grails application.

  • download and unpack the postgres Regperf dump

  • create the db createdb regperfdb

  • import the database schema pg_restore -d regperfdb grails-regperfdb.tar

  1. Install / create the Grails application
    (assumes you have Grails installed at $GRAILS_HOME and $GRAILS_HOME/bin is in your PATH)


  • grails create-app regperf

  • edit $GRAILS_HOME/regperf/grails-app/conf/\*Data\* to point to your database
    class DevelopmentDataSource {
        boolean pooling = true
        String dbCreate = "create-drop" // one of 'create', 'create-drop','update'
        String url = "jdbc:postgresql://localhost:5432/regperfdb"
        String driverClassName = "org.postgresql.Driver"
        String username = "regperf"
        String password = "regperf"

  • copy the JDBC driver into $GRAILS_HOME/regperf/lib

  • copy the following Subscriber.groovy class into the $GRAILS_HOME/regperf/grails-app/domain directory
    class Subscriber {
        String name
        String address
        String email
        Date renewDate = new Date()
        String location

  • cd $GRAILS_HOME/regperf

  • grails generate-all Subscriber

  • grails run-app

  • point your browser to http://localhost:8080/regperf and click on the SubscriberController link

  1. Setup Faban and the Faban Regperf driver

  • Download and install Faban (look at the bottom of the page for download link and installation instructions)
    Note Faban has a driver and also a harness for automatically queuing and running multiple benchmarks but in these instructions we are going to run just the driver component and again assuming localhost

  • set the FABAN_HOME environment variable and add $FABAN_HOME/bin to your PATH

  • mkdir $FABAN_HOME/output (this is where the tests results will be stored)

  • download the Faban Regperf driver for Grails and unjar it into $FABAN_HOME/samples

  • edit the $FABAN_HOME/samples/regperf-grails/config/run.xml
    change the value for outputDir to match your $FABAN_HOME/output directory
    modify the number of simulated users by changing the value for <scale>1<scale> you will get 10 users for each scale factor e.g. scale 1 will give 10 simulated users
    Note the number of loadedsubscribers should match the number of rows you load into the database (this is set initially to 100)
    refer to the Faban documentation for other values to change in the run.xml

  1. Running the performance using the Grails runtime (jetty)

  • cd $FABAN_HOME/samples/grails-regperf/sbin
    assuming that the database and the grails application is still running execute ./ 10

  • start the rmiregistry (this runs in the background)
    ./ &

  • start and execute the benchmark
    Starts up all of the simulated users and writes results of the run into $FABAN_HOME/output/xx directory.
    The results can be conveniently viewed with a web browser.

  1. Deploying and testing the Grails application using Glassfish Java EE 5 application server

  • Assuming Glassfish is installed and running on localhost on port 8080 (the default install)

  • Grails allows the application to be packaged into a .war file and deployed to the application server, to create regperf.war
    cd $GRAILS_HOME/regperf
    grails war

  • Login to the browser admin console. By default this is http://localhost:4848

  • Choose “Deploy Web Application (.war)” from the Deployment column

  • Browse to the $GRAILS_HOME/regperf directory and select regperf.war

  • Click ok to deploy, and test for successful deployment by browsing to http://localhost:8080/regperf )

  • Use the and scripts as per step 4. above to load the database and run the tests (start the rmiregistry if it is not already running).  If you have Glassfish running on a different port or host adjust the run.xml to reflect this.


  • There are other values in the run.xml you might want to configure such as the rampUp, SteadyState and rampDown times.

  • One some platforms we tested deletes seem to run faster than inserts , so over the run the number of rows in the database may decrease. For this reason it is probably a good idea to load a few more rows in the database than is indicated by the loadedsubscribers entry in the run.xml

  • Running the performance tests with the driver, application and database all on one system is great for getting going but is of limited use for doing performance analysis of the application as it is hard to identify or separate the resource requirements of the 3 components when running on one box.

Disclaimer/Important note

Regperf and the simple performance testing tools offered here and on Damien's blog are designed to allow performance testing but do not constitute a “benchmark” per se. A benchmark has process, run rules, criteria for comparison and peer review or audit that can enable relevant performance comparisons to be made with a reasonable (or high) degree of confidence, so please feel free to use the tools to do performance testing of Grails, Rails but understand that this is “not a benchmark” it is just a few interesting and hopefully useful testing and tuning tools!

Monday Jul 16, 2007

Comments on JavaOne Session on Grails Rails and J EE

JavaOne session on Grails Rails and JEE

I noticed that the JavaOne session that Damien Cooke and I put together titled "Comparing the Developer experience of JEE 5, Ruby on Rails and Grails" has been commented on by a few folks firstly we noticed the "government computer news" and secondly and only recently we noticed an article from June 07  by the Grails community

I want to comment on these articles but first let me say that the intent of the session was to address Java developers like Damien and I that are not experts in both Grails and Rails but who want to know a bit more about both Rails and Grails and how they fitted with or competed against Java and J EE. One area we though would be valuable to explore was the performance characteristics of the frameworks, initially not in an exhaustive or robust way but to start looking at performance comparisons of these frameworks. Of course we realised saying anything about relative performance of the frameworks was going to get us offside with someone so we didn't try and tune anything or come to any stunning conclusions (yet) rather we wanted to just present some out of the box and very initial figures.

Comments on Government computer news

First up thanks to the author for the write-up it was much appreciated.
Overall I believe what is written is pretty accurate synopsis of what I said during the session importantly the author Joab Jackson highlights that I went to some lengths to point out that the tests were not benchmarks and that little (no) tuning was done on any of the platforms. What was also pointed out in this article is that somehow the wrong slides got loaded for our session and so I ended up presenting from something that still had typos and more unfortunately had the wrong performance graphs. Happily the JavaOne team has helped post an updated version of the presentation and so we now have the correct performance slides although I notice some typos are still there (mainly from the trademarking exercise)

Comments on the Grails blog entry

Well as per the invitation in the article Damien and I intend to contact the Grails team shortly. Certainly I think there are some things we can set straight and agree upon whilst other things we might have to agree to disagree. In the meantime hopefully the following responses to the points raised in the blog entry are at least a little helpful or clarifying.

  1. J2EE comes out smelling of roses unfairly
    Well probably not a surprise given our backgrounds and the fact that we have at least a little experience in J2EE and J EE ( mainly with performance testing) however in the session you will see that I actually really like Grails and I am quite excited about it. I am particularly excited about the prospect of Grails applications deploying to J EE 5 application servers and using JPA. I am certainly excited about the easy way a Grails application can be deployed into an application server such as Glassfish today but I am looking forward to when more of the underlying plumbing of the appserver can/will be used rather than bundled in the .war
    Do I believe that in general J EE 5.0 applications will scale much better than Grails application today bet (see SPECjAppServer related enties on this blog). However in future and perhaps as Grails uses more of the app server infrastructure I am excited about what might be achieved in terms of Grails performance (similarly we are also very interested in the performance achieved when deploying Rails apps to glassfish J EE application servers)

  2. After all, we have run benchmarks that seem to be fair against Rails, and Grails
    I apologise if I have this wrong but the only grails / rails performance comparison tests I could find before JavaOne were based on a single Mac (laptop?) and had no J EE implementation in the mix. Further it appeared the application server (or mongrel server for Rails) was on the same box as the driver /load generator and the database server.
    Damien and I figured it would be helpful if we provided a JEE version of an application as well as a rails and grails version and if we created the same driver for each application. We developed this with Faban and as stated ran just some initial tests , but we did make sure that we had the hardware available to scale up a little so we could look not just at latency but also at scaling (a J EE 5.0 strength). By having each component (app server driver, db) on a separate box we were able to also examine and do some initial reporting on resource utlisation.
    So did we publish a good comparison? Heck no, again we have only done very initial work , we ran grails on jetty and rails on webbrick and we certainly did not optimise the version of glassfish v1 we were using and by the way glassfish v2 yeilds some very significant performance benefits over the v1 which we used.
    Bottom line this is only stage #1 of grails vrs rails vrs JEE performance and scalability testing.

  3. There are so many typos in the document
    Well actually there are a lot of typos and the performance graphs were wrong, this was because somehow the wrong slides were loaded for the presentation in fact as I already mentioned some of the typos in this older version came from the trademarking process e.g. the reference to JavaScriptTM programming language.
    The reason I mention this is that it felt a little disingenuous that you would trash the work we did based on the typos which I feel were not necessarily the fault of Damien and I.

  4. 4 simple bullet points for creating a CRUD J2EE application
    Actually seeing might be believing on this one, so perhaps you have to watch the video to see the demo I did, which was to take the grails application data (same database) and then without doing any coding on the fly I produced an equivalent application in Java Server Faces/Java Persistence Architecture using NetBeans 5.5. It was dead easy and in many ways similar to the Grails tasks i.e. generate objects from the DB, then generate the MVC code to create the application. Here though it is not the framework doing all the generation but the tools and so where I can generate a simple grails application with a text editor with Java I really need tools. I believe that this is a difference in philosophy between J EE and Grails and yes I agree there is a lot more to learn up front for a J EE CRUD application but in many ways the tools like Netbeans that are available for J EE mean that some (many) Java tasks have become trivial and the entry point to being productive in J EE is far lower than it has been in the past. Importantly I would also note that Java Persistence Architecture is also simplifying life for many J EE applications developers! Anyhow I believe I demonstrated how easy it is to build a JEE / JSF CRUD application so I stand by this point.

  1. They say that one of the "problems" with Grails is that its made up of lots of other bits of tech
    I will stick by this and I really think it is one area that maybe the framework can be improved and simplified especially as/if things like JPA are embraced. The reason that all the different technologies are a problem is that Grails is attempting to be simple and dynamic but the fact that as a developer I have know about some many different technologies (best of breed or not) and learn more and more about them as my application complexity grows means that much of this simplicity is quickly lost. Also what about when you want to host a large application with a large number of users, the tuning of each of these at times unrelated technologies is going to be time consuming and a challenge and once again the deep understanding of each (or at least most) of the technology stack is going to be required to achieve a cost effective solution. I started to believe this might be an issue when I did "grails war" and saw the size of the .war file generated for a simple 1 table application.
    However as I say I think if/as things like JPA are embraced this will really help because there is lots of time and effort going into optimising JEE technologies and you can get a flavour for this at the SPECjAppServer2004 benchmark site.

  2. Then there are things like "Not clear how to regenerate if you have made changes to the controller/views"
    Well I was having trouble with this and I was talking about the experience I had again figuring I might not be the only one. In the session I believe I mentioned that this was more likely to be my lack of knowledge of Grails rather than the framework.

  3. Pah, oh well. Damien and Tom
    Well I have already confessed to not being a Grails guru, but I think Damien and I are fairly experienced developers and have good knowledge of performance measurement and I think  we are typical of Java developers who are starting to look at Grails and Rails so I would hope that the Grails developers would not write off much or all  of what we have presented especially when in fact we were actually trying to be helpful.

I hope the above clears up some things and provides at least basis for the points in the slides (without having to watch the session).  Damien and I are doing this investigation as we are interested in Grails and Rails but it isn't really our day job so it is taking a little while to get our tools posted. However we are really close now and I would welcome your feedback once we make the tools and a bit more testing information available.

Finally  I refer the Grails community to slides #40 and #41 which are titled "Grails cool stuff"  :-)





« July 2016