Continuous Integration with embedded API and browser testing
By Ari Shamash on Nov 30, 2009
A "good" software release is one that has more features and fewer issues than the previous one. Yet, Agile software practices stress the importance of frequent and iterative software releases. Do these two statements contradict each other? For example, doesn't it make sense that if you want a quality release, you should spend more time testing software, so you can discover more issues, and hence have more time to fix them? This seems to imply that waterfall based software development practices make more sense than Agile based ones. What's going on here?
Turns out, for a variety of reasons, that the above statement is actually not true. This blog entry dives into one specific aspect of how Agile helps produce better software. There are others that I'll explore in other blog entries.
Through many personal experiences, as well as papers that I've read, it's actually best from a quality and time to market perspective to know about an issue as soon as possible after its been introduced into the software. There are many reasons for this, but the most intuitive one is this: the developer still has the context for the buggy code fresh in their minds, so they can find the issue and fix it faster. If months or years elapse, the developer will need longer time to regain the context (assuming the same developer is still even involved with the software), and have far less confidence that the fix actually improves the software without introducing even more issues. Comically, the same thing happens with computers: if a memory page is context switched out from main memory, it takes longer to retrieve it. Perhaps the human mind works the same way, where short term memory is somehow "faster to retrieve"?
Anyway, detecting issues as quickly as possible after they have been introduced into the code base seems like a good, logical idea. But how do you actually implement this? Agile gives us several different tools to help us here: this blog entry describes the tools that we, the Sun Software Library engineering team, has embedded into our continuous integration cycles, and some of our thoughts around them.Since videos are worth a thousand words, we first share our Continuous Integration Process in a screen cast. For those of you looking for slick screencasts, please look elsewhere - we're engineers, not marketing people. It's the content that matters to us, not the polish :-)
We use Hudson, for actually managing our Continuous Integration Process. Hudson polls our repository periodically, and kicks off a build if anything new has been checked in. We also do a build once a day overnight at a fixed time each night. This ensures that we get at least one build a day.
We have a custom Ant based top level build script, that goes and builds the roughly dozen or so NetBeans projects that together form the Sun Software Library. The screencast above runs this top level build script manually for demonstration purposes. We developers also run this script before we commit any code, to make sure that our build works. The top level build script follows these steps:
- Build all the projects. This is a basic check - does the code even compile?
- Some of the NetBeans projects have been modified to include embedded testing and/or automation tools.
- Our Web UI project build script has the following tools integrated into NetBeans' build.xml script:
- Automated sprite generation with smartsprite, also described here.
- Our CQS (CatalogQuerServer) Project, which is the component that implements our web services interface, has the following:
- Reloads the database with a known set of reference data (this data is needed and used by the various tests).
- If deploying to GlassFish, configure the necessary GlassFish resources.
- Deploys all the projects to an environment dedicated for continuous integration. This environment has both a Tomcat instance as well as a GlassFish instance in it. Even though we only use GlassFish for our production environment, we still validate our builds against both Tomcat and GlassFish - this doesn't cost us much, and has helped us resolve some issues.
- Runs a set of API functional tests, written using the Apache Commons libraries
- Runs schemaspy if the database schema has been changed
Is this all perfect? Of course not, nothing ever is. The following issues still exist:
- The automated testing that is integrated into our continuous integration environment only uses FireFox for testing. We have deployed several Windows based environments into our lab using virtualization technologies, and are intending to integrate IE based testing in an automated fashion, but we're not there yet.
- We still do some manual Q/A testing before each release, since there are coverage gaps in our automated tests (for example, our automated tests only cover our primary applications, not our customer help center).
- We still haven't fully analyzed the cost/benefit aspect of this. There is definitely a cost of implementation (though it's not that much, especially not since you can replicate what we've implemented). For us, we intuitively feel that the implementation has saved us more than it has cost us, but a more formal study may be worth while.
- There may also be alternative tools that are better/easier to use, we haven't really researched this.
PS While writing this blog, I ran across these blog entries, written by John Morrison, one of my colleagues at Sun, which describe testing in an Agile world. They are interesting reads, and restate much more elegantly and precisely my own personal thoughts on this matter.