Hudson as a tool for testing? Yes!

Lots of folks in NetBeans started to use Hudson as a tool for continuous integration, or just for building NetBeans from various clones, or to build test, whatever you can imagine. Just take a look at Deadlock machine and you will see a plenty of NetBeans related jobs. We (NetBeans QE) recently got an idea to utilize Hudson as our testing infrastructure. Yes, NetBeans have a lot of automated tests that are being ran daily and test results are evaluated for potential problems. So you have tests, you have machines, but you still need an infrastructure to connect this together. We chose Hudson, because it's a great tool, and what's really cool - is extensible by various plugins (available on Hudson's web, or you can write your own if you wish). The basic idea was to use Hudson as "master-slave" architecture, where you simply set up on master what do you want to run, and master distributes the work to the slaves, that execute the jobs. More detailed approach to this - configure on master which tests do you want to run on which platforms, and they will get executed on appropriate slaves. And getting one level lower - to achieve this, you have to have something we called generic Ant script, which will provide all the stuff needed to execute test:
  1. Download IDE - the IDE build which will be tested
  2. Download NetBeans Test Distribution - Test Distribution contains binaries of all tests from NetBeans repo.
  3. Run selected tests - Specify which tests from Test Distribution will run in tested IDE
So i wrote and Ant script with plenty of useful targets to provide this functionality (Yeah, by this I learned a lot how to write Ant scrips:-) ). This Ant script is generic, and all the stuff like where to take IDE build, where to take Test Distribution, which tests run, what to do with results etc. are controlled by passing corresponding properties. Once we have this working, the only thing we have to do, is to distribute the Ant script to machine we want to run tests on, and provide some better UI than command line for the main properties, which control the test execution. Therefore me and Max Sauer extended Hudson by Test Run Plugin (sources are not public right now), that enables Hudson to copy our generic Ant script on target slave machines and extends job configuration with our panel for setting properties for testrun. With all this stuff working, people can simply log in on master, select one or more slave machines on which will tests run, configure which build will be tested by which test (through nice GUI :-) ), run the job and take a nap while watching the progress of tests execution on target machines. When test run is finished, you get a nice overview of test results (yes, built-in feature of Hudson).
The main benefits we take from Hudson (aka features we really like):
  • Master-slave architecture - you can have various machine with various architectures/OS added into testing farm.
  • JUnit results processing - nice HTML, results trends, charts,...
  • Scheduler - triggering of testrun by another job (which can listen e.g. if new production build is available), or "cron-like" settings.
  • Email notification - send email if tests do not pass.
  • SCP plugin - we can send test results to remote machine that can parse them and store into DB.
  • Authentication - simply configure and point Hudson to your LDAP server, really out of box functionality.
  • Multiple JDK installations - specify location where are JDKs located and voila - you can run whole matrix of tests - one axis is JDKs, second is platform.
  • ...many many others:-)
And conclusion? Hudson is not only contentious integration tool, it can be almost whatever you like.

Nice work! Please publish test results and code coverage outside SWAN ( sun internal network). Do you have any plan to extend binary test distribution to build BTD for modules suite project?

Posted by zajo on červenec 22, 2008 at 02:36 odp. CEST #

Dik Zajo:-) This is some kind of T4U replacement, quite new, we're on the start. We are currently establishing regular daily test run for qa-func tests - you aren't be probably interested in those, are you?:-) Code coverage? I can ask if results can go public...
BTD for module mean that there would be an action for module suite project that would build tests for all modules in suite?! That would be interesting! However..I'm not aware of such plans..
Btw. there were some changes in BTD during migration from Xtest to simpletest - did you notice? I'm going to blog about this soon...

Posted by TM on červenec 22, 2008 at 04:29 odp. CEST #

Yes, I already know about simpletest.

Posted by zajo on červenec 23, 2008 at 06:22 dop. CEST #

I'd be very interested to know more about how you're using Hudson to help with build and testing.

I am knowledgeable about Hudson and the NetBeans Platform, but it's great to learn from others especially since the switch from XTest to simpletests has left very little documentation to follow.

Posted by Tom Wheeler on červenec 23, 2008 at 10:48 dop. CEST #

Hi Tom,
I'm not very skilled in Hudson utilization for building - I am able to set up regular build of NetBeans (or probably whatever is buildable with Ant) - you are probably able to set up this too. However Hudson is used for more advanced scenarios, e.g. in Platform/core dev team as some kind of gate - it propagates changes from team repository into main repository if some conditions are fulfilled (tests are passing). If you are interested in details, take a look at scripts in nbbuild/hudson in hg repo. There are scripts for some jobs running on Deadlock's Hudson.
If you are interested in Hudson as a tool for testing, connection of Hudson with binary test distribution, let me know, there can be more blog entries about this topic;-)

Posted by TM on červenec 23, 2008 at 11:21 dop. CEST #

I would like to thanks Tomas for writing this here as well as great job of Tomas & Max on this "T4U replacement" in past few months ... the current plan is to move Test Run plugin sources into nb-quality repository, once the support we need from Core Hudson is integrated into Hudson standard distro. Stay tuned ...

Posted by Marian on červenec 25, 2008 at 03:41 dop. CEST #

Yes, I would be very interested to know more about testing with Hudson, especially as it relates to building NetBeans and/or platform applications. I am doing this now but there's always room to learn something new.

Posted by Tom Wheeler on červenec 31, 2008 at 03:43 odp. CEST #

I am fan with Hudson since I was read that uses it : More than 6500 unit tests and more 500 IHM tests.

We try to use it as our continuous integration tool but with maven. Using Junit is not possible now for several reasons but we try to integrate some plugins such as PMd, find bugs ....

At the end thanks for this Blog, As we haven't decide yet to use Hudson or other tool (cruise controle...) I can this blog in order to convince the team to use it :)

Posted by slim ouertani on srpen 04, 2008 at 05:18 odp. CEST #

Thanks for comments:) There will be more blog entries about Hudson (I'll try to spare some time) and how to setup some larger infrastructure based on Hudson...

Posted by TM on srpen 06, 2008 at 07:26 dop. CEST #

I run a test which is working on my desktop, but when I run it in Hudson it's failed because it can't connect to the main database
can somebody give a help?

Posted by Hardaway on březen 05, 2009 at 01:54 dop. CET #

Post a Comment:
  • HTML Syntax: NOT allowed

Tomas Musil


« červenec 2016