Test Harness and tools
By mohanpraveen on May 18, 2007
As David pointed out in his blog, we do have different set of tests such as JCK, Unit, Regression, Performance etc to ensure OpenJDK quality at different levels of the the product life cycle. Out of all, we (Software Quality Engg Team) are directly involved in developing the Functional tests which is equivalent to a home owner visiting the site and inspecting if his/her new home meets all the requirements from a user's perspective. On the other hand, the unit testing (Done by the Development Engineering team) could be considered equivalent to a building inspector's visit who would ensure that the internal systems such as electrical, plumbing, foundation etc are of high quality.
Coming to the point, the crux of discussions are centered around what harness/framework we should be using for the tests that we open up for the community? When I say Harness (Tonga - internal name, being used internally within SQE), it is something that is loosely coupled with the tests as such and can execute tests, collect results regardless of how the tests are written. Whereas, a framework (Say Junit) is a tightly coupled one which gives you a pattern, an interface where you can implement the necessary portions of the interface to test a scenario. Now which approach we would want to take?
There are pros and cons in each of these approaches. The harness approach gives you the flexibility that you need in writing tests and you could potentially write tests for scenarios having different requirements, data, etc. For eg, the tests written for the client side (2d/awt/swing/deployment) do have requirements that vastly differ from the tests written for core components such as io, vm, libs etc primarily due to the user interactions and associated visual aspects. with the given flexibility, you could use the same harness to write and execute tests for client as well as core components. Of course, there will be certain rules on how the test should communicate it's status, messages to the underlying harness but not as stringent as a framework.
The framework approach sometimes gets limited to what it can cater to or we would require multiple frameworks to deal with varied requirements of different component areas of OpenJDK (Consider client, core in the above example). But on the other hand, it would be relatively easy to write a test since you already have a pattern that you have to follow and all the tests are guaranteed to be consistant. Consistancy helps in debugging, maintaining especially when dealing with a huge testbase.
We had this discussion many years ago within SQE and we decided to have transitioned to the harness approach from framework approach due to the flexibility that it offers and the varied requirements of different component areas in the JDK. That's when we chose the Tonga (internal name) harness. But now JDK is open to the community and we would like to revisit this and see what would be the best for the community as a whole. Your feedback is valuable. You can comment right here or in David's blog or send email to OpenJDK Quality mailing list.