By jjg on Apr 01, 2013
Now that OpenJDK has an improved new build system, it is time to re-examine our test infrastructure with a view to gaining similar improvements for writing and running tests.
We have an ongoing problem with test reliability. Part of the issue is that we write our tests in Java, which apparently, and regrettably, has bugs in it. (Otherwise, why would we be testing it?) With the recent progress in Java scripting technology over the past couple of years, we should convert all our tests to use a scripting language. Although, we currently specify the use of Bourne shell, there are too many evolutionary variants of that, and so it is proposed we instead use Perl for all our test code. Of course, there are different versions of Perl, and so users will have to run a configure script ahead of time to determine if they have enough versions of Perl available on their system, and to recommend how to compile additional versions if necessary.
From early days, JavaTest, jtharness and jtreg have embraced the web, providing HTML reports, support for an HTTP server to be available while tests are running, and servlets to display test results in full color, eliminating the need for thousands of words. We should build on those ideas and embrace web-based test execution and reporting using a new PHP-based back-end for jtreg.
Finally, we should rewrite jtreg itself in Python. This will facilitate easy integration into our work flow as a Mercurial extension. Initially, this can be provided as "hg test" but the long term goal is to integrate jtreg functionality into jcheck, so that all appropriate tests are run automatically before any code can be pushed to a jcheck-controlled forest.