Whack a Mole Testing
By kto on Jan 11, 2010
Of late I seem to have entered a Twilight Zone game of Whack a Mole with the jdk tests. It appears that the odds that a test can fail on some particular OS or machine, with or without any jdk change is higher than I ever thought possible. Very frustrating. Why is that? I have a list of possible contributing factors:
- Some tests are just downright unpredictable and need to be fixed.
- Minor differences in the environment variable settings can cause failures.
- Minor differences in the machine or OS configurations can cause failures.
- Some tests are rarely run on all platform (OS/ARCH) combinations.
- The ramification of any jdk change is often poorly understood, and all jdk tests, on all platforms are rarely run.
- The use of jtreg -samevm can influence the stability of a testrun, if any test is not 'samevm' safe.
- Using the same machine and/or same user to run multiple sets of tests can also influence the stability of a testrun, if any test is using shared resources (like port numbers or shared directories like /tmp).
Maybe I just have too much Fuzz (unpredictable environment) in the test runs? I can certainly nail down many of the above items to increase stability, but until I have some kind of Hudson or continuous build/test system watching every changeset, this will be a difficult task. And to be effective, it would need to build and test all possible platforms, a platform list that seems to be growing lately.
I found this cartoon from the health care industry (that's scary), anyway I thought was appropriate for product releases too:
If I knew how to draw cartoons, I'd draw one for the testing matrix.