Whack a Mole Testing

Of late I seem to have entered a Twilight Zone game of Whack a Mole with the jdk tests. It appears that the odds that a test can fail on some particular OS or machine, with or without any jdk change is higher than I ever thought possible. Very frustrating. Why is that? I have a list of possible contributing factors:

  • Some tests are just downright unpredictable and need to be fixed.
  • Minor differences in the environment variable settings can cause failures.
  • Minor differences in the machine or OS configurations can cause failures.
  • Some tests are rarely run on all platform (OS/ARCH) combinations.
  • The ramification of any jdk change is often poorly understood, and all jdk tests, on all platforms are rarely run.
  • The use of jtreg -samevm can influence the stability of a testrun, if any test is not 'samevm' safe.
  • Using the same machine and/or same user to run multiple sets of tests can also influence the stability of a testrun, if any test is using shared resources (like port numbers or shared directories like /tmp).

Maybe I just have too much Fuzz (unpredictable environment) in the test runs? I can certainly nail down many of the above items to increase stability, but until I have some kind of Hudson or continuous build/test system watching every changeset, this will be a difficult task. And to be effective, it would need to build and test all possible platforms, a platform list that seems to be growing lately.

I found this cartoon from the health care industry (that's scary), anyway I thought was appropriate for product releases too:

If I knew how to draw cartoons, I'd draw one for the testing matrix.

-kto

Comments:

Post a Comment:
Comments are closed for this entry.
About

Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.

Search

Archives
« April 2015
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today