OpenJDK 6: Some more regression test results for b11

To address a configuration issue with the previously reported regression tests results for OpenJDK 6 b11, I reran the jdk tests with the display set to a virtual framebuffer and 77 more tests pass. In summary now:

  • JDK, 2,989 tests pass, 32 tests fail, 22 tests have errors.

Note that the jtdiff command, included as part of jtreg, can be used to compare the output of different regression test runs.

Comments:

3k tests seems low for the entire JDK. Curious, are code coverage metrics obtained during the run? It would be a great way to identify potential gaps that the community can address.

Posted by Abraham Tehrani on August 20, 2008 at 08:14 AM PDT #

Abraham,

Internally the full JDK code base is now split into a number of component workspaces/repositories: corba, hotspot, jaxp, jaxws, langtools (javac and friends), jdk, etc. The 3k tests are in the core jdk workspace; there are another 1,300 tests in langtools and a few regression tests in the other components. In addition, there are also closed regression tests that have not been open sourced for a variety of reasons as well as other test suites, like the JCK tests. There are no externally published code coverage numbers from running these tests, although I agree that would be useful information to have. Over time in developing the JDK, we've had increasing expectations of what it means to have acceptable regression tests so newer parts of the platform will generally have more extensive regression test coverage than older ones.

Posted by Joe Darcy on August 20, 2008 at 09:30 AM PDT #

How do you run the tests? It seems some of the errors you are seeing are because you run tests marked with @ignore. In IcedTea the default test run uses -v1 -a -ignore:quiet which on my local machine gives:

HOTSPOT: Tests that passed 5, Total: 5
LANGTOOLS: Tests that passed 1342, Tests that failed 1, Total: 1343
JDK: Tests that passed 3182, Tests that failed 50, Tests that had errors 3, Total: 3235

Posted by Mark Wielaard on August 20, 2008 at 09:24 PM PDT #

Mark,

For these runs I just used "-v1 -a", but it would be more informative to really ignore the @ignore tests as you've done using "-ignore:quiet" so I'll report the builds' test status using that option going forward.

Posted by Joe Darcy on August 21, 2008 at 07:11 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

darcy

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
News

No bookmarks in folder

Blogroll