will be a week of contrasts for me. I'm currently in Cannes-Mandelieu, France
where I'm attending the annual Technical
Plenary meeting of the World Wide Web Consortium
(W3C). The W3C, of course, is the organization that's responsible for defining
the standards and protocols on which the World Wide Web is built. Like most
standards organizations it operates through a relatively small permanent staff
and a much larger number of volunteers, most of whom (like myself) are sponsored
by their employers to participate in standards development activities.
The organization functions primarily through Working Groups, each of which
is responsible for defining a particular standard. (The W3C's Working Groups
are the equivalent of the JCP's Expert Groups.)
(If you're interested, you can compare the two organizations' process documents
here and here.)
My involvement in the W3C goes back several years to the release of JAXP
1.2, which revised the spec to require that XML parsers incorporated into the
Java SE platform support XML schema.
Oops! Suddenly my team became responsible for verifying that XML parsers conform
to specifications for a language that is arguably more complex than the Java
language itself. Since we'd spent more than 50 staff-years developing tests
for the Java language it seemed extremely unlikely that we'd be able to develop
a comprehensive conformance test suite for XML Schema.
What I wanted to do was to simply revise our conformance requirements to say
"plug in any parser you want so long as it passes version x.y of the W3C
XML Schema conformance test suite." I looked for such a test suite, and
didn't find one. The XML Schema Working Group had developed a large body of tests, but these were not packaged into a well-documented test suite
with the kind of rigorous conformance requirements that we associate with our own TCKs. Consequently it would have been very difficult for our licensees to run the
W3C tests and to definitively determine whether or not their parsers conformed
to the specs.
As we have learned over the years, there's much more to conformance
testing than developing a collection of tests. These must be packaged together
with a test harness, documentation, and conformance requirements into a version-controlled test suite. There must be a well-defined process for users of the test suite to challenge the validity of a test and for invalid tests to be excluded or updated.
In the absence of such a test suite our only option seemed to be to incorporate
the existing W3C Schema tests into our own TCK. (In a presentation
I gave at last year's Technical Plenary on the re-use of test suites I called
this testing by value rather than by reference.)
We did eventually incorporate several thousand W3C tests into the JAXP TCK
and more recently we donated back to the W3C several hundred additional tests
that we had developed for the JAXP TCK. With the intent of promoting more of
this kind of collaboration between Sun's TCK developers and the W3C, and in the
hope that we could influence the W3C's conformance test developers in a direction
that would make it easier for us to reuse their work in the future, I joined
what was then the QA Working Group (we
have since re-chartered as an Interest Group).
This group has focused on developing guidelines for and promoting best practices
in the development of specifications and conformance tests within the W3C. A
complete list of the documents we have published can be found here here.
The Test Development FAQ,
in particular, addresses the difference between a "collection of tests"
and a "test suite."
The Interest Group met yesterday and today to discuss how we should focus our
efforts during the coming months. As always we have much more work to do than
we have people to do it, so we would very much welcome assistance from people
in the conformance testing community. Whether or not you're able to help, please
join our mailing list
and read our blog.
Tomorrow (Wednesday) is the actual Technical Plenary day, during which we all
get together for a series of presentations and then on Thursday I'm leaving
for a two week visit to St. Petersburg, Russia, where most of my team is based.
More on that later...