W3C Technical Plenary, 2006
By Patrick Curran on Feb 27, 2006
This will be a week of contrasts for me. I'm currently in Cannes-Mandelieu, France where I'm attending the annual Technical Plenary meeting of the World Wide Web Consortium (W3C). The W3C, of course, is the organization that's responsible for defining the standards and protocols on which the World Wide Web is built. Like most standards organizations it operates through a relatively small permanent staff and a much larger number of volunteers, most of whom (like myself) are sponsored by their employers to participate in standards development activities.
The organization functions primarily through Working Groups, each of which is responsible for defining a particular standard. (The W3C's Working Groups are the equivalent of the JCP's Expert Groups.) (If you're interested, you can compare the two organizations' process documents here and here.)
My involvement in the W3C goes back several years to the release of JAXP 1.2, which revised the spec to require that XML parsers incorporated into the Java SE platform support XML schema. Oops! Suddenly my team became responsible for verifying that XML parsers conform to specifications for a language that is arguably more complex than the Java language itself. Since we'd spent more than 50 staff-years developing tests for the Java language it seemed extremely unlikely that we'd be able to develop a comprehensive conformance test suite for XML Schema.
What I wanted to do was to simply revise our conformance requirements to say "plug in any parser you want so long as it passes version x.y of the W3C XML Schema conformance test suite." I looked for such a test suite, and didn't find one. The XML Schema Working Group had developed a large body of tests, but these were not packaged into a well-documented test suite with the kind of rigorous conformance requirements that we associate with our own TCKs. Consequently it would have been very difficult for our licensees to run the W3C tests and to definitively determine whether or not their parsers conformed to the specs.
As we have learned over the years, there's much more to conformance testing than developing a collection of tests. These must be packaged together with a test harness, documentation, and conformance requirements into a version-controlled test suite. There must be a well-defined process for users of the test suite to challenge the validity of a test and for invalid tests to be excluded or updated.
In the absence of such a test suite our only option seemed to be to incorporate the existing W3C Schema tests into our own TCK. (In a presentation I gave at last year's Technical Plenary on the re-use of test suites I called this testing by value rather than by reference.)
We did eventually incorporate several thousand W3C tests into the JAXP TCK and more recently we donated back to the W3C several hundred additional tests that we had developed for the JAXP TCK. With the intent of promoting more of this kind of collaboration between Sun's TCK developers and the W3C, and in the hope that we could influence the W3C's conformance test developers in a direction that would make it easier for us to reuse their work in the future, I joined what was then the QA Working Group (we have since re-chartered as an Interest Group).
This group has focused on developing guidelines for and promoting best practices in the development of specifications and conformance tests within the W3C. A complete list of the documents we have published can be found here here. The Test Development FAQ, in particular, addresses the difference between a "collection of tests" and a "test suite."
The Interest Group met yesterday and today to discuss how we should focus our efforts during the coming months. As always we have much more work to do than we have people to do it, so we would very much welcome assistance from people in the conformance testing community. Whether or not you're able to help, please join our mailing list and read our blog.
Tomorrow (Wednesday) is the actual Technical Plenary day, during which we all get together for a series of presentations and then on Thursday I'm leaving for a two week visit to St. Petersburg, Russia, where most of my team is based. More on that later...