What's a TCK when it's at home?
By Patrick Curran on Feb 01, 2006
(As my mother might have asked...)
You all know about the Java technology releases (the platforms, and various optional packages). You might not know that each technology release is also accompanied by a TCK (Technology Compatibility Kit), which is a test suite designed to verify that implementations of the technology conform to the specifications.
The JCP requires three simultaneous deliverables: the specification, the TCK, and the Reference Implementation. Each of these depends on the others, as this diagram tries to explain:
The specification defines the functionality of the Java technology and how it may be accessed from Java programs. The Reference Implementation (RI) is a proof of concept that the specification is actually implementable. It doesn't have to be a commercially available product, but it does have to correctly implement the complete specification. The TCK verifies that the RI does in fact conform to the spec. Conversely, the TCK is tested by running it against the RI. Finally, the process of creating the TCK helps to clarify the specification.
To the extent that our specifications are of high quality (the public development and review process help to ensure this), the TCK and the accompanying Compatibility Requirements (more on these in a future posting) promote compatibility and interoperability between implementations.
My team is responsible for developing the TCK for the Java SE platform. (Just to confuse you further it's called the JCK, or Java Compatibility Kit). We also develop TCKs for various technologies that start out as standalone "optional packages" but that we expect will be incorporated into the platform at some point (for example, JAXP and JAXB).
The interdependence of these three components complicates the TCK development process. If we start too soon both the spec and the RI will still be unstable, and we'll have to make many changes as they evolve. If we leave it too late we won't be able to develop enough tests to provide reasonable coverage of the spec before the implementation ships. We can't execute our tests until we have an implementation to run against, and once we do so we can never be quite sure whether test failures are due to errors in the implementation or bugs in the tests. Somehow it all comes together eventually, but the journey can be quite a challenge.
The first step in developing conformance tests is to review the spec carefully. During this process we provide feedback to the spec authors where we find ambiguities or where we feel that clarification is needed. We identify "assertions" (statements of specific behaviour that potentially can be tested) within the spec. Deciding which assertions to test is something of a black art. It's usually not possible to test everything (though we come pretty close for critical specifications such as the Java Language and the Java Virtual Machine specs). We try to focus in areas where the likelihood or consequences of incompatible implementations are greatest.
We take this very seriously. Over the past ten years we've spent approximately 250 engineer-years (really!) developing the JCK. The JCK currently consists of about 80,000 test cases and several million lines of code. We run the entire test suite against every build of the platform and on multiple configurations. In addition to ensuring compatibility, it forms a critical component of our quality assurance processes.
Unlike Java SE itself, we don't release snapshots of the JCK as we're developing it. We do however release it for public review at the end of the development cycle. If you want to take a look at the last published version, see here. You might also want to take a look at this blog from Graham Hamilton in which he discusses the licensing terms that apply to that release.