On Monday (12/8/2008) the creation of the Web Services Test Forum (WSTF) was announced. At first glance the WSTF may seem, as some analysts seem to think, "yet another web services forum". On closer examination, though, I think you'll agree that the WSTF represents a radical and innovative departure from business as usual in the interoperability space.

Primary Use Case

There are many aspects to the WSTF, but here is the main story: Suppose you are a developer or an architect and you are trying to figure out how to integrate some systems, build out a system, etc.. Further suppose that you are planning on using web services in some parts of this project. Finally suppose that you know that these systems will involve two or more different web services implementations (for example, Apache Axis and Oracle WebLogic). Given this situation, the natural question to ask is "Will these different implementations interoperate across the technologies I intend use?"

State of Interoperability

The state of web services interoperability today is that, if you stick to the technologies covered by WS-I's Basic Profile (synchronous SOAP messaging, WSDL 1.1, etc.) and your target implementations comply with BP, it is unlikely that you will encounter any significant interoperability issues. That's the good news. Once you leave the safe harbor of these tried-and-true technologies, however, the picture is a bit murkier. What if you need to use some form of asynchronous communication, or need your messages to be integrity protected, etc.? The truth is that these technologies simply haven't received the necessary amount of real-world testing to flush out all their interoperability issues. This is particularly true when you "compose" two or more of these technologies in unique and interesting ways.

"But wait!" I hear someone say "What about the promise of interoperable, composable web service implementations?" To be honest, the promise of web services interoperability has been somewhat oversold by the vendors (my present and former employers included). I say "somewhat" because there is a degree to which the promise of web services interoperability has always been more abstract and architectural than a statement of guaranteed, out of the box interoperability. To clarify, consider the example of JMS. JMS does not define a wire-level message format. This means that, although there are adapters between the various MOM systems with JMS APIs, you can't have standards-based interoperability between one JMS implementation and another; there is a crucial piece of the architecture that is simply not defined by any standard or specification. Web services does not suffer from any such "missing pieces". The web services stack is, more or less, described in standards well enough to enable interoperable implementations of things like reliable, asynchronous, secure message exchanges.

So What's the Problem?

The problem lies in the level of detailed testing required to make sure that different implementations of the same specifications really do interoperate. Keep in mind that standards are a necessary but insufficient pre-requisite for interoperability. Regardless of how hard or long the authors worked on a standard (and our industry is notoriously impatient when it comes to standards), it is inevitable that there will be areas that are unclear, contradictory, or under-specified. The majority of these areas probably won't impact your project, but some of them might. The only way to tell for sure is to test the implementations and technologies with constraints and configurations that match your project.

How Is Interoperability Testing Currently Done?

Let's return to our primary use case. If interoperability is a potential issue, the last thing you want to do is leave it until the end of your project. Like any problem, interop issues can be addressed but they usually take a lot more time than "ordinary" bugs because it is not always clear which implementation is at fault (sometimes it's both, sometimes it's neither). You need to know early on if there are any interoperability problems in the products and technologies you intend to use so you can either get them fixed or re-factor your architecture to work around them.

There are some simple things you can do to figure out if your project may hit any interop issues. The obvious thing is to search the web for information about the interoperability of the products and technologies you intend to use. A good (but difficult to locate) source of this information is the results of any tests that were conducted as part of the standardization process for these technologies. There are a couple of problems with this approach, though. Some of the implementations you are working with may not have participated in these tests. The other problem is that there may not be any tests which cover your intended use of the standard. This is pretty much guaranteed to be true if you intend to combine multiple standards/technologies (e.g. WS-ReliableMessaging and WS-SecureConversation) since the tests used to verify a standard generally only test that particular standard. Finally these tests may not have been conducted using the version of the products or packages you intend to use.

So here's what you end up doing to perform the proper "due diligence" with regards to interoperability:

1. Define some test scenarios that cover the crucial aspects of your project. These should be simple enough to implement quickly yet still cover enough ground to reduce your risk of unexpected problems.
2. Get the appropriate versions of your underlying products or packages (e.g. WebLogic Server, Apache Axis). This may require you to obtain evaluation licenses, etc.
3. Implement the various components of your test scenario using the appropriate products and technologies.
4. Install and configure the servers, libraries, etc.
5. Deploy your test components and run your tests.
6. If your tests fail figure out whether the problem was (a) in your code, (b) in your configuration, (c) a real interoperability problem, or (d) something else.

It is important to note that various versions of this process occur in many other contexts. For example, a customer may require "proof of interoperability" as a pre-requisite to a particular sale. To demonstrate this interoperability a sales engineer will execute some version of the above flow at the customers site. Vendors do this sort of thing internally to test their products (Oracle has an entire lab devoted to this).

So what's wrong with this picture? Obviously it's expensive both in terms of time and resources. Most medium to small organizations will only be able to do this on a limited scale, if at all. When conducted as part of a sales engagement these activities tend to be done under strict time constraints (and thus not very thoroughly) and the resulting artifacts (installations, configurations, etc.) are seldom preserved. When individual vendors perform this process there is a lot of duplicated effort (each vendor installing and configuring the others products) and the test results generally cannot be shared with customers.

How This Is Done In the WSTF

So what does the WSTF bring to this picture? Basically, if you are an end-user member of the WSTF, you won't have to perform steps 2-6 (above). If your project architecture is fairly mainstream, there may be existing WSTF scenarios that already cover your problem and you may not even have to perform step 1. Let me explain why by describing how the WSTF works.

Scenarios

The WSTF process is based around the concept of a scenario. A scenario is a number of things rolled into one. Scenarios are based on the business problems and use cases of interest to the members of the WSTF. The fundamental dilemma in web services interoperability testing is "Given a nearly infinite combination of technologies, options, constraints, etc., which ones should I test for interoperability?" The WSTF answer to this question is "The ones the end-users care about." Along with a description of the business problem, a scenario includes an architecture that describes the technologies and standards that will be used to address the problem and how they will be used. Finally the scenario includes a set of test cases along with the artifacts (WSDL, XML Schema, etc.) necessary to implement these test cases.

Here's a rough example: Suppose there is an end user organization that needs to share documents with its customers. The customer's systems will generally reside behind a NAT/firewall combination and will not be externally addressable. The documents must be integrity protected and authenticated. The system should be robust enough to handle temporary network outages and hiccups. The architecture portion of the scenario describes how to address this problem with a combination of WS-Addressing, WS-MakeConnection, WS-SecureConversation, and WS-ReliableMessaging (if you are curious about what this architecture looks like, join the WSTF and we can work it out). Finally there are a number of test cases accompanied by the XML Schema and WSDL definitions necessary to implement them.

A key provision of the WSTF's charter is that any member can create a new scenario or participate in the discussion and development of an existing scenario. Scenarios can be created from scratch, imported from some other organization (mod any IP restrictions), or "forked" from an existing WSTF scenario. Vendors and other web services implementation providers are expected to "vote with their feet" and implement those scenarios that align with their customer base and technical direction.

Process Overview

Once a scenario has been defined, members of the WSTF may implement it using their products or open source projects. These implementations are deployed onto publicly available systems (maintained by the individual implementers) and testing is conducted in with other implementations in a crosswise fashion. Problems and issues are discussed on the WSTF mailing lists; the scenario may need to be clarified or re-factored during this process. If enough implementations of the scenario are produced and the implementers choose to do so, the scenario and its implementations can be made visible outside the WSTF by publishing it. Whether published or not, the endpoints that provide the scenario implementations are expected to be maintained indefinitely. This allows other members of the WSTF to perform regression testing, test new implementations, verify behavior, etc. without requiring the active participation of the implementer.

Win, Win, Win, Win

If all of this sounds too good to be true, think about what each of the parties gets out of it. End users get a way to test for interoperability problems in scenarios that they define without having to install any systems or write any code. They also get the benefit of design advice on the use of various standards; each scenario architecture can be regarded as a recommendation on how to best solve the business problem.

Vendors and other web services implementation providers get direct input from the end-users on the scenarios that interest them. This allows the providers to focus their testing efforts for maximum efficiency. The providers then get to distribute the cost of cross testing the scenario amongst all the implementers; each provider is only responsible for installing, configuring, and maintaining their own implementations.

If a situation in which both parties benefits can be referred to as a "win, win", clearly the WSTF is a "win, win, win, win, . . ."

Results

So what about tangible results? Here are some of the things the WSTF produces:

• A yes/no indication of interoperability for a given scenario: Getting back to our primary use case "Does A interoperate with B using C?", you get your answer.
• The endpoints that implement a scenario: The fact that these endpoints are long-lived means that you can re-check the interoperability results at any time. You also can create your own implementation of a scenario and check what you have done against any of the existing endpoints.
• A set of "findings" for the scenario: These are notes that discuss what was discovered during testing. For example, here are the findings for the Purchase Order scenario (perhaps the press release for the WSTF should have read "New Interop Group Finds Hole in JAX-WS"?). These findings can/should/have-been feed back into the WS-I and/or the relevant SDO.
• A set of guidelines or best practices on how to address the business case using web services technologies.

The Obvious Questions

Whenever I explain the WSTF to anyone a couple of questions always come up. In the interests of saving time I'll address them here.

What About the WS-I?

This may seem like polite, political spin but I view the WSTF and the WS-I as complementary. The WS-I's job is to address interoperability problems by writing specifications (called profiles) that constrain and clarify existing standards. The WS-I functions best when the interoperability problems it is addressing are already well understood by the working group participants. For example, the Basic Profile has been successful because it addresses the common interoperability problems in SOAP 1.1 and WSDL 1.1. These problems were well understood largely due to the efforts of the SoapBuilders community/mailing list (the WSTF was deliberately patterned after SoapBuilders). It is the WSTF's job to find interoperability problems but it is not its job to create new specifications or profiles to address those problems. The most the WSTF should do is (a) notify the WS-I or relevant SDO of the issue and (b) craft a work-around that avoids the issue.

In fact, the WSTF is a necessary pre-cursor to the successful functioning of the WS-I. It is true that, in recent years, the WS-I has gotten ahead of itself with regards to practical experience in the standards that is is profiling (e.g. the RSP) but, hopefully, the WSTF can come up to speed quickly enough to help the WS-I in this area. As of this writing, the WSTF has already found a number of issues that have been contributed to the WS-I's Basic Profile Working Group.

What About Plug Fests?

If you are familiar with web services interoperability you are doubtless familiar with the idea of a "plug fest". This is where one web services vendor invites a bunch of other web services vendors to a face-to-face meeting where they cross test their implementations over a set of scenarios. On the face of it, this seems a lot like what the WSTF does, but there are a number of key differences:

1. Since the event is held and sponsored by one company the focus is mainly on interoperability with the host and not so much on interoperability with other parties (i.e. it's more 1xN rather than NxN).
2. The scenarios are developed in advance by the hosting party and reflect the business interests and technical direction of the host. Participants are usually not allowed to contribute scenarios.
3. As part and parcel of the pre-defined scenarios, the tests and their success criteria are determined by the host and the behavior of the hosts' implementations are, by definition, correct. This is basically "interoperability redefined as compatibility". As a guest your implementation is "interoperable" to the extent that it is compatible with the hosts implementation, regardless of what the specification says, etc.
4. The endpoints that make up the test implementations usually do not survive for long after the plug fest.

The WSTF avoids these problems by creating a forum where any member can propose and create scenarios and any issues can be discussed and diagnosed in a neutral manner. Furthermore, the endpoints for a given scenario (which are listed on the WSTF website) can remain up for as long as the implementer sees fit.

What About Microsoft?

Microsoft has been invited to join the WSTF. Obviously it would be best for the entire web services community if they were to join and actively participate. While, to the best of my knowledge, they have not responded either affirmatively or negatively to these invitations Paul Cotton has stated that Microsoft sees no need for this kind of effort.

"We have not heard customer interest in the creation of new, alternative interoperability organization such as that recommended by the WSTF proposal," said Microsoft's Paul Cotton, group manager for Web services standards and partners, in a statement. "Given the incredible industry-wide momentum and leadership of WS-I, Microsoft has chosen to continue to invest in driving advances in Web services interoperability through existing means. We believe that WS-I provides a proven and open organization and process that best suits our customers’ needs." [1]

Key Principles

Earlier I claimed that the WSTF represented a "radical departure" from previous interoperability efforts. "What's so radical?" you may ask. Here are the key points:

1. Low barriers to participation: The WSTF has no dues primarily to encourage as many people and organizations to participate as possible. If there is a mom-and-pop consulting shop with a good idea for a scenario, the WSTF wants to hear it. Having dues also requires you to have some sort of board to watch over the money which brings us to our next point.
2. No centralized control: Anyone can create a scenario and implementers can implement it (or not) as they wish. Having a board or other type of oversight committee inevitably leads to attempts to control what should and shouldn't be tested. The WS-I has shown that, if you allow vendors to veto the testing of technologies they don't like, you end up in a stalemate where little testing gets done.
3. Interoperability by consensus: Like the standards on which it depends, interoperability must be based the a consensus of the community. Interoperability defined as compatibility with a single vendor's (or subset of vendors') products only creates islands of non-interoperability.
4. Real-world use cases: Interoperability testing must be constrained and directed by the business cases and scenarios of interest to end-users. Any other approach amounts to "boiling the ocean" and the ocean of web services interoperability is vast and deep.

In a certain sense, the WSTF is an attempt to apply some of the tenets of open source to the process of interoperability testing.

Summary

If you have experience in the area of interoperability you know that these problems are never going to go away. As long as we continue to standardize new technologies there will be the inevitable gaps, misalignments, and misunderstandings that lie at the root of all interoperability problems. The WSTF is founded on the simple yet radical notion that the only effective way to discover and address these problems is to put the end-users in the drivers seat and have them direct the testing efforts. Only time will tell if this approach is workable but, in the meantime, I'm looking forward to working with those of you willing to join us in getting this stuff straightened out.

[1] Krill, Paul "Is the WSTF one Web services forum too many?" InfoWorld 8 Dec. 2008 <http://www.infoworld.com/article/08/12/08/Is_the_WSTF_one_Web_services_forum_too_many_1.html>.