Thursday May 21, 2009

Introducing Twitter Data

My good friend and colleague, Jiri (Jirka) Kopsa, and I recently formulated a means of embedding semi-structured data in Twitter. We call it Twitter Data.

From the site,

Twitter Data is a simple, open, semi-structured data representation format for embedding machine-readable, yet human-friendly, data in Twitter messages. This data can then be transmitted, received, and interpreted in real time to enable powerful new kinds of applications to be built on the Twitter platform.

Here is an example Twitter Data message:

I love the #twitterdata proposal! $vote +1

We'd love to hear your feedback on our proposal. Catch us on, or on Twitter: @toddfast, @jirikopsa. If you tweet about Twitter Data, please add the #twitterdata hashtag so we can track your comments.

Friday Jun 06, 2008

zembly is here!

I've been working in stealth mode for the last 18 months, and today I'm able to show the results to everyone: is now live!

zembly lets you collaboratively create and host social applications of all shapes and sizes, including Facebook apps, meebo apps, OpenSocial apps, iPhone apps, Google Gadgets, embeddable widgets, and other social applications, using just your browser and your creativity.

More info later, but head on over to to sign up for the private beta before the invitations run out!

Monday May 12, 2008

Applications for the Masses by the Masses: Why Engineers Are An Endangered Species

I did a session last week at JavaOne 2008 where I talked about how application development, and the role of application developers, is changing. Here's the abstract:

When we engineers normally think of applications, we think of elaborate technology that takes highly trained developers weeks, months, or years to develop and debug. But our days are possibly numbered. In this era of light-speed boom and bust, the demand for technology is higher than ever and engineers and their traditional development techniques simply can’t keep up. What’s needed is “disposable applications,” applications so quick and easy to write that they are cheaper to throw away than to maintain and which the increasing numbers of casual technologists can consume and even create themselves.

Fueling this demand in recent months has been the release of high-profile web platforms (such as Facebook, Ning, Meebo, and others) that increasingly enable nontechnical users to compose mashups and other social and situational applications out of widgets and RESTful-style web services, all built by use of lightweight technologies and composed right from the browser.

This session describes how these factors are coming together to produce a new paradigm of application development in which hordes of 16-year-olds are in charge and software engineers are overwhelmed by the flood of applications created by tech-savvy novices. It also examines the roles of high technology versus technology for the masses and shows that they are actually complementary and a boon to engineers and nonengineers alike.


I presented to about 500 people over two sessions on Wednesday and Friday, and despite the fact that I could not show a demo, it was a rapt audience. Since then, coverage has been popping up around the Web.

Here are slides and links to articles and other material:



Tuesday Dec 19, 2006

Undermining Web 2.0

It's come to light that Google has shut down their SOAP-based search API. David Megginson has an interesting post on the impact of this:
Over on O’Reilly Radar, Brady Forrest mentioned that Google is shutting down its SOAP-based search API. Another victory for REST over WS-\*? Nope — Google doesn’t have a REST API to replace it. Instead, something much more important is happening, and it could be that REST, WS-\*, and the whole of open web data and mash-ups all end up on the losing side.
Many of the more interesting ideas around Web 2.0 and mashups assume that the data being mashed up will be available from somewhere, in a pure form that is consumable by the end user from the browser. So far, a few leaders like Google have made this possible, but there are strong economic incentives for others not to do this, as data is really the only "thing" of value anymore.

For example, Yahoo has a geocoding service (Google does not), but they have to pay a lot of money to buy that data from a provider and then make it free. Same goes for Google map data--they didn't generate that data, they just lease it from someone else. This is actually a pretty strong argument for server-side mashups (which may or may not be "enterprise mashups"), where partner agreements (i.e. restrictions on use and payment for service) can be enforced. I think in economic terms, we have a very long period ahead of us where only a very few large companies or governments will be able to or be interested in making large, useful databases available. There will be free alternatives, but like much of open source software, free data will be of lower quality than commercial versions.

To me, this issue is a major hurdle that undermines a lot of the interesting ideas that Web 2.0 visionaries (and zealots) are promoting. The socialist ideas underpinning open source software have been easy for corporations to accept because many of them recognize that the value is in something else besides the bits, like data, or usage, or support. To me, unless data is free, Web 2.0 is largely reduced to just some cool UI widgetry. Without data APIs, mashups, at least on the client, are not possible.

Now, having said all this, there is at least one alternative if data APIs are not available: screen scraping. This time-honored approach has taken on new life with Dapper, which in a nutshell is "Screen Scraping 2.0". While I'm guessing the Dapper guys are probably not happy about being referred to as screen-scrapers given their mottos like "It's like Legos" and "Unleashing the World's Creativity", that's effectively what they do, though in a way that's elegant and powerful, and geared precisely to Web 2.0 mashup use cases.

While technology like Dapper is very interesting, it has at least two limitations: first, I don't believe it does a good job of extracting data from sites with complex UIs (especially AJAX UIs), and second, armies of lawyers will probably sue them out of existence. If Google's strategy is really to replace direct data APIs with UI components which support their business model by serving advertising along with the data, it's unlikely that they will tolerate extraction of that data by third parties (in fact, their new terms of use explicitly forbid that). While I really like Dapper, and it may be useful for departmental or hobbyist use, I doubt it would be able to support Web-scale business.

Sunday Jun 26, 2005

Must-Attend Session: A Hitchhiker's Guide to SOA

Please join me for what will be the funniest technical session at JavaOne 2005: TS-7121, A Hitchhiker's Guide To SOA: Orchestrating Loosely Coupled J2EE Services with BPEL & BPMN. Not only do we have a very entertaining presentation (unlike most JavaOne sessions <yawn>), but we'll also be giving a very meaty demo and lots of info on SOA, BPEL, BPMN, and building loosely coupled Web services. If you want to learn about any of these topics or just want to be entertained after a long day of boring sessions, don't miss it.

Monday Jun 06, 2005

Colliding Worldviews: Kitty Hawk vs. Indigo

Over time, I've come to understand that there are two predominant worldviews, championed by Microsoft and Sun, concerning Web services and the role they play in relation to developers. Here are a few, high-level thoughts on the topic for your Friday-night enjoyment.

For Microsoft, Indigo and the WS-\* stack form the basis of their .NET and Longhorn OS platform. If you look at the role of each of the WS-\* specs embraced by Microsoft, it becomes clear that each one is primarily a formalization of patterns, conventions, best practices, and designs that have been in use for years, if not decades. There really is very little new about WS-\* except terminology and syntax. For Microsoft, however, these specs allow them to build an open architecture to support all the things they want to include as part of their platform. The key question to ask, however, is who are the users of this platform? I posit that Microsoft will be the primary consumer, at least in terms of direct interaction with the WS-\* technologies. For others, Microsoft will deftly hide all of these details (and complexity) underneath developer-friendly APIs and components that may or may not involve Web service message exchanges. Does the developer care? In Microsoft's worldview, the answer is no. For Microsoft, SOA is the platform.

Kitty Hawk and several Web service proponents at Sun think in quite different terms. Rather, we (meaning Sun, not necessarily me) tend to think in terms perhaps best described as SOA is the application. This is consistent with Sun's track record with J2EE. J2EE was a success precisely because it directly put the tools of distributed computing in the hands of developers. The fundamental premise of the old J2EE worldview, and perhaps now the Kitty Hawk worldview, is that distributed computing is something developers care about directly. This manifests in Web service terms as developers wanting to construct and controls precisely the messages sent over the wire. For productivity, they want APIs that are modeled closely on these specific tasks. Sun gives developers APIs to construct a SOAP message and invoke a Web service; Microsoft gives developers APIs to invoke a method on an object like any other, but which may be exposed as a Web service.

One might argue that Indigo is an ill-fated direction for Microsoft and that perhaps they are missing the boat. After all, their previous efforts to bring distributed computing technologies (DCOM) to the developer masses were, for lack of a better description, squashed by J2EE. J2EE was in large part a reaction to the fact that Microsoft, as the 800 lb. gorilla, either had nothing solving problems in the same domain, or hid everything so well that developers couldn't peek under the hood to go beyond existing limitations. J2EE made distributed computing available to everyone, and generally it has been easier (in terms of capability vs. complexity) and more transparent than the alternatives. One may also argue, however, that J2EE's prominence at the forefront of developers' minds is coming to a close, and that it is now an inelegant, complex morass that complicates more than complements developers' lives.

However, there is evidence to support Sun's worldview that developers care about the deeper levels of the platform. Today, armies of consultants and corporate developers are employed to build enormous distributed systems using J2EE as the foundation, and there is an enormous information market surrounding it. These developers live and breathe distributed computing as digested and presented by J2EE, and in general, it works. Clearly there has been something right about the approach Sun has taken with J2EE.

Back to Web services: Kitty Hawk seems inclined to treat SOA and Web services as both means and ends: Web services have value in and of themselves as the way to write applications. Indigo seems to shy away from this grand scheme and instead treats SOA and Web services as merely a means to a different end: helping developers write apps easily using the Microsoft platform. Is the worldview difference attributable then to motive? I don't think so. Sun has as much interest in perpetuating the Java platform as Microsoft has in perpetuating its .NET platform.

I think the answer is instead a matter of philosophy around developers and tools. Inarguably, Microsoft deeply understands software developers en masse (and arguably, Microsoft understands them better than anyone else). If there's one thing Microsoft does exceptionally well, it is satisfy developers by giving them terrific tools to make use of the Microsoft platform. As evidenced by Steve Ballmer's legendary "developers, developers, developers" speech (which I think, unlike some, was stunningly brilliant), Microsoft's commitment is to the developer and in giving him easy-to-use tools that make his job easier.

Sun, on the other hand, has been historically less committed to the idea of ease-of-use and tools as a means to Java platform adoption. Capability to write certain kinds of applications has been the driving force behind Java and J2EE, and Sun certainly understands the developers that write these particular types of applicationss quite well. If I'm being intellectually honest, I must also say that Sun has often fallen short when it comes to appealing to the broader developer community via tools. For many reasons, historical and otherwise, Sun has traditionally thought of tools and their place in the developers' universe quite differently than has Microsoft. If the desired end result is an application, tools are the primary means to this end for Microsoft. For Sun, APIs have traditionally been the primary means to this end (though this is rapidly changing).

So the big question: is there anything wrong with either of these two worldviews? Perhaps, but not necessarily; each confers both advantages and disadvantages. But, these differing worldviews lead to some interesting schisms. Take for example the notion of Web service conversation. In the Indigo worldview, conversation is session, likely tied to the notion of transport-level connection (or perhaps its analogue using WS-ReliableMessaging and WS-Transport) and spanning one set of interactions between client and service. In the Kitty Hawk worldview, conversation is a broader abstraction that gives additional meaning and context to business data across multiple sets of interactions between clients and services, independent of transport and session. Both perspectives are valid, but difficult to reconcile. They lead the broader Web services standardization effort down paths with different conclusions.

Which of these worldviews is superior, and which will inflame the minds of developers to drive respective platform adoption? I'm not going to try to answer such unanswerable questions. I don't think the future is yet written, and if the past is any indicator, there are reasons to believe that both worldviews are viable and can coexist. Perhaps they are best described as the opposite ends of a spectrum between which different developer communities swing like pendulums. Now that Sun and Microsoft are officially cooperating on Web service standardization and interoperability, I think the most important result of this discussion is simply understanding these different, perhaps reconcilable--but perhaps irreconcilable--perspectives on the shape the Web services universe should take.

Thursday Mar 03, 2005

Ontology-based Web Services

IBM has an interesting example of an ontology-based Web service over at alphaWorks. For mere humans, this means rule-based Web services that can infer conclusions based on a variety of facts (for example, message content, message exchanges, and/or partners) and dynamically modify their behavior in response. From the IBM article:
This demonstration facilitates enhanced discovery, composition, and transformation of business processes through semantic enablement and intelligent agents. The technology used in this demonstration can aid in the discovery of services, invoke an ABLE rules-based system, infer the processes needed, and compose the appropriate BPEL-based business processes.
This goes well beyond Web service orchestration using, say, BPEL. A process expressed in BPEL is still just a developer-defined program, albeit geared toward Web service interactions. However, the process could technically be expressed in various other programming languages like Java (though arguably not as easily). By contrast, using ontology-based services and basing service behavior on updatable rules instead of a rigid, pre-conceived scenario allows for construction of much more flexible, robust, and loosely-coupled services. Imagine: rather than rebuild and redeploy a service implementation to take into account new behavior, a developer could instead add a new rule annotated with behavior and pre- and post-conditions to an already-deployed service in order to change the way it functions. This is a much more lightweight approach, and allows for complex superposition of capabilities based on rule matching.

For example, let's say you need to update your service to take advantage of a new version of a message format used between certain partners. Using today's technologies, you would have to create a new version of the service and deploy it. Depending on your environment, you may need to deploy this in parallel to the existing version and notify your partners of the new service interface and endpoint. In the worst case, you would have to update the service and lose compatibility with partners that aren't yet using the new message format.

With a rule-based Web service, however, you could just deploy new rules that would match the appropriate facts (e.g. the message format or partner) and apply behavior based on conclusions from these rules. The same service could then concurrently handle both the older and the new message formats because it would dynamically match the appropriate behavior with the incoming message.

Ultimately, these ideas speak to the inevitability of going beyond imperitive specification of behavior to functional specification of behavior. I think it will be a long, long time before we see such approaches go mainstream, but even small steps in this direction can blend with existing development techniques and provide palpable benefits in the near term--especially with tools that lower the barrier of entry and give the typical, non-Ph.D.-holding developer the capability to understand and use these technologies with a minimum of overhead.

Monday Feb 28, 2005

SSDL - The SOAP Service Description Language

One of the big gaps in building real-world Web services is that there is no common description of the semantics of a service and its operations. WSDL tells a service consumer how to invoke a service, but not why or in what order. In other words, WSDL gives syntax, not semantics. As a result, this information must be transmitted out-of-band to service consumers, where it is likely to be misplaced, misinterpreted, or ignored. Just this month, the SSDL spec was released by folks over at with the goal of filling at least some of these gaps.

Friday Jan 28, 2005

Interoperability and Document-Based Web Services

Sameer Tyagi has an extensive article (part 1, part 2) over at called "Patterns and Strategies for Building Document-Based Web Services". Nice work Sameer; keep 'em coming.

Thursday Jan 20, 2005


Hardly a day goes by that I don't wish for a specification-less world, especially when it comes to the steady dribble of WS-\* specs that just won't stop. Tim Bray has made some amusing remarks about the situation (also check out Matt Powell's rebuttal), but if you ever wondered how the WS-\* specs get done, check out the Web Services Protocol Workshops Process Overview by our friends at Microsoft. Or, if you just want to know what the heck is going on, Matt's blog has a nice pointer to a Microsoft whitepaper, An Introduction to the Web Services Architecture and Its Specifications.

[Find more about at Technorati]

Thinking about Web services policy

While scrounging around for WS-Context info, I also found this nice blog entry on Web services policy from Greg Pavlik. Specifically, Greg makes several compelling arguments for why policy should not be part of a service's WSDL description. Thanks, Greg.

[Find more about at Technorati]

Some context on WS-Context

I stumbled across this meaty WS-Context presentation by Aad van Moorsel at the University of Newcastle. He has some interesting things to say about using context for multilateral conversation (which I discussed in an earlier article). Some related reading: the WS-Context spec; Web Services Composite Application Framework (WS-CAF); some comments from Greg Pavlik over at Oracle.

[Find more about at Technorati]

Friday Jan 07, 2005

Academic work on BPEL

Last year, Paul Brown at FiveSight conveniently listed a number of links to academic work done on BPEL.

[Find more about at Technorati]
[Find more about at Technorati]

Thursday Jan 06, 2005

Web Service Standards List

A handy list of Web service standards from BEA.

Friday Dec 17, 2004

Implementing ebXML's Reg/Rep

The following article, Implementing the ebXML Registry/Repository by Chaemee Kim, gives a decent overview of the ebXML Reg/Rep initiative by OASIS and its relationship to UDDI. Sun has been working on an open source implementation of Reg/Rep called ebxmlrr, part of the project.

[Find more about at Technorati]




« July 2016