Thursday Apr 06, 2006

I've been promoted to Chief Architect for Java Enterprise Tools

I've recently been promoted to the position of Chief Architect of the Java Enterprise Tools product group at Sun. Charles Beckham, the previous architect, will be moving on to direct the architecture for all Java tools, and I will be reporting dotted-line to him, as will the architects of NetBeans and Creator.

I don't yet know all the changes this new position will bring, but I know one thing: it's not going to help me find more time to blog. <g>

Wednesday Feb 09, 2005

'Tools for X': Frustrations of a Tool Architect

A Common Situation

As a tool architect, I often hear from various teams or other engineers that they want me to build "tools for X", where X is some specific technology. Such requests are well-intentioned but often misguided. In such requests, I believe there are two factors at work:

  1. Most engineers are impressed by tools
  2. Tool concepts are unknown to most engineers

Engineers generally love tools and understand their appeal, even if they can't always articulate what makes one tool more useful, easier-to-use, or more productive than another. I tend to think that this is the result of being primarily tool users and not tool authors; tools are a specialized domain like any other and require a certain amount of experience from an author's perspective in order to fully understand and articulate the opposing forces that must be balanced when designing a good tool.

Yet, the perspective of the non-tool experts is entirely understandable. In the broadest sense, a tool is just a piece of software specialized for making some task easier, and in this sense, all software qualifies as a tool to someone. Every professional technologist has used thousands of tools, and written perhaps hundreds of them in some form or another. Tools come in many distinct flavors—administration tools, analysis tools, application development tools, compiler tools, programming tools, Mars rover path plotting tools, and many more. However, for those unaccustomed to speaking about these specific variations, they are all just "tools". This leads to a fundamental disconnect between those immersed in tools and others who are not, and this is the primary motivation behind requests for tools for X.

I believe the main gap in understanding is that one can only understand a tool by understanding what general problem its user is trying to solve. This problem definition and the use cases behind it are usually implicit assumptions of someone when they ask for tools—they have a hard time solving some problem P, in problem domain D, using X. Naturally, they want a tool to help with X, but P and D often go either unmentioned or under emphasized. The hard part for those not accustomed to it is in understanding and judging the applicability and relevance of D, the problem domain, to a broader audience.

Let's take an example. I encounter a common perspective in my interactions with various Java technology groups at Sun. For years, Sun has been pumping out great specs and technologies at a phenomenal rate, and we have armies of brilliant people relentlessly expanding the boundaries of the state of the art. But most of these people are not tools experts (nor should they be). They are experts in other areas, but they all sense the need for great tools that enhance the value of their various technologies. But what are tools for X when X is JAX-RPC, JMS, or JBI? Is it possible to write tools for these technologies? Yes, absolutely. But the question we must ask is: should we?

A $4,000,000 Hammer?

How do we answer the question of whether we should build a tool for a specific technology? There are any number of equally valid axes on which we could measure value, but there is one constant goal for tools: adoption. Tools must be adopted in order to be useful; indeed, the only true value of a tool is in codifying a solution to something that it is otherwise hard to do over and over again into a repeatable, easy to replicate form. The value of a tool grows in direct proportion to the number of times it will be used. Thus, we can make a decision on the attractiveness of a tools-for-X request by evaluating whether the resulting tool will be of use to a sufficient number of users such that its development and maintenance costs are justified. Few people can afford to build a $4,000,000 hammer to pound $0.01 nails (though some part of me can't resist thinking what an awesome hammer it would be).

So, using this simple yardstick, do tools for, say, JAX-RPC the technology make sense? Given its current state and the way it is used in the industry, my answer is no. This isn't to say that tools for JAX-RPC wouldn't be of terrific value to some people, but it would be hard to argue that they would be terrific value to most people.

The primary reason is that JAX-RPC is just one technology out of hundreds that developers will use to build something that will solve a broader problem they face. The real problem that developers have is not in using JAX-RPC per se; although that may be difficult in itself, it's just a speed bump on the much-longer road to something bigger—the application. JAX-RPC will be just one small part of an application. The meat is not in tools for JAX-RPC, but in tools for using JAX-RPC and a large number of other technologies together in order to build an application.

The Reason Behind The Request

I tend to think that Microsoft panders to users when it uses the slogan, Where do you want to go today?™, but this question is actually the beginning of wisdom when applied to tools. Tool writers—and Microsoft is arguably at the top of the heap—must always start with this question when confronted with a request from someone asking for tools. It's dangerous to take such a request and execute on it blindly, as the result will almost certainly be a failure because the underlying reason behind the request for tools was not discovered.

In nearly all cases, people that ask for tools for X are actually asking for tools for application development, where application is a fluid concept. An application is really a solution to a heterogeneous, recurring set of problems, independent of any particular problem domain. An application is the bringing to bear of a solution to an instance of a recurring problem set. When this solution is canned for reuse, we call it a tool.

As an example, to engineers building say, JAX-RPC, one recurring problem they encounter is likely to be the building of JAX-RPC services (perhaps in order to test that all the other one-off, non-recurring problems they've been working through in the creation of JAX-RPC have been actually solved). This then becomes their problem domain, and naturally they gravitate toward thinking of tools for JAX-RPC as the specific problem domain which they need help solving repeatedly. However, JAX-RPC will be just one small part of another, broader type of application that includes many more parts, and many more technologies.

I'm not trying to be hard on the JAX-RPC people here; they are just serving as a convenient technology example. But the disconnect I've hopefully illustrated is at the core of quite common misunderstandings around what a tools organization is, and what it should be doing when building tools.

Give 'Em What They Want

As a part of the Enterprise Java Tools group at Sun, I believe our ultimate goal is to give people the tools that they want. And what do they want? Well, this is the enduring question in the tools business, and knowing the answer is a key part of running one. But, there are some pretty obvious clues to the right answer, which I think I can illustrate with a few specific examples: most people don't want administration tools, because administrators are few and far between; most people don't want JBI deployment tools because integration architects are rare (at least today); most people don't want disk format tools because the need to reformat a disk is infrequent; etc.

No, what most people want are application development tools. But what type of application? For the Java Studio Creator group, application means the traditional, Web-based, business application where information is collected from users via forms and processed information is then reported back to them. This is arguably the most prevalent type of application today, and the one that most people refer to implicitly when they use the term "application". This type of application and its related patterns are quite well-understood in the tools industry now, and the stratospheric ease-of-use and capability that tools for this application type have achieved is quite remarkable.

However, in the Java Studio Enterprise group, the definition of application is significantly different. Although there will always be a need for corporate applications, developers are beginning to focus less on building the same kinds of corporate applications over and over again. Instead, focus in the industry is now moving quickly toward Web services and service-oriented architectures used to solve advanced integration and B2B requirements. I contend that this is a new era, and now that corporate application development has been solved well within the Java realm, attention has turned to the next frontier: distributed applications that consist of heterogeneous, loosely-coupled services which collaborate in ways more complex than traditional client-server, human-to-computer interactions. These new types of applications are an order of magnitude more complex, and all the more difficult to build good tools for. For me, it's an exciting time to be building application development tools, and there's no time to waste building "tools for X".

Tuesday Jan 18, 2005

The Great Metadata Revolution -or- Why You Can't Keep Good Development Tools Down

This article discusses some consequences of the introduction of metadata annotations to the Java language, and how instead of obviating application development tools, annotations will in fact make them all the more valuable.

Annotations to the Rescue?

As most of you already know, the Java Development Kit 5.0 (or JDK 1.5 for the traditionalists) introduces language-level metadata via annotations, the general syntax of which are specified in JSR-175. In addition to several novel uses, annotations offer a partial solution to the overbearing complexity of building enterprise applications in J2EE and other application environments. For example, EJB 3.0 will use annotations to generate home and remote interfaces for an EJB from a single source artifact. Annotations allow what is generally known as metadata-driven programming, in which metadata is attached directly to programming constructs to precisely describe their semantics. Over time, the increasingly overused (and widely disliked) XML-based configuration technique most of us are used to will be replaced—or at least largely supplanted by—use of metadata specified via annotations directly in source code.

As the Java-based software industry (and the software industry as a whole) moves toward what ex-Microsoft employee and Intentional Software's founder, Charles Simonyi, calls Intentional Programming (IP), Java annotations will be a key element of reducing the overall complexity and amount of hand-coding needed to perform application development tasks. Annotations formalize the semantics of program elements, for the first time making it truly possible to define what it is that a developer knows but heretofore could not express formally in the actual application source. That is, metadata encoded in annotations help get knowledge out of developers' heads and put it directly into the source without requiring an irreversible translation to traditional programming constructs. The savings in terms of maintainability, portability, and transparency of Java applications will be very real, though it is perhaps still a bit early for most developers and architects to understand just how fundamentally annotations will change how they go about application development. Trust me when I say that annotations in Java will change everything you think you know about application development—they represent a true revolution for the Java platform.

With the introduction of annotations and metadata-driven programming for the Java platform, there are a few questions that are only just being asked by the general developer community. One of these is how annotations will affect application development, and specifically, how will they affect application development tools? Does the introduction of annotations mean that development tools will wane in their usefulness until we somehow arrive at a Nirvana where programming is so easy that all we need is the simplest of text editors to write the most complex applications?

I imagine there are a few people who would like nothing more, and for the small but outspoken group of developers who prefer to code every line by hand, annotations in their simplest form represent a potentially huge leap forward in productivity. However, these gains won't last, and thus the answer to the question posed above is a resounding no (which is fortunate for me, a tool architect in Sun's Enterprise Java Tools organization). Let me explain why.

Complexity Begets Metadata Begets Complexity Begets...

Metadata via annotations is one solution to the increasing complexity of application development. Rather than manage a number of separate artifacts (source code, metadata, etc.) that must be carefully kept in sync, annotations allow metadata to be attached directly to programming constructs. Although in the near term the same effective information may be stored in, say, an XML deployment descriptor and the first-generation of annotations now being defined (thereby making them semantically equivalent), language-level annotations are ultimately more capable, more powerful, and more convenient than metadata stored separately. In short, annotations, being easier to use, lead to increased use of metadata.

This is a a great result—increased use of metadata is a boon for application development because it makes more possible using less. But, it has at least one significant drawback: annotations become complex and unwieldy after a certain point. This is by no means an annotation-specific issue; any set of information becomes unwieldy and hard to manage when it passes a certain size. In the case of annotations, they work quite well when only a few are used and structurally they are relatively simple. However, the more numerous and precise the semantics of the elements being annotated, the more unwieldy annotations become as their number and structural complexity grows.

To that point, many people don't yet realize that annotations are themselves Java types, with most of the capabilities of traditional classes and interfaces. Furthermore, many people I talk to don't yet realize that annotations can effectively be arbitrarily complex, just like other types defined in Java. If you are familiar with JavaDoc-style attribute annotations (such as those used in XDoclet, whose values consist of just arbitrary string values), this is actually quite a difference. Annotations can be specified with most of the complexity found in normal Java classes, thus making them far more powerful and robust, but also potentially quite complex.

Although the nominal situation today is that there are few (if any) annotations used per unit of actual code, this trend will very quickly change as the complexity and adoption of annotations grows exponentially. The more that can be done by annotating something, the more annotations will be used. Our work here in the Java Studio Enterprise group has shown that it takes just a handful of annotations before they begin to overwhelm the element with which they are associated. It certainly can seem a bit ridiculous to see a one-line method with twenty lines of annotations preceding it, but in order to achieve the order-of-magnitude productivity gains that metadata-driven programming brings to the table, it is largely unavoidable and will become increasingly common.

Does this mean we're back in the same boat we were in before annotations were introduced? Frankly, yes and no. Keep in mind that once we begin talking about application development using metadata and annotations, we have suddenly begun talking about a fundamentally different type of application development. It may not yet be apparent, but we have entered a new era in which application complexity will increase by at least an order of magnitude, all made possible by extensive use of metadata. However, by using metadata (and eventually meta-metadata, meta-meta-metadata, etc.) application development itself doesn't need to become more complex, especially where tools can assist in limiting the impact of these quantum shifts.

Ultimately, application development is a zero-sum game—any gains in the ability to do more with less are almost immediately erased by the fact that even more complex problems become solvable. Thus, we've essentially just refactored the application development problem by introducing metadata-driven programming to Java. The tasks developers perform today will become easier in the short term, but these gains will soon be outstripped by the increasing complexity of next-generation applications. This is not something to lament; rather it is just the way of the world. For example, in the same way that 2nd-generation languages (2GLs) supplanted 1st-generation languages when they became too complex to use for generalized software development (a ridiculously short journey), 3GLs eventually supplanted 2GLs, and now Java plus metadata has effectively become a 3.5GL, featuring many of the benefits of a 4GL via annotations while retaining its general 3GL nature. Such is the technological treadmill that we perpetuate.

So, if annotations are ultimately not the long-term solution to application development complexity, what is? The answer should hopefully come as no surprise.

The Value of Development Tools

The oft-misunderstood and underestimated mission of development tools is essentially three-fold:

  1. Sanity - to make sense of a complex world on behalf of the developer
  2. Simplification - to reduce complexity and cognitive load for the developer
  3. Speed - to increase developer productivity

I assert that the Three S's as I'll call them (or S\^3 if you prefer) have been put to the test and proven true over and over again. Despite their near-term appeal as a way to reduce complexity in hand-written code, annotations will not decrease, but will in fact dramatically increase, the value of development tools.

This conclusion may initially seem counterintuitive, but upon reflection should be apparent. Apart from temporary improvements achieved by quantum shifts in methodology (of which the introduction of annotations is one), the long-term solution to application development complexity has always been the presence of good development tools, regardless of the technologies involved. Although development tools going forward using annotations will be significantly different than tools we have all used in the past, the fact that we now have annotations at our disposal really has no bearing on the fundamental value of tools because the level of complexity of the application development problem itself will essentially remain unchanged. We will instead now be able to achieve more for a given amount of effort, but the required level of effort will remain constant.

Ultimately, as a tool provider, I see annotations and metadata-driven programming as greatly beneficial to my business, and using them I look forward to helping developers tackle the next level of complexity in application development. Long live metadata, annotations, and good development tools!

[Find more about at Technorati]
[Find more about at Technorati]




« June 2016