The Great Metadata Revolution -or- Why You Can't Keep Good Development Tools Down
By toddfast on Jan 18, 2005
Annotations to the Rescue?
As most of you already know, the Java Development Kit 5.0 (or JDK 1.5 for the traditionalists) introduces language-level metadata via annotations, the general syntax of which are specified in JSR-175. In addition to several novel uses, annotations offer a partial solution to the overbearing complexity of building enterprise applications in J2EE and other application environments. For example, EJB 3.0 will use annotations to generate home and remote interfaces for an EJB from a single source artifact. Annotations allow what is generally known as metadata-driven programming, in which metadata is attached directly to programming constructs to precisely describe their semantics. Over time, the increasingly overused (and widely disliked) XML-based configuration technique most of us are used to will be replaced—or at least largely supplanted by—use of metadata specified via annotations directly in source code.
As the Java-based software industry (and the software industry as a whole) moves toward what ex-Microsoft employee and Intentional Software's founder, Charles Simonyi, calls Intentional Programming (IP), Java annotations will be a key element of reducing the overall complexity and amount of hand-coding needed to perform application development tasks. Annotations formalize the semantics of program elements, for the first time making it truly possible to define what it is that a developer knows but heretofore could not express formally in the actual application source. That is, metadata encoded in annotations help get knowledge out of developers' heads and put it directly into the source without requiring an irreversible translation to traditional programming constructs. The savings in terms of maintainability, portability, and transparency of Java applications will be very real, though it is perhaps still a bit early for most developers and architects to understand just how fundamentally annotations will change how they go about application development. Trust me when I say that annotations in Java will change everything you think you know about application development—they represent a true revolution for the Java platform.
With the introduction of annotations and metadata-driven programming for the Java platform, there are a few questions that are only just being asked by the general developer community. One of these is how annotations will affect application development, and specifically, how will they affect application development tools? Does the introduction of annotations mean that development tools will wane in their usefulness until we somehow arrive at a Nirvana where programming is so easy that all we need is the simplest of text editors to write the most complex applications?
I imagine there are a few people who would like nothing more, and for the small but outspoken group of developers who prefer to code every line by hand, annotations in their simplest form represent a potentially huge leap forward in productivity. However, these gains won't last, and thus the answer to the question posed above is a resounding no (which is fortunate for me, a tool architect in Sun's Enterprise Java Tools organization). Let me explain why.
Complexity Begets Metadata Begets Complexity Begets...
Metadata via annotations is one solution to the increasing complexity of application development. Rather than manage a number of separate artifacts (source code, metadata, etc.) that must be carefully kept in sync, annotations allow metadata to be attached directly to programming constructs. Although in the near term the same effective information may be stored in, say, an XML deployment descriptor and the first-generation of annotations now being defined (thereby making them semantically equivalent), language-level annotations are ultimately more capable, more powerful, and more convenient than metadata stored separately. In short, annotations, being easier to use, lead to increased use of metadata.
This is a a great result—increased use of metadata is a boon for application development because it makes more possible using less. But, it has at least one significant drawback: annotations become complex and unwieldy after a certain point. This is by no means an annotation-specific issue; any set of information becomes unwieldy and hard to manage when it passes a certain size. In the case of annotations, they work quite well when only a few are used and structurally they are relatively simple. However, the more numerous and precise the semantics of the elements being annotated, the more unwieldy annotations become as their number and structural complexity grows.
To that point, many people don't yet realize that annotations are themselves Java types, with most of the capabilities of traditional classes and interfaces. Furthermore, many people I talk to don't yet realize that annotations can effectively be arbitrarily complex, just like other types defined in Java. If you are familiar with JavaDoc-style attribute annotations (such as those used in XDoclet, whose values consist of just arbitrary string values), this is actually quite a difference. Annotations can be specified with most of the complexity found in normal Java classes, thus making them far more powerful and robust, but also potentially quite complex.
Although the nominal situation today is that there are few (if any) annotations used per unit of actual code, this trend will very quickly change as the complexity and adoption of annotations grows exponentially. The more that can be done by annotating something, the more annotations will be used. Our work here in the Java Studio Enterprise group has shown that it takes just a handful of annotations before they begin to overwhelm the element with which they are associated. It certainly can seem a bit ridiculous to see a one-line method with twenty lines of annotations preceding it, but in order to achieve the order-of-magnitude productivity gains that metadata-driven programming brings to the table, it is largely unavoidable and will become increasingly common.
Does this mean we're back in the same boat we were in before annotations were introduced? Frankly, yes and no. Keep in mind that once we begin talking about application development using metadata and annotations, we have suddenly begun talking about a fundamentally different type of application development. It may not yet be apparent, but we have entered a new era in which application complexity will increase by at least an order of magnitude, all made possible by extensive use of metadata. However, by using metadata (and eventually meta-metadata, meta-meta-metadata, etc.) application development itself doesn't need to become more complex, especially where tools can assist in limiting the impact of these quantum shifts.
Ultimately, application development is a zero-sum game—any gains in the ability to do more with less are almost immediately erased by the fact that even more complex problems become solvable. Thus, we've essentially just refactored the application development problem by introducing metadata-driven programming to Java. The tasks developers perform today will become easier in the short term, but these gains will soon be outstripped by the increasing complexity of next-generation applications. This is not something to lament; rather it is just the way of the world. For example, in the same way that 2nd-generation languages (2GLs) supplanted 1st-generation languages when they became too complex to use for generalized software development (a ridiculously short journey), 3GLs eventually supplanted 2GLs, and now Java plus metadata has effectively become a 3.5GL, featuring many of the benefits of a 4GL via annotations while retaining its general 3GL nature. Such is the technological treadmill that we perpetuate.
So, if annotations are ultimately not the long-term solution to application development complexity, what is? The answer should hopefully come as no surprise.
The Value of Development Tools
The oft-misunderstood and underestimated mission of development tools is essentially three-fold:
- Sanity - to make sense of a complex world on behalf of the developer
- Simplification - to reduce complexity and cognitive load for the developer
- Speed - to increase developer productivity
I assert that the Three S's as I'll call them (or S\^3 if you prefer) have been put to the test and proven true over and over again. Despite their near-term appeal as a way to reduce complexity in hand-written code, annotations will not decrease, but will in fact dramatically increase, the value of development tools.
This conclusion may initially seem counterintuitive, but upon reflection should be apparent. Apart from temporary improvements achieved by quantum shifts in methodology (of which the introduction of annotations is one), the long-term solution to application development complexity has always been the presence of good development tools, regardless of the technologies involved. Although development tools going forward using annotations will be significantly different than tools we have all used in the past, the fact that we now have annotations at our disposal really has no bearing on the fundamental value of tools because the level of complexity of the application development problem itself will essentially remain unchanged. We will instead now be able to achieve more for a given amount of effort, but the required level of effort will remain constant.
Ultimately, as a tool provider, I see annotations and metadata-driven programming as greatly beneficial to my business, and using them I look forward to helping developers tackle the next level of complexity in application development. Long live metadata, annotations, and good development tools!