Source-code management for an open JDK

The aging source-code management (SCM) system we’ve been using for the JDK all these years is unsuitable for open development. To which newer system should we migrate as we open-source the code?

A bit of history For over twelve years the JDK source code has been faithfully maintained by a trusty old Sun-developed SCM system called TeamWare, and its predecessors. These systems built upon the seminal SCCS single-file revision-control system—which itself dates almost to the beginning of time—to support large-scale, distributed software development.

The key difference between TeamWare and more-traditional SCM systems such as CVS and Subversion is that there is no central source-code repository. Each working copy of the source tree is, in effect, its own repository. This approach does tend to take up a bit more disk space, but it has several advantages:

  • A branch is just another repository, not some state in a central database. You can create branches at will to explore new ideas and, equally easily, simply delete them if things don’t work out.
  • You don’t need to be connected to a server—or even be on the network—in order to get work done.
  • You don’t need to set up and manage a central SCM host with sufficient disk space, compute power, bandwidth, and backup to support the concurrent SCM operations of your entire development community.

The main trouble with TeamWare is that it depends upon NFS for distributed operation. This works reasonably well so long as you’re not too far from the repository you’re trying to talk to, but doing an update across the U.S. is pretty painful and doing one halfway around the world—to, say, St. Petersburg—is basically impossible. Aside from performance there’s also the problem of how to set up NFS authentication across administrative domains—theoretically feasible, but practically fairly difficult.

TeamWare has other flaws, to be sure—some of them quite fundamental. We could easily open-source it, but from a technical perspective it’s just not a very good solution. Rather than invest a bunch of effort to improve it, then, perhaps it’d be wiser to look further down the evolutionary path.

Darwinian forces The distributed-development facilities pioneered in TeamWare were taken further in the closed-source BitKeeper product, which was used to maintain the Linux kernel tree for a few years, and also in various open-source projects such as Bazaar, Darcs, Git, Mercurial, and Monotone.

These philosophical descendants of TeamWare abandoned the antique SCCS file format and replaced NFS with protocols built on top of HTTP or SSH, or in some cases both. They also introduced many other improvements, perhaps the most important of which was to redefine the notion of a commit from a group of updates to individual files in the repository to genuine a atomic transaction upon the entire repository. This is not only a (much!) more robust approach but it also allows file creates, renames, and deletes to be tracked right along with file-content changes.

Natural selection Now if we want to choose the best of the current crop of distributed SCM systems to use for the open-source JDK we should formulate a list of requirements, conduct an in-depth study of the various candidates, and then pick a winner.

It turns out—handily for us—that the OpenSolaris Tools Community has already done just that. They started out pretty much where we are now, trying to figure out how to move forward from TeamWare in an era of open-source development. They wrote up a list of requirements, conducted a thorough evaluation of the best alternatives, and eventually settled upon… Mercurial.

Definite leanings Our goal is to make the right SCM choice for the open-source JDK; we’re not going to blindly follow the OpenSolaris team. Given our very similar starting points we’d be foolish to ignore their conclusions, however, and after doing some research and experiments of our own we are strongly leaning toward Mercurial. If you think there’s a better choice here then by all means please do let us know!

Migrating a project of this size from one SCM system to another is, of course, a major effort. There are both technical problems (e.g., how do we transfer history information from TeamWare into the new system?) as well as process and planning challenges (e.g., how do we do this in a way that minimizes the disruption to the team?). My colleagues Kelly O’Hair and Martin Englund are busy working out these issues and have more to say in their own blogs.



Just some minor points from personal experience (a) SVN gives atomicity of commit (b) SVN branches are visible at the filesystem level thus minimising the need for special "metadata" tools (c) SVN works well offline, muchmuch better than CVS (d) SVN is efficient over WAN, eg I use it all the time over 3G (e() SVN really just does work and has some nice GUI and IDE integrations available for free.

I remain an active daily user of SCCS, CVS and SVN.



Posted by Damon Hart-Davis on September 26, 2006 at 09:55 PM PDT #

Hi Damon, I agree that SVN is nice, but this is just like on Slashdot when people were discussing which distributed source-code management tool to use for Linux... for a project this size, CVS and SVN is not enough.

When you have tens or hundred of thousands (hopefully!) developers and many subprojects and experimental branches, you can't have a single repository even if it runs on a cluster of servers, you need a management tool that can handle a distributed project. From my understanding, it works something like this: Think of it as a big tree where each branch has its own repositories and administrators, and users can use cvs/svn similar clients. The local administrators can give rights to users for their particular branch, and a few of the administrators have the right to move submitted patches further up a level for review and possible inclusion. Just like Linus and a few others are at the top of the Linux kernel tree, and have trusted subordinates to handle subprojects and experimental branches.

Mark, it would have been nice for bragging rights perspective to use a Java framework for this, but if Mercurial seems to be a superior product, go for it. No need to reinvent the wheel, and besides the Python folks can use a bit of cheering up now that Nuxeo ditched Zope in favour of Java and JBoss. :-)

Posted by Lars Westergren on September 27, 2006 at 12:44 AM PDT #

I did some some searching to see what I could find on Bazaar-NG (bzr) vs. Mercurial (hg). Seems hard to tell in 5 minutes which is better, but these presentation slides (PDF) on Mercurial definitely know how to sound convincing that Mercurial means business.

Posted by Tom Palmer on September 27, 2006 at 01:07 AM PDT #

Damon: Everything you say about Subversion is true, but the fact remains that it’s fundamentally a centralized SCM system and therefore can’t scale well across thousands of developers all around the world. (SVK is a decentralized system built on top of (parts of) Subversion, but relative to Mercurial it has a more complex usage model, nontrivial deployment dependencies, and poor performance.)

Posted by Mark Reinhold on September 27, 2006 at 02:28 AM PDT #

BitKeeper, GIT... Ask Linux devs, they used to use BitKeeper before, and when some license issues arose they developed GIT. I suppose that both projects (Linux kernel and JDK) are similar in size, so their experience should be noteworthy.

Posted by guest on September 27, 2006 at 02:38 AM PDT # GIT was one of the finalists in the OpenSolaris evaluation, but in the end it was found to be inferior to Mercurial. Details are available in the final report on GIT.

Posted by Mark Reinhold on September 27, 2006 at 02:47 AM PDT #

Will there still be the Subversion repository available via for Java 7 or is it Mercurial only?

Posted by Scott Palmer on September 27, 2006 at 03:42 AM PDT #

While the theoretical expectation that "there must surely be some level of activity that can only be supported in a distributed way" seems unassailable, I've never been able to find formal or anecdotal evidence as to exactly where that line may be. Do you have some links of that sort? Certainly, the Apache community, and the KDE community, are some very large communities working very happily in centralized form.

What seems more pertinent to the Distributed/Centralized debate is: what sort of work-flow do you need? The Linux kernel community is an interesting case here, actually: a tremendous amount of work and value arose form the original distributed arrangements, but commercial success didn't come until companies like Red Hat stepped forward, picking and choosing, testing and packaging--centralizing. The DCM culture serves still today to encourage involvement (particularly at the read-only level), and to lower the barriers between observation and contribution, but it's not convincingly a performance decision, rather a cultural one.

(I should disclaim: I work for CollabNet; we sell centralized CM. Whether my opinions come from my job or the other way around you're free to decide for yourself ;-)

Posted by Jack Repenning on September 27, 2006 at 04:46 AM PDT #

@Jack Repenning: I agree with you that there doesn't seem to be a clear line where one solution is obviously better than another, maybe such a line doesn't even exist. But the fact is that if the requirements include "Unbiased and disconnected distribution" none of the centralized solutions work. The question then becomes "WHY require unbiased and disconnected distribution?". It might just be that most of Sun's developers are used to working that way. And personally I'm leaning more and more to such a system myself for projects where a lot of experimentation is done (and where you don't want to lose the history of how that particular experiment came to be). Centralized systems just don't really support that way of working.

Posted by Quintesse on September 27, 2006 at 06:08 AM PDT #

@ Quintesse: yeah, I agree: your "used to working that way" idea is parallel to what I meant by "what sort of work-flow do you need."

My sense for this community, call me crazy, is that the liberty of something like Mercurial would be very useful in a lot of ways (and therefore, should be encouraged), but that the centralized control will also have its value. What to do, what to do?

Answer: hybrid. Plenty of centrally managed teams already allow non-central people freedom of choice (for instance, Linux). I haven't played with Mercurial enough (yet?;-) to know how best to do it, but I happen to know that Bazaar works just slick this way. What I do know about Mercurial makes me suspect it could be done pretty easily there, too.

In this model the community of Mercurial users makes its own decisions about whose repositories are "official" or not, whatever that may mean to this particular community. Typically, you get a somewhat lose hierarchy. Folks somewhere near the top have in their repository something pretty darned close to "the" official version; other folks have great freedom to stay as "official" or be as "creative" as their needs dictate.

To go hybrid, the "official" folk just check their Mercurial view in as the new version in the central repo, and voila! people like release-build engineers no longer have to keep track of which push/pulls they need, they just check them out from the central repo. Similarly, the central repo can be managed with all that good IT foo, backups and RAID and capacity planning and security and all that, while the workers (even the top-o-the-heap "official" workers) are still free, day to day, to Mercurialize to their hearts' contents.

Posted by Jack Repenning on September 27, 2006 at 09:26 AM PDT #

Scott: We plan to maintain the read-only Subversion repositories for JDK 6 and 7 for a while yet. These sources are licensed under the (non-F/OSS) JRL, so they include all of the encumbered code that we won’t be able to ship when we open-source the full JDK. We’ll probably maintain these Subversion repositories at least until all of the encumbered code has been replaced by F/OSS alternatives.

Posted by Mark Reinhold on September 27, 2006 at 01:25 PM PDT #

Jack Repenning & Quintesse: You’ve both hit on important points here. The performance advantages of a distributed SCM system are obvious at an abstract level, but the cultural element is equally important.

With the open-source JDK we definitely want to encourage all sorts of wild experimentation—most of which will never make it back into the mainline code—and that style of development is best served by a distributed SCM.

At the same time we’re absolutely committed to the continued production of high-quality, on-time platform releases, and that does require some degree of “hybridized” centralization.

In fact that’s pretty much how we’ve used TeamWare all these years: There’s a sacrosanct, tightly-controlled “master” repository from which the release-engineering team does its official product builds, there are loosely-controlled component-team integration repositories, and individual engineers typically maintain as many private repositories as they need to manage their current work. As changes move up the hierarchy they’re subject to successively stringent review and testing criteria which (usually) ensure that the code in the master is always of production quality.

Posted by Mark Reinhold on September 27, 2006 at 02:31 PM PDT #

First, on centralized vs. distributed source control: I've had years of experience with both styles, and I strongly prefer distributed. Fundamentally, there are very few things you can do with a centralized system that you can't also do with a good distributed system, but the reverse is definitely not true.

For example, all distributed systems I'm aware of will let one developer pass code to another developer privately, e.g. for a code review, without the changes becoming visible to anyone else. All distributed systems allow private forks of a code base with seamless integration of upstream changes. And of course the ability to work, and perform version control operations, when you don't have connectivity to the central repository is key.

Centralized systems usually claim strong enforcement of workflow as an advantage, but exactly the same effects can be achieved in distributed systems too: on a given node of the distributed system, you can impose restrictions on who has permission to perform operations, add post-merge triggers, etc.

Now about Git: After reading the OpenSolaris evaluation, I believe you guys (the JDK team) should reevaluate it. This time you should use it via the "Cogito" interface, which is the most popular Git frontend. It addresses some of the concerns raised in the OpenSolaris evaluation, so you might end up thinking more highly of Git than the OpenSolaris team did. After which you may nonetheless decide to use something else, of course, but at least you'll be giving the current form of Git a fair shot.

I've been evaluating VCSes recently and Git is currently my front-runner. Its major downside is the lack of true rename support -- but Mercurial isn't exactly stellar in that department either.

It will be interesting to see how this evaluation turns out, in any event; many of the major free VCS tools have changed significantly since the OpenSolaris evaluation. Git has new frontends, Monotone has gotten significantly faster and has more robust rename support, etc.

Posted by Steven Grimm on October 01, 2006 at 04:51 AM PDT #

Steven: Thanks for the pointer to Cogito. We’ll take a closer look.

Posted by Mark Reinhold on October 01, 2006 at 06:47 AM PDT #

Also the FreeBSD is considering an exodus from CVS, it may interest you because they also defined some "wanted features" in a version control system, that can be general enough to help compare various VCSes: VersionControl

Monotone "replied" creating a parallel tree in its own wiki and analyzing the various features (work in progress): EvaluationFeatures

Posted by Lapo Luchini on October 07, 2006 at 11:24 AM PDT #

Post a Comment:
Comments are closed for this entry.

This blog has moved to <script>var p = window.location.pathname.split('/'); var n = p[p.length - 1].replace(/_/g,'-'); if (n != "301") window.location = "" + n;</script>


« June 2016

No bookmarks in folder


No bookmarks in folder

RSS Atom