Wednesday Nov 04, 2009

OSCON 2009: FreeBSD

The last "people" talk that I went to at OSCON was Marshall Kirk McKusick's talk "Building and Running an Open-Source Community". I was interested in this talk for a couple reasons. First, I don't know much about how the BSD communities work, and I'm always interested in how large open-source communities do things. Second, I was at Berkeley during the time of the Computer Systems Research Group (CSRG). And while I got to know some of the CSRG staff, I was not directly involved with the development of Berkeley Unix. So I wanted to find out more about what was going on while I was there.

Kirk mentioned that the CSRG started up in the 1970s, after Bill Joy was already at Berkeley. At first, they didn't use a source code control system. Then around 1980 they started using SCCS. There are various reasons for using a source code control system, such as making it easy to review changes if a regression is discovered. For the CSRG, introducing SCCS enabled better productivity for the CSRG staff. Although they still reviewed all the changes that were checked in prior to a release, they could hand off some of the mechanics, such as merging patches and testing, to trusted committers.

This basic structure, with a core team, a group of committers, and a group of developers, is still used today for FreeBSD development. Kirk mentioned a couple details that I thought were interesting. In particular, he said that most developers don't want to be committers. This is usually because they don't want to be involved that much; they just have a change or two that they want to see made. Kirk also mentioned that committers are held to higher standards for things like email etiquette. And all changes must be reviewed by at least one other committer.

The FreeBSD core team is 9 people who are nominated from and elected by the committers every 2 years. They maintain the FreeBSD roadmap, they resolve conflicts between committers, and they admit and remove committers.

Kirk pointed out that people can contribute to FreeBSD in ways other than writing code. They can write documentation, they can do testing, they can do release engineering, and so forth. These people can be committers, too, and there's no relative weighting between code committers and other committers. This also means that these other folks can be elected to the core team. In fact, the latest election brought an advocacy/marketing committer onto the core team. At the time of the talk, there were 390 committers and around 6,000 developers.

FreeBSD does a stable .0 release every couple years. The stable branch has a 5-year lifetime and a binary compatibility guarantee. Minor (dot) releases happen on the stable branch every 6 months or so. Development happens on the trunk, and important bug fixes are merged into the stable branch. They use Subversion and provide a CVS mirror.

The pre-release freeze times vary: for a new stable branch it's about a month, and for a dot release it's about a week. I'm not sure how to compare those times with the times for OpenSolaris. For example, OpenSolaris freezes are gradual: first there's a period where only bug fixes are allowed in (no new features), then there's a period where only fixes for stopper bugs are allowed in. I wish I'd thought to ask Kirk for more details on how FreeBSD manages freezes.

One of the ways the BSD license is different from the GPL or the CDDL is that it lets people make proprietary changes to the code. Kirk said that this does happen, but those changes are usually specific to a particular product. Because they aren't generally interesting, the FreeBSD project probably wouldn't take the changes even if they were offered.

Saturday Oct 10, 2009

OSCON 2009: Building Belonging

Ubuntu is widely known for its community-building efforts, so I was very interested to hear what Jono Bacon, the Ubuntu community manager, had to say at OSCON.

Jono suggested that people want to be in a community (any community) because it gives them a sense of belonging. This makes sense to me--we are, after all, social creatures. And Ubuntu seems to do a good job at this. Jono showed a photo from the recent Ubuntu Developer Summit in Prague, in which all the volunteers were highlighted. I didn't have time to count, but it appeared to me that over half the people there were volunteers, not Canonical employees.

So how does a community foster this sense of belonging? Jono had several suggestions. One was to treat all contributions as gifts. That doesn't mean accepting every contribution. But if the contribution can't be accepted for some reason, be constructive and gentle in your feedback.

Jono suggested some ways that structure can help foster belonging. For example, Ubuntu structures everything in teams, even user groups. This makes everything simple and uniform: to get involved, find a team that you want to join. And because the teams are smaller than the community as a whole, it's easier to get started and make connections.

Structure can also be used to encourage socializing. By this Jono didn't mean the structured team-building exercises that one sees in the corporate world. Rather, the idea is to budget time just for chatting, sharing stories, etc. Besides encouraging social bonding, this promotes information exchange that might not happen otherwise.

Jono also noted the importance of the virtual environment. First, he drew an analogy with physical neighborhoods. People are more inclined to hang around someplace that is kept-up, safe, and inviting. Second, the virtual environment can help build belonging by making contributions (and contributors) visible.

Community members can reinforce a sense of belonging by framing questions or issues in a way that leads to progress. Jono's example was to change the thought "this sucks" to "I won't live life this way".

Saturday Sep 05, 2009

OSCON 2009: No Unicorns

Between the site migration, a code review for Darren Moffat [1], and catching a cold, I'm still having to write my OSCON 2009 trip report in dribs and drabs.

The next "people" talk on my list was a talk by Rolf Skyberg of eBay called "There Are No Unicorns: And Other Lessons Learned While Running an Innovation Team". Having worked in a few research settings, but not being a researcher myself, I was hoping the talk would give me a better understanding of how innovation happens. And Rolf did have some things to say here. In particular, he pointed out that innovation is by nature disruptive, which can lead to resistance. So be sure to focus on the cultural changes that will be needed for your innovation to be successful, don't just focus on the technology or specific product.

The bulk of the talk seemed more about navigating corporate politics. Rolf's points included the importance of knowing how your work and your team's work will be evaluated, and how your project fits into the larger organizational picture. Understanding how your project fits in is particularly important when times are tough. If your project isn't in a core area for your organization, and it doesn't have someone high-up to defend it, it's more likely to get cancelled when funding runs low.

Rolf also talked about how traumatic events in a company's history can affect its behavior for a long time. In 1998 eBay was off-line for 3 days, which led to a very strong emphasis in the company on reliability and scalability. But, Rolf argued, the outage also led to an inflexible organization that has a hard time incorporating new innovations.

I've seen this never-again effect in other contexts. When I was working for Xerox, I wanted to look at the source code for Pilot, which was the operating system we were using. I was told I couldn't do that, so I asked why. I was told that sometime in the past, some application code had been written that depended on an interface that was private to Pilot. I doubt that this was accidental usage, because all this stuff was written in Mesa, which made it easy to keep public interfaces separate from private ones. Anyway, in the normal course of development, the Pilot group made some incompatible changes to that interface, which broke the application--right before the application was about to ship. The Pilot group wanted the application programmers to fix their code, even if it meant a schedule slip. But they were overruled by management and forced to revert the interface change. They vowed that they would not let this happen again, and from then on they kept their source squirreled away on private servers, accessible only to people in the group.

I've also seen this effect at Sun. I started at Sun in 1992, not too long after Solaris 2.0 had shipped. Solaris 2.0 introduced a lot of changes that were incompatible with SunOS 4.x. This caused considerable disruption for customers, and I gather that many were not shy in voicing their displeasure. I remember being told that all the pain around Solaris 2.0 was a big factor in the decision not to do any more major (.0) releases of Solaris.

Of course, that policy has born considerable fruit over the years. The ISVs that I've spoken with have praised Solaris because new Solaris releases rarely broke their code. Other platforms that their code ran on tended to cause breakage pretty regularly.

On the other hand, this compatibility doesn't come for free. Project teams must design new interfaces carefully, so that they can evolve in a compatible fashion. The interface review and cross-checking that is done by the ARCs is largely manual. Standard (cross-platform) interfaces are not always stable, so project teams must sometimes do extra engineering to support the new interface without breaking code that depended on the old interface. There is a mechanism for removing old, obsolete code, but again, this requires supporting the old and new interfaces for some time.

Rolf compared companies to creatures when he talked about the effect of traumatic events. That seems to imply some sort of organizational memory of the traumas. But I wonder if that's really how it works. Is it really an organizational memory, or is it more the memory of a few key individuals? If it's the memory of key individuals, then I think it's not until these individuals move on (or relax around the trauma) that the never-again imperative starts to lose its hold.

[1] Darren is removing 92,000 lines of closed-source code--the obsolete smartcard support--from ON.

Friday Aug 07, 2009

OSCON 2009: Lying and Geniuses

Alas, it's taking me much longer to write up my OSCON trip report than I had planned. Part of that is that I've been busy with other things, like resurrecting a testbed for the SCM framework, so that I can start on the workaround for the "unwanted mounts" issue. And part of it is that it's just taking me longer to do the writing, despite already having an outline. So since I have finished my comments on a couple talks, I'll go ahead and post those now.

My favorite session at OSCON was "How to Lie Like a Geek" by Michael Schwern, which was a lively talk about things we can do to help or hinder communication. Most of us are familiar with technical lying, particularly bad benchmarking and abuse of statistics. Michael recommended as a place to get good benchmarks.

Michael also suggested several ways to "lie" that are more generic. Many of these had to do with getting so caught up in details that the big picture is lost. For example, there's lying by information overload: providing too much detail. There's also lying by pedantry or by being excessively literal: focusing on the wrong details.

A fascinating way to mislead is to state the obvious. For example, consider this exchange (which I've adapted from the example Michael used):

Alice: We should change this code.
Bob: But that could introduce bugs.
Alice: Of course. Why are you arguing against making the change?
Bob (annoyed): I'm not, I'm just saying we might introduce bugs. Don't put words in my mouth.

I've seen exchanges like this in the past. At best, there's this unnecessary hiccup in the conversation. At worst, the conversation goes off into the weeds, with endless rounds of claims and counter-claims about who said what and what was really meant.

When I mentioned this example to my wife, she pointed out that people often make implicit requests in their statements. So if Bob says something obvious, he's clearly not offering new information, so it's not surprising for Alice to interpret his comment as a request. If Bob had a different request in mind, like "let's defer that until after the code freeze", it'd be more helpful to say that in the first place.

Another talk that I liked a lot was "Programmer Insecurity and the Genius Myth", by Ben Collins-Sussman and Brian Fitzpatrick. I had to chuckle at this quote, which Ben and Brian said came from the 2008 Google I/O developer conference:

Can you guys make it possible to create open source projects that start out hidden to the world, then get "revealed" when they're ready?

The site supports hidden projects. And for awhile we created new projects as hidden, so that the project team could get the project page set up before making the project visible. But hiding the project doesn't provide that much benefit. After all, the project space is a work area; it's okay for things to be rough. And we've had a few problems with projects lingering in hidden mode for a long time. So we've moved towards making projects visible from the start.

Of course, it's natural for people to want to get things right before sharing them with the outside world. But there are potential advantages to sharing early. One is that if you're going down the wrong path, early sharing improves the odds that you'll discover the problem quickly. And if you have to explore a couple dead-ends before you come up with something that works, the information from those explorations is available for others to learn from.

And then there's the "bus factor", which refers to the number of people who would have to all be run over by a bus to stop the project. Working in the open--with archived discussions, publicly visible code, any plans and documentation that you've written down--makes it easier to recover if a team member is unavailable for whatever reason.

Ben and Brian did point out that it is possible to share too early. The project needs to be far enough along that it won't get stalled when outsiders show up and start asking questions and suggesting changes.

But since working in the open like this can be challenging, what are things we can do to encourage it? Ben and Brian made several suggestions, both social and technical.

On the social side, the first thing to remember is not to let your ego get tied up in your design or code. You've probably heard that before, but I think it bears repeating. When someone points out a problem in my design or code, I try to think of it as an opportunity to improve something that I care about, and as an opportunity to learn. But that's often not my initial, automatic response; it takes some effort.

Even if you don't expect your project to benefit from outsiders' comments, engaging in a conversation can have benefits. As Ben and Brian put it, if you want to influence people, you need to be open to influence.

On the technical side, consider what behavior your tools are encouraging. For example, the current portal doesn't keep page histories, which discourages its use for collaborative writing and editing. That's one of the things that will be fixed by the move to XWiki in September.

Ben and Brian also recommended responding to questions and arguments on the project web site, rather than by email. With that approach, they said, the discussion is less likely to degenerate into pointless argumentation. I don't recall them saying why they think this works. I suppose one reason is that it helps keep arguments from being repeated.

Wednesday Jul 29, 2009

OSCON 2009

The O'Reilly Open Source Convention (OSCON) was at the San Jose Convention Center this year. It's been in Portland in the past, and while I like Portland, the additional expenses for air fare and hotel probably would have meant staying home this year. So I'm glad it was in San Jose. We'll see where O'Reilly decides to hold it next year. At the feedback session after the closing keynote, there seemed to be quite a few people who would like the conference to return to Portland.

Lunches on Wednesday and Thursday were catered. At my last OSCON (2006) the conference provided basic box lunches; nothing special. The lunches this year were impressive, with a variety of well-prepared dishes.

I attended several talks, which I've organized into 2 categories: people talks and technology talks. The people talks covered things like communication skills, community-building, etc. The technology talks that I went to covered topics that I wanted to learn more about, either for use at work or for personal projects. I also went to a couple BOFS and the closing keynote session. In order to make the writeups more digestible, I'll cover the BOFS and keynote here, with separate postings over the next few days for the "people" and "technology" sessions.


I went to Brian Nitz's SourceJuicer BOF session Wednesday night. Alas, only a few people attended, mostly from Sun.

Brian gave a short presentation and demo. This helped fill in some of the holes in my knowledge of SourceJuicer, like how Pkgfactory and Robo-Porter fit into the picture. (Pkgfactory is an automated mechanism that feeds into SourceJuicer. Robo-Porter is a component of Pkgfactory.)

Individuals can contribute spec files to SourceJuicer, though apparently the tags aren't quite the same as they are for RPM spec files. Builds are done in a freshly-created zone which has a minimal build environment. Thus the spec file must list build dependencies (as well as runtime dependencies).

Thursday night I went to the Silicon Valley OpenSolaris Users Group (SVOSUG) meeting, which was relocated to OSCON for this month. John Weeks demoed a couple toys that he has built and talked about what went into making them. John Plocher also demoed his programmable xylophone. I always find this sort of presentation fascinating, even though I've never built anything similar myself.

Closing Keynote

A few things struck me from closing keynote address, which was given by Jim Zemlin of the Linux Foundation.

The first thing that grabbed my attention was how Microsoft's attitude towards open source and the GPL have changed over the years. Jim had a Microsoft quote from a few years ago about how open source and the GPL are just horrible (a threat to business, if I remember correctly). But this year, Microsoft is releasing some code under the GPL because that's what customers want.

The second thing was Jim's discussion about the introduction of netbooks and the changes that he predicts for the PC business ecosystem. In particular, Jim predicts that wireless service providers will be making discounted PCs available, just as they make discounted cell phones available today. If I understand his argument correctly, end users will focus more on the applications and services that they get from the wireless provider, and less on the underlying operating system. And the platform - hardware plus operating system - providers will be under pressure to make their platforms attractive to the wireless service providers (cheap, good functionality). The resulting competitive pressure should improve the opportunities for operating systems other than Windows (Jim was focusing on Linux, of course).

I suppose this could happen; I don't know the PC business well enough to have an opinion. It'll certainly be interesting to watch. And it'll be interesting to see whether these subsidized netbooks are treated more like computers or like phones. Linux is already bundled on some netbooks, and from what I've read, users can run into problems if they upgrade from someone other than the netbook provider. And Jim mentioned that the Linux Foundation has been getting a lot more phone calls in the last 6 months. Some of them are from people offering kudos, some are like the one he played for us, which was from a guy who needed technical support. Compare that with my cell phone: I know who made the hardware, but I have no idea what OS it's running.

The third item was a discussion about software patents and the danger they present to the open source movement. This was mostly old news to me, but Jim mentioned something I hadn't heard about before: One of the problems with the current patent system, at least in the USA, is getting information recorded so that it is counted as prior art. Filing defensive patents is a pain, and the US patent office doesn't keep up on the zillions of articles that are published at academic conferences and in trade magazines. You can challenge a patent after it's been granted, but that's a pain, too. But now there's an organization dedicated to collecting technical disclosures and publishing it in a prior art database that the patent office will check. Very cool.


Random information that I hope will be interesting to Oracle's technical community. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.


« July 2016