The FREENIX track - has it succeeded too well?
By val on Jul 22, 2004
OLS is my favorite conference, bar none, despite the fact that I started attending shortly after I stopped doing Linux development. OLS is a great conference at all levels - technical content, speakers, location, atmosphere, beer supply... Every year, each time slot has at least one talk or BOF that I'm interested in, and this year was a new high for quality in both talks and presentation. I think most of the excitement comes from the relevance of the work people are doing. Often, the code being presented is running on the laptops of half the people in the audience. OLS had a tendency in the past to present somewhat amateur rehashes of design ideas already considered old-hat in the proprietary world, but that tendency has declined markedly. Most of the papers now focus on the real-world implementation of useful new functionality, with a thorough review of the roadblocks and the evolution of the design along the way. Rusty Russell's papers are a fine example of this style. Going to one of his talks gives you an realistic view of the way real world design and development proceeds - with many bad ideas, bugs, obstinate personalities, and wrong turns along the way.
The 700 pages (yes, 700) of proceedings (currently available as preprints) for this year's OLS led me to think about the purpose of the FREENIX track at USENIX Technical. FREENIX was intended to encourage publication of work going on in open source that wasn't, at the time, being published or recorded anywhere. FREENIX was to be a sort of gateway drug to get open source developers hooked on publishing, so the rest of the community could find out what was going on without trawling through hundreds of thousands of posts to linux-kernel. Hefting the two large tomes handed to me at the OLS reservation desk and comparing them to the rather more sparse FREENIX '04 proceedings, I had to wonder: Has FREENIX served its purpose? It certainly seems to me that a lot of publication is going on the open source world, as my aching shoulder can attest. Adding fuel to the fire, half of the USENIX general track papers use open source software (\*BSD, Linux, Apache) as the base for experimentation.
At the same time that FREENIX seems less necessary, the USENIX general track has become more academically oriented, which is certainly not a bad thing. However, these two trends together are squeezing out what I consider the classic USENIX paper: the industry paper describing a useful new idea which is quickly adopted by the rest of industry, e.g., NFS or VFS or the slab allocator. While the trend towards more academic quality papers has resulted in more rigorous and thoroughly researched papers, I think we have paid a price. For a variety of reasons, papers about shipping products have a harder time fulfilling academic style requirements. An analogy may be the best illustration: This week's issue of "Science" (9 July 2004, Vol. 305, No. 5681) included a special section on immunology, in which one article ("Immunotherapy: Bewitched, Bothered, and Bewildered No More") decried several trends causing researchers to avoid human research; that is, research on actual, live human beings. One of the reasons they cited is as follows, from page 198:
Many elite, basic science journals often consider valuable human studies to be inadequate because they lack the mechanistic depth we have come to expect from studies using laboratory organisms. Often precluded from publishing the best of human research in such journals, clinical scientists struggle for professional advancement, after having been excluded from the scientific mainstream. As editors ourselves, we now feel that basic research journals should get equally excited about hard-earned advances in understanding valid scientific problems as they exist in humans, trading some expectation of mechanistic insight for inherent relevance and respect for the demands of working within a proscribed lengthy human protocol.
To put it bluntly, research papers about humans can't be as thorough as those about mice because you're not allowed to "sacrifice" the human at the end of the study and examine their lymph nodes under a microscope, or implant wires into their brains without consent. Similarly, papers on open source or shipping software of any sort are under more constraints than, say, a paper about a research operating system. Standards compliance, stability requirements, backward compatibility, inability to get data from most customer sites, sheer time pressure - all contribute to a paper that doesn't fit the academic model. On the other hand, product-oriented papers can offer many things an academic research paper cannot - real-world feasibility, full implementation, long term experience, much wider user base, and more subtle revelations that only come with shipping a product to thousands or millions of users. I think there is a place for this kind of paper, and that place should be at USENIX Technical.
What, then, is to be done? I have a suggestion: Replace the general track and the FREENIX track with something akin to the "Product Track" and the "Academic Track" (terrible names, but you get the idea). The "Product Track" would be for software which is intended to be used by the masses. Examples include proprietary software products and a new scheduler for Linux someone is trying desperately to get merged. No one can force users to use a particular piece of software, so intent would be the key in classifying a paper as a "Product Track" paper. The "Academic Track" would be for blue sky research, or experiments on free software not intended to ever be merged back into the mainstream (most research being conducted by graduate students would probably fall into this bucket). Papers in this track would tend more toward the academic style. Ideally, all submissions would go into a common pool. Decisions on how to classify the papers and divide the work of reviewing them would be up to the conference organizers and the program chair.
I believe these tracks would encourage industry participation, preserve USENIX's hard-won reputation for academic excellence, and continue to encourage publication of work on open source software. Of course, reorganizing the tracks is not, by itself, a complete solution, but I believe it would be a step in the right direction.