A big mess...
By user12618941 on Nov 14, 2007
Recently I've been thinking about, learning about, and contributing to IPS, the Image Packaging System, a next generation package management solution project which is happening in the OpenSolaris community. IPS happens to also be the packaging system project Indiana has elected to use. Stephen has written extensively about his thoughts on his blog. And Bart has too. IPS subsumes a lot of existing functionality which appears in various parts of the system today. But a lot of people seem to be willing to look at it only as a package manager in the sense that "pkgadd" is a package manager.
My problem with this is that "pkgadd" is only a small part of a larger problem. So, to explain that, I want to distill a series of email posts I made to pkg-discuss last month into a coherent blog entry, since I've referred back to them a couple of times.
My feelings on this topic are pretty nicely summarized by an article by Peter J. Denning which recently appeared in Communications of ACM, entitled "Mastering the Mess." The whole article is instructive, but see in particular: "Signs of Mess."
If one accepts Dr. Denning's "mess" framework, then the next question is whether we are in what he dubs, "a mess." I personally think the answer is "yes." In no particular order (apologies to anything I left out), as a community, we have:
- SVR4 package creation tools
- SVR4 package deployment tools
- Sun's patch creation tools
- Sun's patch application and inventory tools (patchadd, showrev -p)
- PCA (Patch Check Advanced, a nice open source tool I use)
- Solaris Patch Manager (smpatch)
- Live Upgrade
- flash archive creation and deployment tools
- graphical install (old and dwarf caiman, etc)
- virt-install (from xVM)
- zones install
- zones attach/detach logic (which knows how to parse various patch and packaging databases)
- So-called "toxic" patching for zones
- Zones support for live upgrade (zulu)
- BFU/ACR (update part of the system, but violates package metadata)
- IDR (patches the system, but renders system subsequently unpatchable until IDR is removed and a "real patch" is applied)
- Solaris Product Registry (I've never really understood what this was for, but you can try it via prodreg(1))
- Service Tags -- a layer which adds "software RFID tags" in a sense: UUIDs backed by a Sun-maintained ontology; helps to inventory what is on your system.
- Network Repositories (like Blastwave)
- DVD media & CD media construction tools (several of these, I think)
- Various other unbundled products which promise to ease "patching pain"
- Various system minimization tools
- Layered inventory management tools
- Numerous hand-rolled and home-grown solutions built on some or all of the above.
Some parts of the mess represent great (from the perspective of the those caught up in the mess) technologies which people have spent a lot of time and effort building. But a lot of the above represent accreted layers with duplicated functionality. In some cases, the various layers interact in complex and subtle, and perhaps interface-violating ways. To people outside of the mess (i.e. new users we would like to entice) the mess looks bizarre, and terrible. Another sign of a "big mess": In several cases, huge engineering efforts have resulting in only modest improvements. In some cases, huge engineering efforts have been total failures: Sun attempted a rewrite of the SVR4 packaging system in the early part of this decade, the project basically failed.
It's easy to look at the above list and feel a sense of hopelessness-- how will we \*ever\* improve upon this situation? Will people keep creating new and different tools which add more layers?
I'll cite a second source which has helped guide my thinking on this topic: Jeff Bonwick. Jeff spent years relentlessly seeking out and blowing up duplicated and broken kernel functionality, and then took on the storage stack. The result was ZFS, which was recently labelled "a rampant layering violation" by a detractor. Jeff responded this way. In particular, Jeff said:
"We found that by refactoring the problem a bit -- that is, changing
where the boundaries are between layers -- we could make the whole thing
Which to me summarizes my thinking about what the \*opportunity\* is here: to rethink the layers and to merge and unmerge them to come to a more complete, efficient, modern
IPS is heading in this direction: Packaging, patching, upgrade, live upgrade, the mechanisms for Software Delivery, the toolset for delivering packages/patches, and the software-enforced policy decisions seem to be condensing here into a coherent entity-- which means we'll have many fewer layers. And because the system will be fast, lightweight, redistributable and shared, we should also be able to discard artifacts such as BFU and ACR (in other words, OpenSolaris developers will use the same tools our customers use to update systems). The huge amount of code which handles zones patch and packaging should be greatly reduced. Package dependencies will be far more accurate and minimization will be easier, and diskless client support should be far more robust.
What I see with Caiman, IPS and Distro Constructor is the opportunity to do for software delivery and update to OpenSolaris systems what ZFS did for storage management. I do not think we have all the answers just yet, but I think we can get there.