Observations on packaging

Over the past few months, a bunch of us have been exploring various options for packaging. (Actually, I suppose I've been pursuing this on a part-time basis for a year now, but it's only recently that it's made it to the top of the stack.) I've looked at a bunch of packaging systems, ported a few to OpenSolaris, run a bunch on their native platforms, and read a slew of manual pages, FAQs, blogs, and the like. In parallel with those efforts, Bart, Sanjay, and others have been analyzing the complexity around patching in the Solaris 10 patch stream, and improving the toolset to address the forms of risk that the larger features in the update releases presented.

In the course of those investigations, we've come up with a number of different approaches to understanding requirements and possibilities; I'll probably write those out more fully in a proper design document, but I thought it would be helpful to outline some of those constraints here. For instance, one way to look at how we might improve packaging is to separate the list of "design inputs" for a packaging system into "old" and "new".

When I make a list of this kind, I know it's bound to offend. (I already know it's more scattershot than the argument we'll present in a design document.) Feel free to send me the important inputs I've omitted, as I have a few follow-up posts on requirements—lower level, more specific intentions—and architectural thoughts where I can cover any issues not mentioned here.

"Old" inputs

The "old" inputs are those that are derived from current parts of the feature set around software packaging and installation. Many of these inputs will really be satisfied by the new installer work ("Caiman") Dave's leading, but the capabilities of the packaging system will change some from difficult and fragile to straightforward and robust. With effort, some might even achieve some elegance.

  1. Hands off. For a long time, via the JumpStart facility, it's been easy to provision systems via scripted installed. This technology is still sound, particularly when we consider that the Flash archives allow a mix of image-based provisioning with the customizations JumpStart offered.
  2. Virtualized systems. As long or longer, we've supported the notion of diskless systems, where the installed image is shared out, in portions, between multiple running environments. Zones was a direct evolutionary successor of the diskless environments, and shows that this approach can lead to some very economical deployment models. Lower-level virtualization methods can also benefit from the administrative precision that comes out of sharing known portions of the installation image.
  3. Availability and liveness. With Live Upgrade, it's been possible for a while now to update a second image on a running system while the first image is active—a command to switch images and a reboot is all that's required to be running the new software. This approach requires too much feature-specific knowledge at present, but provides a very safe approach to installing an upgraded kernel or larger software stack, as reverting to the previous version is just a matter of switching back to the previous image. So, a package installation option that doesn't ruin a working system is a must.
  4. Change control. In principle, with the current set of package and patch operations, it is possible to create a very specific configuration—that may never have been tested at any other site, mind you—from the issued stream of updates. From a service perspective, the variety of possible configuration combinations is too broad, but the ability to precisely know your configuration and make it a fixed point remains important.
  5. Migration and compatibility. There's a very large library of software packaged in the System V format that will run on most OpenSolaris-based systems. Providing well-defined compatibility and migration paths is an obvious constraint, given the goals of the development process.
  6. Networked behaviour. Believe it or not, the current installer and packaging support a large set of network-based operations—you can install specific packages and systems over the network, if you know where everything is. That is, the current components need to be assembled into some locally relevant architecture by each site to be useful—any replacement needs to make this assembly unnecessary, potentially via a number of mechanisms, but definitely via a list of known (trusted?) public package repositories.

"New" inputs

But, as you might expect, the efforts around Approachability, Modernization, and Indiana have brought to light some new qualities a packaging system must possess.

  1. Safety. One of the real complications from virtualized systems, at least in current packaging, is that a developer has to understand each of the image types his or her package might reach, and make sure that the set of pre- and post-install scripts, class actions scripts, and the like are correct for each of these cases. When that doesn't happen, the package is at a minimum incorrectly installed; in certain real cases, this class of failures compromises other data on the system. Restrictions in this space, particularly during package operations on an inert image, seem like a promising trade-off to achieve greater safety.
  2. Developer burden. Current packaging requires the developer provide a description of the package, its dependencies, and contents across a minimum of three files, in addition to knowing a collection of rules and guidelines around files, directories, and package boundaries. Most of these conventions should be encoded and enforced by the package creation and installation tools themselves, and it should be possible to construct a package with only a single file—or from an existing package or archive.
  3. smf(5) aware. OpenSolaris-based systems have an understanding of services and service interdependencies, derived from the Service Management Facility's service descriptions. smf(5) service management during packaging operations is awkward under current packaging and very much needs to be improved but, more importantly, the service graph provides a rich set of relationships among larger components that should lead to a better understanding of the rules around consistent system assembly.
  4. Minimization. For a while, the current system has had some very coarse package "blocks", from which one could construct a system—the various metaclusters. These, with the exception of the minimally required metacluster and the entire metacluster, split the system along boundaries of interest only to workstation installs. Any suitable packaging system must provide query and image management tools to make image construction more flexible and less error-prone (and eliminate the need for things like a "developer" metacluster, for that matter).

    It's also pretty clear that the package boundaries aren't optimized in any fashion, as evidenced by the differing rates of change of the binaries they currently enclose—in the form of the issued patches against Solaris 10.

  5. Multiple streams of change, of a single kind. Although we noted the continued need to control change above, it's also important to be able to subscribe to a stream of change consistent with one's expectations around system stability. The package system needs to allow one to bind to one or more streams of change, and limit how the interfaces the aggregate binaries from those streams evolve. That is, it should be possible to subscribe to a stream of only the most important updates, like security and data corruption fixes, or to the development stream, so that one's system changes in a way consistent with one's expectations.

    Conversely, the tradeoff between complexity and space optimization in current patches—which introduce a separate and difficult-to-calculate dependency graph and distinct namespace entries for each platform, among other issues—has slid much too far, given the increase in system richness and the increases in disk space and bandwidth. There seems to be little long-term benefit in preserving the current patch mechanism, particularly since Sun never offered it in a form useful outside of Sun's own software.

  6. ZFS aligned. ZFS offers the administrator access to so many sophisticated options around deployment and data management that it would be foolish for a packaging system to not explore how to take advantage of them—zfs snapshot and zfs clone are the most obvious capabilities that allow the packaging system to limit the risk of its modifications to an image, without excessive consumption of storage space or system time.
  7. Prevent bad stuff early. Another classic OpenSolaris-style requirement is that the set of available packages be known to be self-consistent and correct at all times, to spare the propagation of incomplete configurations. In a packaging system, this input expands on our intent to reduce the developer burden to assist the developer in writing as correct a package as possible, and to enable the repository operator to block incomplete packages from being made available to clients. There's a rich set of data here, as we noted for smf(5) service descriptions above.
  8. Friendly user deployment. Direct from Indiana, but sensible and apparent to all is that packaging systems have advanced to a point where the usability of the client is expected, and not an innovation. I haven't got the complete timeline of packaging developments—the literature survey continues—but it's clear that Debian's apt system marks the current expectations about the client-side capability and ease-of-use.

In the course of the Indiana discussions, Bryan raised one point, which I'll paraphrase as "it's not an idea until there's code". That's a sound position, which I also happen to hold—Danek and I (and Bart and Daniel, I hope) and have been quietly prototyping for a little while now, to see if we had a handle on this collection of inputs. I'd like to give a bit more background, in the form of requirements and assertions, in an upcoming post or two. Then we're hoping to start a project on opensolaris.org to take the idea all the way from notional fancy to functioning code.

[ T: ]


Nice problem summary Stephen - I look forward to seeing the ideas you lot have been coming up with, but so far, this looks good to me. I'm interested in seeing how ZFS can help things (I'm already enjoying ZFS root partitions for many of these reasons).

In section 1. Safety, or perhaps section 8. Friendly user deployment, it'd be lovely to be able to get a summary of changes made by newly added or updated packages to a copy (like a snapshot diff ?) before deciding to commit to using that inert copy. Users could then determine the implications any new or updated package would have on the system. At the moment, with SVR4 packages, a list of updated files is provided, but the pre-install/post-install scripts are loose cannons, so to speak.

Posted by Tim Foster on July 19, 2007 at 07:24 AM PDT #

A critical constraint seems to be omitted, perhaps intentionally at this point. A packaging system should place bits on media. Any configuration of those bits which requires human intervention should be done elsewhere. Interactive packages which ask questions should be flogged, flayed, and publically humiliated. Such a constraint would have greatly reduced the complexity of installing Solaris packages and expedited the move to net-based installation many years ago.

Posted by Richard Elling on July 19, 2007 at 08:19 AM PDT #

@Richard Even apt based systems have interactive package installation. Sometimes interactive installation is absolutely, positively, necessary \*if\* no pre-defined "answers" are available. As long as there is a way for an administrator pre-define the answers to the questions an interactive install asks, I see no problem with a package doing one.

Posted by Shawn on July 19, 2007 at 08:42 AM PDT #

i'm particularly interested in using zfs in a live update process, given there is enough space, zfs should help avoiding the need of an extra slice to do lu, also we could use zfs snapshots as alternate boot environments. we would need the ability to tag the snapshots as boot environments some way though. the only problem i see is that conservative admins will remain using ufs a while more, and why shouldnt they, ufs works nicely and is very well tested. in any case, this will really be a short term problem

Posted by nacho on July 19, 2007 at 09:58 AM PDT #

You write that you looked at different packaging systems. I'm curious, which ones did you look at? I noted that none of the requirements you listed deal with fundamental problems that have been plaguing us for the past 20 years on Solaris: real package clusters (for bundling of software and its dependencies) and lack of namespaces, so that unified software packaging effort will be technically feasible. These are big issues, which seem to be silently dropped by any packaging discussion. Why?

Posted by ux-admin on July 19, 2007 at 04:38 PM PDT #

Sounds good Stephen! I agree we need to move from doing two layers sub-optimally (packages and patches) to doing one layer really well. Please also consider the requirements of application layer products on packaging and patching - such as the requirement from middleware layer products such as JES for non-root install and multiple instance installs. See Chris Kampmeier and Chris Kasso for further information. Too much of the current implementation is highly OS-level centric - for example there's no way currently to define conditional dependencies in the base patch pkginfo metadata of the type: if OS=5.9 then SUNW_REQUIRES Patch-X elif OS=5.10 then SUNW_REQUIRES Patch-Y Such capabilities are required by application layer products such as JES, Oracle, etc. where the same application version can be applied to multiple OS versions.

Posted by Gerry Haskins on July 22, 2007 at 11:06 PM PDT #

Good to see one of the cornerstones of OpenSolaris getting the proper attention here - long overdue.

After reading this, it hit me that another limitation of the SVR4 packaging system that we've been saddled with for years is the arbitrary limits on things such as the length of the package "abbreviation" (PKG in pkginfo).

Although we can certainly pkgtrans a package image into a file with any filename, it would seem to be important to have package abbreviations/names of any length when you move to a repository distribution model for packages/package sets. Moving from a limit of 9 chars to 32 was an improvement, but once again, an arbitrary limit has been set.

One thing that can be learned from the RPM packaging model is that it's quite helpful to see very descriptive abbreviations/names for our packages. In short, that arbitrary limits of any kind are something to be avoided.

Granted, this is a single nit, and you may have intended to classify the "removal of arbitrary limits" under one of your headings above. The problem just seems to have been so pervasive for so long that there's the danger that this issue will be overlooked as an improved (or new) packaging system is brought into being.

Posted by Gordon Marler on July 23, 2007 at 02:16 AM PDT #

did you look at Conary? I'm guessing you didn't.

Posted by anon on July 23, 2007 at 05:49 AM PDT #

@Tim: I would expect to make this kind of assessment via a dry-run option, but I suppose image-to-image comparison is possible. @Richard, Shawn: We're going to take a position on this issue, but I'll leave it for a future post. @nacho: Yes, this outcome is one of the benefits of expecting ZFS. What functionality is preserved on non-ZFS root filesystems I think is still open. @ux-admin: You're right in that this listing says little directly about dependencies or namespaces. I'll try to get to that in a future post--we have actually thought a fair bit about it. @Gerry: Already exchanging messages with Chris K. I think your conditional dependency idea is interesting, but potentially demanding on the end administrator. Will explore further. @Gordon: Agreed. @anon: You guessed wrong. Thanks for all the feedback. -- Stephen

Posted by Stephen on July 23, 2007 at 06:49 AM PDT #


Posted by Mikael Gueck on July 23, 2007 at 09:55 PM PDT #

I would add to the old inputs, "supportable patch mechanism." And to the new inputs: Reversable atomic transaction based patch and package mechanism. In other words, treat the installation of a each package (or patch==basket of packages) as a transaction. If anything in that transaction fails, it should be possible to return to exactly the state before the package was installed. This isn't necessarily true with scripted or poorly written packages but we can force reversible behaviour by requiring packages to snapshot the filesystems they touch before and after the install.

Posted by bnitz on July 25, 2007 at 08:25 PM PDT #

Would be great if Project Indiana would do "application virtualization" (every application bundled inside one container) instead of simple "package management"...

Posted by probono on July 30, 2007 at 01:39 AM PDT #

I like the existing SVR4 packaging mechanisms. They are quite powerful, to a degree that most people don't realize. The "new requirements" you list here, are rather high level, handwavy things. I would be interested in reading a more concrete, cross-referenced list of 1. which of the above "requirements", the existing packaging tools do not currently meet 2. Which of those, could be met by the existing packaging tools, with small compatible adjustments 3. which of those, could NOT be met by tweaking existing pkg tools, and why?

Posted by Philip Brown on August 02, 2007 at 08:22 AM PDT #

I simply can't wait for this! I've been at the mercy of the Solaris package system for years, and now this promises to put the Admin into complete control of the system. I especially appreciate the ideas of including SMF and ZFS as indispensable parts of the package system.

Posted by Dale Sears on August 03, 2007 at 04:32 AM PDT #

@Mikael, Dale: thanks. @bnitz: Transactional behaviour is a requirement, although I don't agree that the current patching model is a good one to preserve. @probono: If you mean that "each application inside its own container", then we do see the easy construction of such images as a goal. @Philip: I would in fact argue that the SVR4 package mechanisms are insufficiently constrained, and have ignored, if not exacerbated, problems mentioned in the new inputs section. Now that I'm back from a brief vacation, I hope to give more of the details that folks have been asking for. -- Stephen

Posted by Stephen on August 03, 2007 at 04:39 AM PDT #

themselzes is a Dvorak typo.

Posted by bustos on August 03, 2007 at 11:26 AM PDT #

I'd be interested to here why you think the v.4 package format has created problems (assuming I didn't misread that comment).

Ultimately, a package format is an archive and a handful of scripts and metadata. I can't immediately see how a different archive format and set of scripts would make much difference (I think of ways that additional metadata and such may make things easier, however, I think).

The only bit in "New inputs" that seems to match with this is "Developer Burden", and I don't think that's as much to do with the format as the tools used to create it.

Posted by richlowe on August 04, 2007 at 08:34 PM PDT #

Post a Comment:
Comments are closed for this entry.



« October 2016
External blogs