Thursday May 27, 2010


Today's my last day but one in MPK17. I've spent the last week or so transferring ideas, status, and hopefully-not-too-incomplete sketches to my colleagues. It's been just over twelve years since I finished school and began working on problems in and around Solaris, with lots of excitement along the way. And now it's time to look for new opportunities, at least after a short break.

I've set my blog here to redirect to my personal blog after a short delay; most of my contact information should be on the Internet somewhere.

Thursday Sep 27, 2007

pkg(5): project opens, development continues

I asked my teammates to take a brief break from prototyping, and we jumped from a collection of systems inside Sun to the clean project hosting at

As the migration registers, you should be able to access

If you're interested, please come check it out.

[ T: ]

Friday Sep 07, 2007

pkg(5): a no scripting zone

In my previous two posts, we examined two packaging system options—installer-specific knowledge and integrated build system—that I believe present costs that exceed their benefits. Here, we will again examine a design choice from a negative perspective: package-associated scripting.

System V packaging is rich with scripting hooks; scripts named checkinstall, preinstall, postinstall, preremove, request, and the class action scripts. Each of these scripts can do anything they like. Scripting, even in a relatively primitive shell, is an open-ended program—opaque to the invoking framework. It's difficult to catch an incorrect script prior to package publication time, which blocks our intent to prevent propagation of bad package versions. With a more limited set of actions—potentially with that limit enforced or marked—a class of incompletely known resource handling mechanisms can be kept off the most conservative systems.

One goal we have is to preserve or improve the hands-off behaviour associated with package operations. Legacy packaging allows hands-off by imposing a series of tasks on the deploying administrator. The pkgask(1M) tool can enable the deployer to develop a response to the request script; coming up with an appropriate admin(4) can restrict the framework's built-in interactive queries. (Interaction with signed packages also requires the deployer to modify their pkgadd invocation.) Removing the scripting degree of freedom means that obstacles to hands-off behaviour come solely from an interactive installer or from interactive services acting during system startup.

There's some amusingly egregious violations of the hands-off principle across the space of known packages. Less fun is that these set a bad example for later package developers.

A particularly error-prone aspect of the scripting interface in packaging comes from the variety of contexts the package developer must understand (and test within). It is legitimate to install packages on live systems, in alternate filesystem hierarchies of the same or different architecture, and in whole-root and filesystem inheriting zones; in fact, you have multiple choices about how your package should install in a zone.

We can expect the proliferation of virtualized systems, via the various mechanisms like LDOMs and xVM, to keep all of these contexts relevant as degrees of sharing make virtualization even more appealing. Making sure that the package system operates safely in these shared contexts is critical—another of our goals.

Returning to the zones case, the example pseudo-script in pkginfo(4)—a series of nested shell if ; then blocks to navigate some of these contexts—is helpful, but misleading. There is much more variable state a package developer needs to consider to reach correctness. In fact, if you aren't required to rediscover or reinvent a set of resource-handling cases for each components your package delivers, it becomes substantially simpler to make the package and return to improving the software it contains. Reducing the set of steps reduces developer burdens associated with packaging.

Two particular resources stand out: device drivers and smf(5) services. Although some limited amount of awareness—or at least easily duplicated code—makes these resources somewhat well-behaved during package operations, there are still problems that scripting presents: the addition of new contexts, the provision of multiple genealogies of copied code, and the failure to discover an associated best practice for any particular kind of resource.

There are other resources, of course; as a start, you could duplicate our survey of the ON postinstall and class action scripts.

I believe the key counterargument supporting scripting is that the set of configuration patterns on Unix-like systems is large, and that the easiest means of upgrading each of these potential patterns is to allow a complete programming environment to the package developer. Probably true, but if we look at service and application configuration with respect to when a correct configuration state is required, the update step appears to separate into three classes:

  1. Correct at system startup, no runtime context needed. These are the configuration settings that the various low-level boot components, the kernel, and the drivers need to bring the system to its running state. This class of configuration is generally limited to a specific set of resources, potentially established by a packaging system via corresponding resource-handling actions—or by an installer.

  2. Correct at system startup, requiring runtime context. These are settings where the manipulating agent might be influenced by policy or require some form of interprocess communication to effect configuration changes. smf(5) is an example of the latter, and handles its configuration evolution via the manifest-import service. Manipulation of the various local name service tables, like passwd or the RBAC configuration is another example, since data about potential principals must be correct for a group of affected services. Since such configuration can be required on the system as a result of package operations, these resources must also be handled via packaging, or require the use of an appropriate installer.

  3. Correct prior to service startup. Most service and application configuration falls into this class. It's not necessary, for instance, to bring a web server's configuration up to date if the service has no enabled instances. There seem to be a number of avenues for handling this kind of configuration: leaving it to the service or application, providing assistance via a configuration mechanism, or giving a hook where such updates can be made as needed. But the packaging system needn't provide this hook—there are a number of possible facilities, of varying suitability.

I should point out that David is making the smf(5) configuration update scenarios much more capable and precise with the Enhanced Profiles project. So, at least, a "configuration mechanism with assistance" is likely to be present soon.

Since the first and second classes and how their configuration manipulations vary in the various operating contexts are generally known, elimination of the third class makes precise, no-scripting packages a viable design choice.

That's a long series of arguments in favour of a scripting-free package system. It would be reasonable to ask: "can you actually do it?" So, as a check on our prototype, we used the branded zone capability to let us create a pkg(5)-based whole root zone. Here's a transcript

# zonecfg -z pkg_test
pkg_test: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:pkg_test> create -t SUNWipkg
zonecfg:pkg_test> set zonepath=/export/pkg_test
zonecfg:pkg_test> commit
zonecfg:pkg_test> \^D
# zoneadm -z pkg_test install
Preparing image
Retrieving catalog
Installing SUNWcs SUNWesu SUNWadmr SUNWts SUNWipkg
Setting up SMF profile links
Copying SMF seed repository
Done (115s)

There's dependency following, but no constraint handling; there's no filtering or snapshotting, but also none of the obvious performance optimizations has been implemented (for our 211MB resultant image). But the main point is: it works—installs, boots, upgrades, and still boots—with no scripting. Time for a project proposal.

[ T: ]

Friday Aug 24, 2007

pkg(5): Leaving the build system "out"

I've been busy the past weeks with school transitions and with getting the community defect tracking system requirements into a shape where we can start evaluating candidates.

Identifying the boundaries of a system during the design phase affects feasibility critically, perhaps more than any other choice. This choice reduces to "know what you're trying to create". As with my previous post, I'm going to describe something we're not doing—and explain why. I envision one more negative post, and then we'll get onto more positive expressions.

So: this packaging system does not contain a build or compilation component in its core architecture. There are pragmatic reasons for this choice, as well as technical ones.

One way to look at a packaging system is, similar to how we saw it as the connecting layer between the installer and lower-level OS services, as a way to collect and organize the set of components, binary and otherwise, into an always "bootable" flow of change. That is, various groups of people are, using their systems, emitting binaries, documents, images, and so on, that other groups of people combine into releases (or atoms of a release, like an updated package). The packaging system has to assist the latter group in that combination, by providing visibility into the completeness of the outputs of the former groups. It's less clear that the latter group needs to have the ability to construct the objects, as long as they can assess that the environment the objects need to execute can be realized.

Just as an aside, we're treating as critical design input exactly who touches and how they touch a software component as it proceeds through development, localization, release engineering, sustaining, and so forth. These touches might be reduced as the development becomes more open, but at present, they're a useful constraint on any tendency to make the system overly rigid (or overly monolithic).

The OpenSolaris consolidations—and related open source outputs like OpenJDK or OpenOffice, among others—don't presently share a common build system. In fact, one of the real impacts of Sun's progression to an open software development culture is that we're moving from being the originator or primary maintainer of 90% of the software we deliver, to a much more modest percentage—let's say 30 – 50%. Forcing a unified build system upon all of these disparate products is asserting the need for a long series of difficult conversations over many months.

Instead, let's defer those discussions, and see if we can get the same benefits while only managing the end outputs of each of the participating (or aggregated) publishers. (A byproduct is that injecting received binaries into the system is the common case, rather than a strange or special workaround.) As we noted in the earlier post, we have goals about safety, developer burden, and stopping "bad stuff" as early as possible.

Static analysis can get most of the dependency cases correct. Binaries, whether they are user applications, libraries, or kernel modules, contain a significant amount of dependency information. It turns out that many scripting languages can be roughly interpreted to determine their module requirements. Similarly, Java class files and JAR files—and even smf(5) manifests—contain a great deal of information that lets us determine the self-consistency of a system.

Of course, a program can evade these dependencies: beyond use of dlopen(3C), it can implement its own linking or overlay mechanism, or simply be implemented in a language unknown to the packaging system. The point is we can drive out most of the inconsistencies that a purely manual dependency statement allows. In fact, we can warn a developer about the incompleteness of their dependency statements, potentially correcting them: adjusting version requests, inserting omissions, even asking about possibly superfluous dependency correctness.

That said, it might be that an ideal build system is lurking out there to be layered atop this system; we'll leave room for expressions, like stating a build dependency on a certain tool, like lex(1) and yacc(1), say, so that a build system (or systems, if folks can't close those difficult conversations I mentioned) can benefit from the metadata discovered about each component. (If you're interested in constructing such a system, we thought a little about requirements for the SFW consolidation.)

[ T: ]




« July 2016
External blogs