Fattening packages - supporting multiple variants in a single package

Dealing with parts of a package

Traditionally, packaging systems have placed optional components of a package in separate packages, and established conventions for naming such components, such as -localization, -locale, -devel, -doc, etc. This method of ad-hoc decomposition makes it more difficult for GUI programs to offer the appropriate choices when selecting components, makes the introduction of new optional components difficult and makes installing documentation after the fact a painful process.

Packaging options also exist which are mutually exclusive; the typical example is which architecture the package supports. One cannot select both sparc and x86, since the two architecture's files collide - /usr/bin/ls is either a sparc binary or a i386 binary. Other examples of such colliding options include debug vs non-debug kernel binaries, and global vs nonglobal zones.

Initially we started solving the problem of selecting parts of a package using more or less ad-hoc client-side filtering, but it became clear that the publisher of the package needed to be able to describe what components intersected or were optional, and that we needed to be able to clearly distinguish between the two cases. Thus, we refer to options that may be selected or not selected in any combination, such as various locales, documentation, etc., as facets. Options which must be mutually exclusive are called variants. Both variants and facet appear as tags on IPS actions, and result in the action being selected or de-selected for installation.

An example of variant tags on an action is:

dir group=sys mode=0755 opensolaris.zone=global owner=root path=kernel/drv/amd64
  variant.arch=i386 variant.opensolaris.zone=global

Here this directory action is tagged with variant.arch=i386, indicating that this directory should only be installed on i386 machines (Solaris doesn't distinguish between 32 and 64 bit architectures) and variant.opensolaris.zone=global, which shows that this directory should only be installed in the global zone and not in local zones as those don't contain kernel components.

Note that any action in the repository can be so tagged, including dependencies.

During installation planning, all actions not applicable to the current image are removed from consideration. If a user with a down-rev version of IPS that doesn't understand variants attempts to install a fat package, either an exception will occur indicating that duplicate actions have been found, or if a set action is found with an architecture tag an assertion will trigger. In either case the solution is either to stop attempting to install fat packages, or upgrade your version of IPS to the latest available for your build.

The initial implementation in build 106 supports only variants (facets are coming later on). To faciliate detection of attempts to install packages on the wrong architecture, our publication code now inserts set actions into packages to indicate what architectures the package supports:

set name=variant.arch value=sparc value=i386 

Publishing Fat packages

We produce fat packages by publishing separately on each architecture, and then merging the resulting manifests. Actions which are identical for both architectures are not tagged w/ variant tags; those that differ or exist only for one architecture are tagged.  Packages that exist only on one architecture are tagged to indicate that as well. This step can be repeated for different variants, and more than two variants types can be combined at once, so packages containing debug kernel binaries could be merged w/ non-debug versions, and then merged across several architectures. An example of a merged manifest can be seen here. Of course, those using pkgsend to create packages can simply insert the appropriate tags directly. For the curious, the code to merge packages can be seen here. Further work is needed to improve the performance of publishing the merged packages; allowing multiple repositories to share the filestore would help greatly here.



does this mean that as a x86 user, I will download (but not install) a number of sparc binaries? If so, it is a good thing that opensolaris does not support many architectures, because this does not scale. Otherwise, that's good, and you might want to add a word on this. Facets sound like a great way to avoid the debian mess with -dev packages.

Posted by Marc on February 04, 2009 at 09:54 AM PST #


No. IPS only downloads the individual files it needs to operate on a package.

So you won't be downloading SPARC-only package files if you're on an x86 system.

Posted by Shawn Walker on February 04, 2009 at 12:31 PM PST #

In fact, if you're upgrading a package you only ever download the files that change... so if openoffice has a bug fix, you pull down only the modified files, not all N hundred MB.

Posted by barts on February 04, 2009 at 01:14 PM PST #

Thanks, sounds great!
I just hope this does not mean too much work for the server, preparing a different tarbal for each client, or too much overhead, downloading each file individually. Guess I'll have to try.

Posted by Marc on February 04, 2009 at 03:29 PM PST #

"Traditionally, packaging systems have placed optional components of a package in separate packages, and established conventions for naming such components, such as -localization, -locale, -devel, -doc, etc."

Well, this is not entirely correct:

operating systems like IRIX and HP-UX always have one single "dist" (IRIX) respectively "product" (HP-UX), which in itself can include:

- "subsystems" (IRIX)
- "subproducts", and "filesets" (HP-UX)

On HP-UX, a fileset can be found either directly inside of a "product", or inside of a "subproduct", which resides inside of a "product".

Dependencies are determined and selected automatically on both IRIX and HP-UX.

Additionally, both software management subsystems (inst(1M) and SD-UX), are capable of automatically selecting the appropriate "subsystem" or "product" / "subproduct", depending on the operating system revision and platform.

Further, IRIX's software management subsystem give the packager the option to define "subsystems" which are marked to be installed by default ("D").

A typical example would be:

eoe.ssh.sw D
eoe.ssh.man D

while "eoe.ssh.sw" and "eoe.ssh.man" would be selected for installation by default, "eoe.ssh.doc" would not. And "install upgrades" (IRIX) respectively ("match what target has" (HP-UX) would only upgrade subsystems which are already installed on the system, NOT remove entire subsystems and products, then install the newest version. One could think of it in terms of delta deployment, I guess.

What gets me is that these software management subsystems are almost 20 years old, and yet it occured to noone to study and implement them on other operating systems.

And considering that there still aren't any other software management subsystems capable of what I described above, it leaves one to pondering just what are other people thinking.

Posted by UX-admin on February 04, 2009 at 04:07 PM PST #

I have hundreds of "ABCDsomething-conf" packages, which are loaded with "preinstall", "postinstall", "preremove" and "postremove" scripts, in which a lot of heavy scripting is involved to automate a configuration of a system.

For example, I have several "ABCDnamed-\*-conf" packages which, when installed, will fully configure DNS master and slave servers, without any interaction whatsoever (everything is preconfigured in the -conf packages).

Another example might be an "ABCDapplication-db-conf" package, which in postinstall automatically creates an Oracle database, based on "ABCD_\*" values defined in the "pkginfo" file, making DB deployment consistent, encapsulated, automated, and repeatable.

Using this method, I've been able to completely automate and make repeatable configuration of tens of thousands of systems (you could say that my solutions are designed for extremely large environments).

And now my questions:

what would my migration path to IPS look like?

Which technologies does IPS offer to create packages as described above?

Posted by UX-admin on February 04, 2009 at 04:21 PM PST #

And looking further into the example you provided, what is the parameter


used for?

This is not apparent, as 32- and 64-bit binaries have different (and distinct) file paths.

Posted by UX-admin on February 04, 2009 at 05:32 PM PST #

And if I have "elfarch=<i386|sparc>", what is the purpose of declaring "variant.arch=<i386|sparc>".

Forgive me, but there are so many unanswered questions with regards to IPS.

Posted by UX-admin on February 04, 2009 at 05:35 PM PST #

UX-admin: IPS is designed to deliver files into an OS image, and produce a bootable system. Contorting the packaging and installation process to permit creation of an Oracle DB in install context makes little sense; this would require that these packages can only be installed from the same OS architecture and version as the image under construction. The fact that you're using the current packaging system as a remote execution and configuration tool is not a vindication of its design.

The tags elfbits=<32|64> are currently not being used by the client.

Posted by barts on February 06, 2009 at 03:46 AM PST #

Post a Comment:
Comments are closed for this entry.

An engineer's viewpoint on Solaris...


« April 2014