By barts on Apr 05, 2011
The marketing (er, product management) folks have been putting together various pieces of information on IPS. You can find all the information at this link.
It's been a while since I blogged on packaging... and we've been busy. I've had several people ask for our developers documentation, but that's still being written, so I thought I'd blog about how to publish a simple package with IPS.
I've been having mysterious networking issues (somewhere, a router doesn't like my system), so I decided I'd package a tool I use often to diagnose flaky networks - mtr. Like many open source programs, it needed some work to get it to compile properly on Solaris. The primary issue is that we've currently hidden libncurses off in /usr/gnu/lib, which is silly - there is no "Solaris" version in /usr/lib, so it should have gone there. Secondly, the configure.in file insists on bringing in -ltermcap - but this causes problems since it exports data that differs in size from the definitions in libncurses. So, after commenting out the offending line:
# AC_CHECK_LIB(termcap, tgetent)
export CC=cc export LDFLAGS="-zguidance -zdirect -zlazyload -zignore -R/usr/gnu/lib -L/usr/gnu/lib -lncurses" export DESTDIR=/home/barts/publish/proto ./configure --exec-prefix=/usr --prefix=/usr make install
The -zignore option tells the linker to not include the libraries that aren't actually needed; for some reason configure insists on placing the transitive closure of the library dependencies on the link line rather than just the ones needed by mtr itself.
So, after all this fiddling, we end up w/ two files installed into our proto area: a man page: usr/share/man/man8/mtr.8 and usr/sbin/mtr, the binary itself. The installation of the man page into section 8 isn't right for Solaris, but we can fix that in packaging. We should patch the man page to correct its section number, but this blog entry is supposed to be about packaging, not coercing OSS into the right form for Solaris.
Well, to package the software the first thing we need is a manifest, which contains actions - files, directories, etc. The pkgsend command will generate a manifest for us from a variety of different inputs, including SVR4 packages, tar files, and directory hierarchies. The latter will be useful here:
: barts@cyber; pkgsend generate /home/barts/publish/proto dir group=bin mode=0755 owner=root path=usr dir group=bin mode=0755 owner=root path=usr/share dir group=bin mode=0755 owner=root path=usr/sbin dir group=bin mode=0755 owner=root path=usr/share/man dir group=bin mode=0755 owner=root path=usr/share/man/man8 file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man8/mtr.8 file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr : barts@cyber;
Here we can see the directories with default owner & group permissions, and our two files. Note that the files have a path to the content in the proto area, and a path attribute that indicates where to install them. They match by default, but we can edit the installation directory in the manifest; this is often easier than getting the configure script to move things around for us. We also want to delete the directory entries; they're not needed for this package as they already exist, and we would need to make them match exactly.
So, let's dump that in a file:
: barts@cyber; pkgsend generate /home/barts/publish/proto > mtr.p5m
and use pkgmogrify(1) to edit the manifest. Of course we could do this by hand - but automating the transformations needs to produce a finished manifest from the raw compilation is an important part of making software changes easy, and reducing mistakes.
If you read the man page for pkgmogrify, you'll find several examples; here we simply modify the manifests with the following transforms:
<transform -> edit path man8/mtr.8 man1m/mtr.1m> <transform dir -> drop>
So, with the above lines in a file named section8, we can invoke pkgmogrify:
: barts@cyber; pkgmogrify mtr.p5m section8 file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr : barts@cyber;
Ok, we now dump this into another file and run pkgdepend generate on the manifest to discover our dependencies:
: barts@cyber; pkgmogrify mtr.p5m section8 > mtr.p5m.1 : barts@cyber; pkgdepend generate -md /home/barts/publish/proto mtr.p5m.1 > mtr.p5m.2 : barts@cyber; cat mtr.p5m.2 file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr depend fmri=__TBD pkg.debug.depend.file=libresolv.so.2 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libnsl.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libsocket.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libgdk_pixbuf-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libgtk-x11-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libm.so.2 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libncurses.so.5 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libgdk-x11-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libc.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libglib-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libpthread.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require depend fmri=__TBD pkg.debug.depend.file=libgobject-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require : barts@cyber;
As you can see, the manifest now contains dependency prototypes specifying which files are needed. We can now run the resolve step to examine the packaging manifests on the system (or in a repository) to determine on which packages we depend. This takes a bit of time to run, since it loads in all the manifests first; this still needs some performance work. It dumps its output in a file with the same name as the input but with ".res" appended; this is so multiple packages can be resolved at once:
barts@cyber; pkgdepend resolve -m mtr.p5m.2 : barts@cyber; cat mtr.p5m.2.res file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr depend fmri=pkg:/firstname.lastname@example.org type=require depend fmri=pkg:/email@example.com type=require depend fmri=pkg:/firstname.lastname@example.org type=require depend fmri=pkg:/email@example.com type=require depend fmri=pkg:/firstname.lastname@example.org type=require : barts@cyber;
The dependencies have all been resolved... the version numbers match whatever packages I have installed on my system. Now let's create a file based repository, set the publisher name and publish the package:
: barts@cyber; pkgrepo create /home/barts/publish/repo : barts@cyber; pkgrepo add-publisher -s /home/barts/publish/repo bart : barts@cyber; pkgsend -s /home/barts/publish/repo publish -d /home/barts/publish/proto email@example.com,5.11-0.1 mtr.p5m.2.res pkg://firstname.lastname@example.org,5.11-0.1:20110328T202205Z PUBLISHED : barts@cyber
We can now install this package; let's do a dry run with lots of output so we can see what will happen:
: barts@cyber; sudo pkg install -nvvg /home/barts/publish/repo mtr Password: Packages to install: 1 Create boot environment: No Rebuild boot archive: No Changed fmris: None -> pkg://email@example.com,5.11-0.1:20110328T202205Z Services: None Actions None -> pkg://firstname.lastname@example.org,5.11-0.1:20110328T202205Z None -> set name=pkg.fmri value=pkg://email@example.com,5.11-0.1:20110328T202205Z None -> file e76b633a531d5b1081d13e449efe7534df25105f chash=16844052aeb4716f4b7dec96d04d76e709c1885f group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m pkg.csize=2047 pkg.size=4939 None -> file f71673bbaa607a5aeb24f73849154b7034ed6026 chash=88d4f4183ab9ff6866c5a699a04254aa27f6b260 elfarch=i386 elfbits=32 elfhash=735cf51f4cbaadb2c1fc9f935b59343354a53b56 group=bin mode=04755 owner=root path=usr/sbin/mtr pkg.csize=48466 pkg.size=131228 None -> depend fmri=pkg:/firstname.lastname@example.org type=require None -> depend fmri=pkg:/email@example.com type=require None -> depend fmri=pkg:/firstname.lastname@example.org type=require None -> depend fmri=pkg:/email@example.com type=require None -> depend fmri=pkg:/firstname.lastname@example.org type=require : barts@cyber;
Looks plausible; let's install and test it:
: barts@cyber; sudo pkg install -g /home/barts/publish/repo mtr Packages to install: 1 Create boot environment: No DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 2/2 0.0/0.0 PHASE ACTIONS Install Phase 8/8 PHASE ITEMS Package State Update Phase 1/1 Image State Update Phase 2/2 PHASE ITEMS Reading Existing Index 8/8 Indexing Packages 1/1 : barts@cyber;
Seems to work; user interface is a little ... clunky... when updating the host to be tracerouted....
Ok, this is a good starting point. Some more things to do before declaring success include:
Fix the man page as noted earlier – some sed script seems appropriate.
Add a facet to the man page so that it's not installed if the user doesn't want documentation. This is easily done w/ pkgmogrify.
Adding a file containing exec-attr info for this binary to match what Solaris does w/ /usr/sbin/traceroute.
Add gnome-menu times, icons, etc. and add appropriate restart_fmri tags (see pkg(5)) so that gnome caches are properly refreshed.
A quick recap:
We used pkgsend generate to generate a manifest... and modified the generated manifest with pkgmogrify. We then used pkgdepend to generate and resolve package dependences... and then published with pkgsend again.
There are four steps to publishing your own packages: generate, transmogrify, determine dependencies, publish.
More on this topic later on... hope to get some of the commonly used pkgmogrify transforms into /usr/share/ips or similar; this will make things easier yet.
During pkg(5) development it has become quite clear that computing the correct set of packages to install or upgrade is a non-trivial task. Initially, we started delivering pkg(5) with a solution engine that simply took the latest available packages. This worked so long as we only delivered packages that were all compatible, no third party publishers existed, and users were happy staying on the bleeding edge.
Since none of these conditions were maintainable, a more sophisticated solution was essential.
We gained some breathing room with the introduction of incorporation dependencies. Such a dependency in a package specifies the version (at a variable level of precision) of compatibility with another package. We have used a package full of these dependencies (termed an incorporation ) during OpenSolaris development to insure that the various operating system packages from pkg.opensolaris.org come from the same build - that there's no way to get build 123's drivers, but build 127's IP stack. In effect these packages define surfaces of compatible package versions, and allow package maintainers to refactor their packages, exchange content, etc. without the need for dependencies at the package level that would prevent incompatible packages from appearing on the same system. The use of incorporations has allowed us to continue OpenSolaris development with a solver that first applied all constraints imposed by installed incorporations, and then attempted to install the latest possible packages.
As we anticipated, however, the existing solver's deficiencies have become steadily more limiting. Since the existing solver doesn't support back-tracking (e.g. revising a selected package version selection backwards during solution generation), trying to install third party packages that were published with different versions for various OpenSolaris releases was difficult if your machine was not running the most recent releases, and dealing with nested incorporation dependencies was impossible. I experimented with a more conservative solver that attempted to upgrade as little as possible; this made upgrades across multiple releases painfully slow, however, and still didn't deal with newer versions being un-installable due to missing dependencies, etc. In addition, we received multiple requests for exclude-type dependencies that would allow packages to prevent installation of incompatible packages; this was definitely outside the capabilities of our naive solver.
Conventional solvers iteratively attempt to satisfy package dependencies by walking the package dependency graph and selecting package versions to try; our experience w/ the large numbers of package being generated by biweekly (or nightly or even every push) builds indicated that such an approach would be very slow in some cases as the order of graph traversal might lead to the need to explore thousands of possible solutions. Reading some of the research (in particular, the EDOS and ZYpp projects) indicated significant interest/progress in attacking packaging computations as boolean satisfiability problems, and we decided to try that approach.
Boolean satisfiability solvers need their problems posed in conjunctive normal form; e.g. as a conjunction (logical ANDing) of clauses containing disjunctions (logical ORed) of variables (or their negation). By assigning the presence of a particular version of each package a unique boolean variable, we can construct clauses for dependencies and existence that allow the solver to compute solutions to our packaging problems. For example, given four possible versions of package A, the fact that we can only install one version of a time of package A yields the following set of clauses (assigning A1 to indicate the presence of version 1 of A, and ! to represent negation and | to represent disjunction):
Clearly, large numbers of versions of a package can generate an inordinate number of clauses; more on that a bit later.
If a require dependency exists on a particular package version, indicating that that version or newer is required, clauses are generated to describe that dependency. For example, if package B@1 depended on A@2:
If an optional dependency exists on a particular package version, that indicates that if that package is installed it must be at least at the specified level. Here, we end up excluding versions we don't want.... For example, if package B@1 optional depended on A@3:
Our incorporate dependencies that specify the version needed also generate such exclusionary clauses. For example, if package B@1 incorporates A@3 (e.g. if A is present it must be at version 3):
Lastly, actual exclude dependencies indicate that if present, the depended upon package must be at the specified level or lower. If package B@1 has a exclude dependency on A@3:
Once the packaging problem can be described as a series of clauses, it can be passed to a SAT solver for solution; the solver generates a set of packages that will meet the specified criteria, or declare that no solution exists. The number of variables used in the solver is the number of package versions installed plus those considered; the number of clauses used depends on the number of versions of a package and the types of dependencies. To minimize the size of the problem and the resulting memory footprint, we don't simply generate clauses for all possible packages and their versions; since we know that packages are not allowed (normally) to decrease in version, we eliminate from consideration earlier versions of any installed packages, and any packages excluded by incorporations we're unwilling to change. We also eliminate duplicate packages from publishers we're not willing to consider. This "trimming" phase is actually the most time consuming phase of generating the list of packages to install.
If all we needed was a single solution, this would be adequate; however, we'd like to find solutions that meet our definition of optimal. We do this by finding solutions and then looking for better ones by resolving the problem w/ additional constraints excluding areas we don't consider optimal. For example, when installing a package we're willing to update other packages if needed (within incorporation constraints, of course), but we'd like to minimize such changes. This is an area we're still exploring, and a likely topic for additional blogging.
We choose the Minisat solver as a good place to start as they had built a C version of Minisat that would be easy to link into our packaging system, which is coded in Python. About the only changes I made were to keep track of the clauses fed to the solver, so that it is possible to cheaply revise solutions by caching copies of the current state of the solver. Introduction of the SAT solver awaited Shawn Walker's very nice catalog rewrite which added dependency information into the package catalog, as it was critical for perfomance reasons to not have to download hundreds of manifests to do package planning. I integrated the new solver into the packaging gate for build 128 of OpenSolaris, now available from pkg.opensolaris.org/dev and other mirror repositories.
One of the interesting implications of the solver change has been that it is more difficult to determine just why there are no image-updates are available. The previous solver would fail (badly) when encountering missing packages in dependencies, etc; the new solver just considers packages with missing dependencies as uninstallable and thus unavailable for upgrade. Image-update will now very rarely generate any error messages, which is nice from a user aspect but makes debugging mis-configured or broken builds more difficult than before. If you think you should be able to upgrade, try explicitly installing the version of entire (the incorporation that currently controls what software build you're running) you think you should be able to install w/ -nv as flags; this will generate much more verbose debugging output when no solution can be found, as the packaging system has some idea of what you'd like to achieve other than just "get me newer bits if you can". Generation of more useful error messages will remain an important area for further work.
Other interesting areas for further enhancements enabled by the SAT solver include constructing the entire incorporation as an incorporation of other incorporations; this will allow developers to easily run the latest kernel and older window system bits, or vice versa. We're also considering conditional (package A requires package B if package C is installed) and disjunction (package A requires package B or package C) dependencies to solve some of the more complex package configuration requests we've seen.
An engineer's viewpoint on Solaris...