Reduced Package Repository for Initial Installs
By ckamps on Oct 24, 2009
In this post I explain how I've used the pkg(5) system's content retrieval client, pkgrecv(1), to establish a reduced form of the OpenSolaris development package repository on a local system for the purpose of optimizing my Automated Installer (AI)-based installations.
In my recent post on Initial Experiments with Base Install Profiles, I described how I am using the OpenSolaris Automated Installer and VirtualBox to experiment with heavily reduced or minimized installations of OpenSolaris. Use of a local copy of a repository will dramatically increase the speed of installing these heavily reduced installations.
Use of a local repository also enables me to begin experimenting with publishing and using an experimental base installation profile metapackage. Use of such a metapackage will greatly simplify the AI manifest by representing all of the base install packages in a single metapackage. Once a developer includes the base metapackage reference in his AI manifest, all he needs to do is add references to other packages to produce his custom installation profile based on the experimental base profile.
Prior topics in this series were:
- OpenSolaris Automated Installer with JeOS and VirtualBox
- Initial Experiments with Base Installation Profile
Only a Subset of Packages are Required: Since my experiments are dealing with heavily reduced installations of OpenSolaris, I don't need to have the full fledged repository hosted locally to support the initial install process. Only the initially installed packages and their dependencies need to be present in the local repository.
Access to All Packages Required After Initial Installation: After initial installation occurs, the installations are configured to depend on the official remote development repository or a suitable mirror for ongoing access to add-on packages and updated packages.
Development Package Repository: Since my experiments revolve around the latest stable development builds of OpenSolaris, I need to host part of the development package repository locally.
Local Traffic Only: Although not a strict requirement in my case, I'd prefer to keep all initial installation traffic limited to my local network.
ISO Image of Release Repository: Although an ISO image of the 2009.06 release repository is available, it doesn't help me much because it's a) the release rather than the development repository and b) it has everything and the kitchen sink (download size is ~7 GB) when all I need is a greatly reduced set of packages.
Mirroring Repositories: Although the process of mirroring the OpenSolaris pkg(5) development repository is well documented, mirroring involves maintaining a complete copy of all of the binary package content on the local system. Since mirroring a repository entails maintaining a copy of all of the packages and going outside the local network to access the origin repository for catalog and metadata, the mirroring approach didn't really fit my requirements.
pkrecv(1): Using publishing tools to establish a subset of a repository for initial installation purposes might seem unconventional, but recent enhancements to pkgrecv(1) make it well suited for my use case: copying just 99 packages out of several thousand to a local repo. Execution of the following command receives a copy of the specified package from the remote repository and publishes it to a local repository - all without having to explicitly start a target repo and invoking pkgsend(1). Very cool.
pkgrecv -s http://pkg.opensolaris.org/dev -d file:///repos/devbase SUNWckr@0.5.11,5.11-0.125:20091013T054131Z
Note: I am using development build 125 of OpenSolaris in this example.
Based on loosely following Rudolf Kutina's experimental repo-self.sh script, here are the steps that I took:
- Created new virtual disk to hold the repository
- Obtained list of packages from base installation
- Copied packages to local repository
- Verified the results
- Configured local depotd service
1. Created new virtual disk to hold the repository
In VirtualBox I created a 12 GB disk named Repository.vdi and attached it to my install server VM.
After booting install server, I determined the device name of the newly added disk using the format command:
$ pfexec format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <DEFAULT cyl 4093 alt 2 hd 128 sec 32>
1. c1t1d0 <DEFAULT cyl 1564 alt 2 hd 255 sec 63>
The first disk is my boot disk while the second one is the disk I created in Virtual Box to hold the repository data.
I used ZFS to create a new storage pool consisting of the new disk and a filesystem to hold the development repository:
$ pfexec zpool create repos c1t1d0
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
repos 73.5K 11.8G 21K /repos
rpool 3.32G 4.50G 87K /rpool
$ pfexec zfs create repos/dev-base
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
repos 105K 11.8G 21K /repos
repos/devbase 21K 11.8G 21K /repos/devbase
rpool 3.32G 4.50G 87K /rpool
$ pfexec zfs set atime=off repos
2. Obtained list of packages from base installation
Moving from the install server to one of the VMs where the base installation profile was already installed, I captured the complete list of installed packages:
$ pkg list -Hv > pkglist.txt
I transferred this list to the install server so that I could use it as the input to the pkgrecv-based copying process.
3. Copied packages to local repository
I cribbed a bit off Rudolf's repo-self.sh script, but given the recent enhancements to pkgrecv(1), I was able to make the copying process simpler (if you're on SWAN, you should use the internal copy of the dev repository):
cat $1 | sed 's/pkg:\\/\\/opensolaris\\.org\\///' | cut -f 1 -d ' ' > $TEMPDIR/packages.txt
for i in $(cat $TEMPDIR/packages.txt)
pkgrecv -s $REMOTE_REPO_URL -d $LOCAL_REPO_URL "$i"
Execution of this script with the input file described earlier yields this type of output:
pkgrecv -s http://pkg.opensolaris.org/dev -d file:///repos/devbase SUNWcs@0.5.11,5.11-0.125:20091013T054414Z
Retrieving manifests for package evaluation ...
Retrieving package content ...
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 2058/2058 18.5/18.5
Republishing pkg:/SUNWcs@0.5.11,5.11-0.125:20091013T054414Z ...
Refreshing repository search indices ...
4. Verified the results
Now all I needed to do was to fire up the pkg.depotd(1M) server to serve up the content of the newly established repository:
$ pfexec /usr/lib/pkg.depotd -d /repos/devbase -p 10001
By accessing the depotd server's URL, I was able to confirm that at least superficially, the desired content was being made available by the depotd server:
5. Configured Local pkg.depotd Service
Finally, I configured the pre-installed depot SMF service on the install server to automatically start with the proper properties:
$ pfexec svccfg -s application/pkg/server setprop pkg/port = 10001
$ pfexec svccfg -s application/pkg/server setprop pkg/inst_root = /repos/devbase
$ pfexec svcadm refresh application/pkg/server
$ pfexec svcadm enable application/pkg/server
In my next post I described how with just a few tweaks to the AI manifest and the post-install steps, I made use of this local, heavily reduced repository to achieve 7-minute installations of OpenSolaris.