Tuesday Oct 13, 2009

testing ON changes with OpenSolaris installers

Installers are funny things. They boot a reduced version of the OS in order to install it, usually in some bizarre context like netboot, or with a live CD. They have all sorts of restrictions -- not being able to write to certain parts of the filesystem, or needing to be bootstrapped from tftp or other ancient crufty protocols. Furthermore, the OS bootstraps itself with less information it has about the system's hardware and configuration than it will have in normal operation -- it needs to put together enough information on the fly and prepare the system for normal operation. But, fundamentally, the initial installer for most OSes boots essentially the same version of the OS it's about to install.

For a developer in any core part of the system that the installer uses, that means certain types of changes must be built and tested in install context. smf(5) is certainly one of those subsystems where a subtle change may have unintended consequences on the gymnastics the OS is doing in install context. Testing is key. But assembling an install image to test for Solaris 10 and earlier used to be a horribly complicated and arcane process. And most changes don't tend to require testing in install context, so people wouldn't do 'just in case' testing. They'd only test install if they really really needed to.

Recently, I needed to figure out how to do testing of nightly ON bits in install context. I started Saturday with my ON nightly repository in hand, memories of the Solaris 10 procedure fresh in my mind, and a sense of dread. But this isn't Solaris 10 anymore, and the Install team has done a great job of changing a painful procedure into something downright delightful. I decided I wanted to test a live CD, so I...

$ pfexec pkg install distribution-constructor
$ cp /usr/share/distro_const/slim_cd/slim_cd_x86.xml ~
$ pfexec zfs create rpool/dc

I modified ~/slim_cd_x86.xml to point at my ON repository first, and then find any packages it couldn't get there from a repository containing the previous build of OpenSolaris. I also didn't install the entire package, as I'm using ON development bits (don't do this on a supported system). And then ran

# /usr/bin/distro_const build ~lianep/slim_cd_x86.xml

About an hour later, I had a LiveCD image and a USB image, ready to test. So much for my sense of dread. One quick note: Distribution Constructor currently requires that you first image-update the system where you're running DC to the same install as you're attempting to build. The DC team knows it's an issue, and the workaround is quite straightforward as it's just "pkg image-update" to your repositories.

And that's pretty much it. The diffs for the DC manifest look like:

--- /usr/share/distro_const/slim_cd/slim_cd_x86.xml  Fri Oct  9 16:29:26 2009
+++ slim_cd_x86.xml  Sat Oct 10 16:53:39 2009
@@ -57,8 +57,8 @@
-				url="http://pkg.opensolaris.org/release"
-				authname="opensolaris.org"/>
+				url="http://ipkg.sfbay/on-nightly"
+				authname="on-nightly"/>
 			     If you want to use one or more  mirrors that are
 			     setup for the authority, specify the urls here.
@@ -75,15 +75,12 @@
 		     If you want to use one or more  mirrors that are
 		     setup for the authority, specify the urls here.
-		<!--
-		     Uncomment before using.
-				url=""
-				authname=""/>
+				url="http://ipkg.sfbay/dev"
+				authname="opensolaris.org"/>
 			<mirror url="" />
-		-->
-                       <pkg name="entire"/>

For cleanliness, I also updated the post_install_repo_default_authority and post_install_repo_addl_authority to point at the same URLs.

The install team does recommend running DC on a system that's already been updated to that package level. e.g. to build a 124 LiveCD, they suggest first image-updating the system to 124, then running distro_const(1M). That shouldn't always be necessary, but it's also easy to do given that DC requires a repository as input that you can just image-update from.

It's pretty nice that the install team has fundamentally converted a task that was challenging for ON developers before into something that's really quite easy. If you're interested in more, there's plenty, including documentation at the Caiman Project over on opensolaris.org.

Sunday Oct 11, 2009

building an ON IPS repository

I've been working with the gracious help of Mark on making the ON consolidation create an pkg(5) repository as part of the build process. If you build the ON consolidation from source ever, this is probably interesting to you.

Our changes are destined to be integrated into the main ON gate, which should happen in November 2009 sometime (though that's subject to change and doesn't constitute a promise). We've tried to make it easy for folks to build their own ON IPS repositories for testing in advance of integration of our changes. You can access the latest instructions for building your own ON repository in the README which lives in our development mercurial repository.

If you do want to try this out, I strongly recommend subscribing to on-ips-dev@opensolaris.org, as that's where we're answering questions, giving heads-ups about important changes, and having development conversations. We've got some sizable changes coming over the next few weeks, including a protocmp which works on IPS manifests.

I'm really enjoying using the same tools as we expect our customers to use. It's now pkg image-update to update my ON development bits rather than the development-only tool bfu. pkg image-update is at least as fast, if not faster than bfu, especially over a slower link. That's because only the bits which have actually changed between versions are downloaded and updated by pkg(5). Nice. And nice that our normal upgrade experience is now as blindingly fast as ON developers have come to expect.


Liane Praza-Oracle


« October 2009 »