Nearly Instantaneous Deployment of Linux Zones

One of the limitations of the lx brand (AKA, a Linux userspace hosted on top of Solaris) is that we only support whole root zones. This means that each zone contains a full copy of the Linux software, which consumes a lot of disk space. The native Solaris brand supports whole root zones as well as sparse root zones. A sparse root zone has its own private copy of some things (like most of /etc and /var), but the vast bulk of the Solaris software is lofs-mounted from the global zone, allowing a sparse root Solaris zone to consume less than 100MB of disk space.

In addition to consuming a lot of disk space, limiting lx to supporting whole root zones means that zone installation can take quite a bit of time.

To address both the time and space issues, we can take advantage of the new ZFS clone functionality. In brief, ZFS allows you to make a snapshot of a filesystem, capturing the filesystems's state at that moment. ZFS then allows you to make an arbitrary number of clones of that snapshot. A clone is actually a copy-on-write copy of the snapshot, meaning that it consumes no disk space of its own until you start modifying it. As you make changes to data in the clone, ZFS makes a new, private copy of that data, which is then assigned to the clone.

When installing a new Linux zone, instead of installing a fresh copy of the Linux software in the zone, we can use the ZFS clone functionality to copy the software from an existing zone.

Creating the initial 'master' filesystem

Before you can start using the ZFS clone functionality to install new Linux zones, you need to construct the initial snapshot on which the clones will be based.

Full documentation on how to create ZFS filesystems and install Linux zones is beyond the scope of this post. For that information, you should seek out the official documentation in both the ZFS and BrandZ communities. The short version is:

        fireball# zfs create zpool/demo
        fireball# zfs set mountpoint=/export/zones/demo zpool/demo
        fireball# zoneadm -z demo install -d <path to centos tarball>

Once the zone has been installed, you need to capture a snapshot of it:

        fireball# zfs snapshot zpool/demo@template

At this point you should have something like this:

        fireball# zfs list | grep demo
        zpool/demo             632M  14.1G  1.91G  /export/zones/demo
        zpool/demo@template       0      -  1.91G  -

Installing the new zone

Now that you have your initial zone created, you are ready to start installing new zones using the initial zone as an installation source. To do so, you will need a version of the /usr/lib/brand/lx/lx_install script that has the clone support added. You can get the modified lx_install script right here:

This script clones the snapshot to zpool/<zonename> and mounts it at <zoneroot>. This script is never executed directly - you should continue to use the zoneadm install command to install the zone. However, where you used to specify -d <linux source location> on the install line, you will now specify -z <zfs snapshot>. The zoneadm tool calls the lx_install script at the appropriate time during the zone construction process, and it will pass the -z option through to the script.

Two things to note about the script: it hardcodes the assumption that the ZFS pool from which the filesystems should be created is called zpool; and it assumes that you want the new filesystem to be created at zpool/<zonename>. If this were being done the Right Way, instead of as an on-the-train hack, these limitations would have to be removed. If these assumptions don't match your system, feel free to modify the script to change them.

The bottom line

        <configuration of zone ldemo1 left as an exercise for the reader.>

        fireball# time zoneadm -z ldemo1 install -z zpool/demo@template
        lx_install: Cloning ldemo1 from zpool/demo@template
        Installation done.

        real    0m0.630s
        user    0m0.047s
        sys     0m0.106s
        fireball# zfs list zpool/ldemo1
        NAME                   USED  AVAIL  REFER  MOUNTPOINT
        zpool/ldemo1            97K  14.1G  1.52G  /export/zones/ldemo1

A new Linux zone was installed in under 1 second, and consumes less than 100K of disk space. Full disclosure: after booting and halting the zone once, which (among other things) modifies some log files in /var, the zone has grown to a whopping 1.65MB:

        fireball# zfs list zpool/ldemo1
        NAME                   USED  AVAIL  REFER  MOUNTPOINT
        zpool/ldemo1          1.65M  14.1G  1.52G  /export/zones/ldemo1

Not only does this save you time and disk space, it also saves time and effort in configuration. You can create a single master zone, modify and configure it to your heart's content, and then take a snapshot of the configured zone to be used as a template when creating new zones. Any new zone that is cloned from the template will automatically inherit all of the configuration.

Removing a zone

When it comes time to uninstall or delete a zone, we don't need to take the time to walk through the filesystem deleting each file and directory individually. Instead, we can simply throw away the clone, which takes a fraction of the time. In fact, given the way ZFS implements cloning, deleting a file in the clone constitutes a change in the filesystem, which triggers a copy-on-write operation in the clone. So, an rm -rf of a clone is even more expensive than on a typical filesystem.

To avoid this cost, a zone uninstall should be done as follows:

        # zfs destroy zpool/<zonename>
        # zoneadm -z <zonename> uninstall


Finally, since this is such an obvious way to leverage the ZFS clone functionality, it should not be surprising to learn that we are exploring the possibility of using it to clone Solaris zones as well. Since we hope to make this a generic zones feature, it is highly unlikely that the Linux-specific hack I've described here will find its way into the final product. Until this is done the Right Way in zones, we'll be using this hack in-house.

Technorati Tags:


We think alike!

Posted by John Clingan on March 09, 2006 at 05:53 AM GMT-05:00 #

Dang it, here is a linked version

Posted by John Clingan on March 09, 2006 at 05:54 AM GMT-05:00 #

Post a Comment:
Comments are closed for this entry.



« April 2014