Cloning Isn't Just for Sheep Any More
By user12611829 on Feb 16, 2007
While it may not have the social implications nor headline appeal of the now famous Dolly the Sheep, the zone cloning feature introduced with Solaris 10 11/06 is worth further investigation. Before we do that, it is probably a good idea to review basic zone creation and installation prior to the new cloning capability.
Building Zones the Old Fashioned WayThe first step in the creation of a zone is establishing it's configuration. This is done by conversing with our friend, zonecfg(1M), who handles all the details of writing the configuration xml file in /etc/zones and updating the zones index file /etc/zones/index.
Such a conversation might go something like....
# zonecfg -z zone1 zone1: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:zone1> create zonecfg:zone1> set zonepath=/zones/zone1 zonecfg:zone1> add inherit-pkg-dir; set dir=/opt; end zonecfg:zone1> add net; set physical=iprb0; set address=192.168.100.101/24; end zonecfg:zone1> add fs; set dir=/export; set special=/export; set options=[rw,nosiud]; set type=lofs; end zonecfg:zone1> verify zonecfg:zone1> commit zonecfg:zone1> exit #
If you grok zones then you recognize this as a typical sparse root zone. If you have attended one of my zones best practices workshops then you will also notice that I'm following my own advice and making /opt an inherited package directory.
A quick check to make sure all is well.
# zoneadm list -cv ID NAME STATUS PATH BRAND 0 global running / native - zone1 configured /zones/zone1 native
All is as it should be (which is always the case for a how-to example).
The next step is a rather magical affair where the zoneroot is populated. This process is initiated by uttering the following sequence
# zoneadm -z zone1 installOnce spoken, fantastic things start happening behind the scenes - all of them by our good friend Live Upgrade. The actual sequence of events is something like
- Create the new zoneroot if it doesn't already exist. If it does exist make sure
the permissions are set to 700 and it is owned by [0,0].
- Mount all of the inherit-pkg-dir and file systems listed in the zone configuration
- Create a candidate list of files for the new zoneroot by looking at the
global zone contents file /var/sadm/install/contents.
On my laptop daily driver, this totals approximately 2 million files.
- Pick from this list all files that should be delivered to the new zone root by
removing all files from packages that are marked as global zone only (SUNW_PKG_THISZONE is
set to true)
We're still over 2 million files, folks!
- From the remaining list of files, remove all of those that will be delivered via
This is why I like inherit-pkg-dir. We are now down to about 2,300 files. If not for inherit-pkg-dir I would be hitting my boss up for a lot more storage.
- Copy all of these files from the global zone into the new zoneroot, replacing
commonly edited configuration files with those that were originally delivered with
the package (ie /etc/passwd).
- Once the files are in place there is one more step to perform. Some of the
package have preinstall and postinstall scripts that might do something important.
These need to be run, even if all of the files are delivered via inherited directories.
So in package dependency order, all of the packages identified as applicable to the
new zone (SUNW_PKG_THISZONE=false) are installed sequentially.
- Update the zones index file /etc/zones/index marking the new zone as installed.
- Unmount all of the file systems mounted in step 2.
Enough of the theory, how long did this really take ?
On a relatively clean Nevada (aka OpenSolaris Community Edition) install it was almost 10 minutes. The output is below and I have annotated it with the installation steps outlined above.
# time zoneadm -z zone1 install   Preparing to install zoneBut we're still not done, are we ? We still have first boot processing which includes initial population of the SMF repository (which is O(number of services, speed of disks)) and system identification (which is either constant if a sysidconfig file is supplied or O(Bob's increasingly bad typing rate) if we choose an interactive dialog).
.    Creating list of files to copy from the global zone.  Copying <1934> files to the zone.  Initializing zone product registry. Determining zone package initialization order. Preparing to initialize <1290> packages on the zone. Initialized <1290> packages on zone.   Zone is initialized. Installation of <1> packages was skipped. Installation of these packages generated warnings: The file contains a log of the zone installation. real 9m38.951s user 1m26.582s sys 2m51.252s
For this example the first boot process took about 3 minutes to complete.
We now have a pristine zone, ready for work. But there is more to do, isn't there ? We have to install some software, or at least configure software that is already present. In fact, these customizations might be more complicated than the zone installation process. If I had invested in developing automation scripts or was using some form of advanced provisioning technology this might not be a big deal. If I'm doing this manually then it may be quite a bit of work - and work that I don't want to repeat with regularity. In other words: I'm not likely to use lots of zones and I don't particularly look forward to OS updates.
Let's look at this a bit more and see if we can make this any easier.
This example comes from my (about to be posted) Zones workshop. In our new non-global zone we will replace the Solaris version Webmin with the community release from webmin.com.
A quick pkgchk(1M) of SUNWwebminu shows that its contents are in /usr/sfw and SUNWwebminr deposits it's payload in /etc/webmin and an SMF manifest in /var/svc/manifest/application/management. Performing the same task on the community edition of Webmin shows that it will install in /etc/webmin and /opt/webmin. The clashing of /etc/webmin indicates that these cannot easily co-exist, but complete replacement is possible (all inherit-pkg-dir destinations are contained in a single directory).
So begin by removing the Solaris version of webmin. This is all done in our new zone.
# zonename zone1 # pkgrm SUNWwebminr SUNWwebminuAt this point the root package SUNWwebminr is completely gone and SUNWwebminu is marked as partially installed. One more pkgrm(1M) and it will gone, at least as far as our package contents are concerned. The bits in /usr/sfw are still there, but without the configuration files in /etc/webmin, they are just that, bits in a directory.
pkgrm: ERROR: unable to remove /usr/sfw/bin /usr/sfw /usr ## Updating system information. Removal of partially failed.
# pkgrm SUNWwebminu The following package is currently installed: SUNWwebminu Webmin - Web-Based System Administration (usr) (i386) 11.11.0,REV=2007.01.23.02.15 Do you want to remove this package? [y,n,?,q] y ## Removing installed package instanceNow to install the new webmin package. While you weren't looking I put the package in /var/tmp. But there are some things to do before we can proceed. Remember, the package wants to write into /opt/webmin, but /opt is read-only. We can do a couple of things: mount a writable file system (LOFS, local real disk or NFS) onto /opt/webmin in our new zoneroot or we could create a symbolic link for /opt/webmin that would point somewhere writable. The link is much less confusing, so let's go that route this time.
(A previous attempt may have been unsuccessful.) ## Verifying package dependencies in global zone ## Processing package information. ## Removing pathnames in class ## Updating system information. Removal of was successful.
In the global zone do something like
# ln -s /local/webmin /opt/webmin # mkdir -p /zones/zone1/root/local/webminNow we are ready to proceed. In zone zone1, do the following
# pkgadd -d /var/tmp/webmin-1.320.pkg The following packages are available: 1 WSwebmin Webmin - Web-based system administration (all) 1.320 Select package(s) you wish to process (or 'all' to process all packages). (default: all) [?,??,q]:Now we have a nicely customized non-global zone with one application ready to go. It wasn't all that bad, but there were a few manual steps. Multiply this by 20 or so for all of the other applications and configuration steps that you need to do for your system standards and then by 20 or so for the numbers of zones you want to provision and it is suddenly looking like a tremendous amount of work.
Webmin has been installed and started successfully. Use your web browser to go to http://zone1:10000/ and login with the name and password you entered previously. Installation of was successful.
Until Solaris 10 11/06.
Send in the Clones: Solaris Zone CloningZone cloning is a new feature that bypasses all of the steps in the zone installation process and replaces them by copying the source zoneroot and performing a sys-unconfig(1M). Of course this makes perfect sense - if you duplicate the installation process you should get the exact same results (a wise science teacher taught me that a long time ago). So why not short cut the process and just copy the zoneroot, sys-unconfig(1M), fix up the zones index file and you are done.
But it gets better than that. If we are copying the zone root then any customization performed on that zoneroot will be preserved. This includes the SMF repository. Not only do we skip the initial import, we also preserve any customizations, such as service related security hardening. Our new cloned zone would also have the community edition of Webmin instead of the one in Solaris. And it's configured, enabled, and will start automatically when the new zone boots - without requiring me to do anything else. Now that's cool.
Let's see how all this works.
Step 1 - create a new zone configuration using our clone source as a template. We need to change the zoneroot and IP address. In more complex configurations, other attributes might need to be changed, but for this simple example this is all that is required.
# zonecfg -z zone2 zone2: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:zone2> create -t zone1 zonecfg:zone2> set zonepath=/zones/zone2 zonecfg:zone2> select net address=192.168.100.101; set address=192.168.100.102/24; end zonecfg:zone2> verify zonecfg:zone2> commit zonecfg:zone2> exitInstead of installing a new zone, let's clone from zone1.
# time zoneadm -z zone2 clone zone1 WARNING: read-write lofs file system on '/export' is configured in both zones. Copying /zones/zone1... real 0m31.135s user 0m0.431s sys 0m3.818s # zoneadm -z zone2 boot # zlogin -C zone2 (or supply a sysidconfig file)Now we're getting somewhere. Zone creation, including application configuration and setup is reduced from about 15 minutes down to 31 seconds. This is getting really cool.
Clones to the left of me, zpools to the rightBut wait, there's more! There's an opportunity to make this even more efficient by taking advantage of ZFS clones. Note that this is only available in OpenSolaris at present, but consider the implications of the following example.
Note the use of zone relocation (move) - also a nifty new feature in Solaris 10 11/06.
# zpool create zfs_zones c4d0t0s2 # zoneadm -z zone1 move /zfs_zones/zone1 A ZFS file system has been created for this zone. Moving across file systems; copying zonepath /zones/zone1... Cleaning up zonepath /zones/zone1... # zonecfg -z zone3 zone3: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:zone3> create -t zone1 zonecfg:zone3> set zonepath=/zfs_zones/zone3 zonecfg:zone3> select net address=192.168.100.101; set address=192.168.100.103/24; end zonecfg:zone3> verify zonecfg:zone3> commit zonecfg:zone3> exit # time zoneadm -z zone3 clone zone1 WARNING: read-write lofs file system on '/export' is configured in both zones. Cloning snapshot local/zone1@SUNWzone3 Instead of copying, a ZFS clone has been created for this zone. real 0m11.402s user 0m0.380s sys 0m0.412sWow! Under 12 seconds and a completely configured and ready to run zone is built. Throw in a sysidconfig file and we're ready to run. And by using a ZFS clone, almost no additional disk space was required for this new zone.
# df -k | grep zfs_zones zfs_zones 1007616 27 808469 1% /zfs_zones zfs_zones/zone1 1007616 198590 808469 20% /zfs_zones/zone1 zfs_zones/zone3 1007616 198592 808469 20% /zfs_zones/zone31GB - 200MB - 200MB should be 600MB, but it's not. Since the zones are nearly identical at this point, only 200MB is consumed from the zpool.
Practical applications of Zone CloningDevelopment environments and testbeds seem a very good fit. Build one standard configuration of a zone and clone it as necessary for each developer or test scenario. If things go wrong, which can happen while testing, just delete the zone and re-clone it. 30 seconds later you are back in business.
Shhhh - don't tell anyone, but I like the privilege restrictions of zones. I'm very likely to give a developer the root password to the zone and let them do what they need to do. The worst they can do is destroy their environment. The impact to me is two zoneadm(1M) invocations and about 30 seconds of clock time.
The better use case comes when you combine this with another new feature in Solaris 10 11/06. Zone migration. Imagine the following scenario.
- Mount a file system containing a company standard non-global zoneroot
- Attach the zone to the system (zonecfg create -a and zoneadm attach)
- Clone this new zone as many times as needed
- Detach the original zone from the server (zoneadm detach)
- Unmount the detached zoneroot filesystem
I hope that this has helped introduce you to a few new zones features with Solaris 10 11/06 (and one in OpenSolaris). As I ponder the combination of these new features I find myself beginning to think that a minimal global zone and cloned full root zones may in fact be a superior practice. We'll explore that in more detail soon.
Technocrati Tags: Sun Solaris Virtualization Zones Containers