Zones in a Flash - Literally

Fantastic improvements have been made in the Solaris installation and upgrade process - even more in OpenSolaris (available in the various community releases). As we examined the cloning feature introduced in Solaris 10 11/06, it became apparent that we have stumbled upon a most intriguing capability. When combining zone cloning with the attach/detach capability we have discovered a model for flashing zones: zoneflash.

In a recent boot camp we took a look at this in more detail. Unfortunately the slides (which will be posted soon) didn't quite follow the level of depth we were exploring. Several people asked for notes on how this works - and here they are. The irony is that it will take longer to read about it than it does to perform the actual process - but it is so cool.

The Promise

We start with a fresh Solaris system. In this case just live upgraded from media, but it could have been jumpstarted from media or a flash archive. The key point here is that the system has had very little done to it, other than naming and some software installation. Since zone attach makes sure that key system components (specifically packages and patches) are compatible, it makes sense to build our flashzones on a system that will look similar to those that will be built in the future.

So how many zones will we build ? That's a good question. If this were system flasharchives the answer would be as few as possible - one per architecture in the most efficient case. But these zoneflashes are different - just applications, some metadata, and perhaps some customizations (naming, security, SMF). It seems reasonable to create one zoneflash for each type of application server you would deploy - think of it as a userspace template. In this example I have chosen four: a blank uncustomized flash (for building a new zoneflash in a flash), database server (MySQL), web server (apache2), and the community edition of webmin (just another application).

Our procedure will be to build a minimal default zoneflash, run it through first boot to populate the SMF repository, and then clone it for the remaining zoneflashes. Each of these will be booted, customized for the particular application, and tested to make sure everything is operating properly.

We will then detach the zones and move the detached zoneroots onto some media that can be transported. Of course, keeping with the theme of zones and flash, the transport could be the flasharchive itself. How cool would it be to jumpstart a server using flasharchives and have all the application zones already present in a known location, such as /zoneflash ? Unfortunately, I'm sitting in seat 18A on an American Airlines flight to Los Angeles and don't quite have the required infrastructure to do that sort of test. But I do have a USB stick and multiple boot environments. That will do nicely.

Once attached, we will clone the zoneflashes as necessary, adding resources (network, local filesystems) and attributes (resource controls) required for the proper operation of the application. When finished we will detach the zoneflashes so they may be used elsewhere.

The Turn

The first step is to build and boot a simple generic sparse root zone. Since this zone isn't really meant for operation, most zonecfg attributes (network configuration, resource limits, et al) will be skipped. We will add them later when we build the real zones - remember, these are just user space application templates.

# zonecfg -z flashdefault
flashdefault: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:flashdefault> create
zonecfg:flashdefault> set zonepath=/local/default
zonecfg:flashdefault> add inherit-pkg-dir; set dir=/opt; end
zonecfg:flashdefault> commit
zonecfg:flashdefault> exit
# zoneadm -z default install
A few minutes later we have an installed zone, ready for first boot. Since I've attended my Solaris Zones Best Practices class, or at least read the materials, I know how to build a sysidcfg file that will satisfy the sysidtool first boot service. This will allow the zone to boot up all the way without any additional console interaction. Let's do that for our new zone.
# echo > /local/default/root/etc/sysidcfg <<EOF
root_password=xxxxxxxxxx        You supply your own encrypted string from /etc/shadow, I'm not going to post mine!
network_interface=NONE {hostname=default}
# zoneadm -z default boot
# zlogin -C default 
We need to let first boot processing complete. Since we supplied a valid sysidcfg, it is just a matter of waiting for manifest-import and sysidtool to complete their magic. When complete, login in and take a look around to make sure all is well. Once satisfied, shut down the zone (either from inside the zone or from the global zone) - we are through with it for now.
(from the global zone)
# zoneadm -z default halt
Now we are done with this first zone. Time to clone it for our remaining application zones. Please pardon a bit of inline shell scripting - I hate to type the same thing over and over and over. Sort of makes for a nice script template, doesn't it ? Not quite the sophistication of Brad Digg's zonemanager, but it will do nicely for our example.

# for zone in webmin mysql web
? do
        echo "create -t default; set zonepath=/local/${zone}" | zonecfg -z ${zone}
        zoneadm -z ${zone} clone default
        echo "name_service=NONE" > /local/${zone}/root/etc/sysidcfg
        echo "nfs4_domain=dynamic" >> /local/${zone}/root/etc/sysidcfg
        echo "security_policy=NONE" >> /local/${zone}/root/etc/sysidcfg
        echo "root_password=xxxxxxxxxxx" >> /local/${zone}/root/etc/sysidcfg
        echo "system_locale=C" >> /local/${zone}/root/etc/sysidcfg
        echo "network_interface=NONE {hostname=${zone}}" >> /local/${zone}/root/etc/sysidcfg
        echo "terminal=ansi" >> /local/${zone}/root/etc/sysidcfg
        echo "timezone=US/Central" >> /local/${zone}/root/etc/sysidcfg
        zoneadm -z ${zone} boot
What in the heck was that all about ? OK, one more time - line by line with annotation.

# for zone in webmin mysql web

A quick interactive loop for the creation of three application zones. The variable ${zone} will be set to the name of the zone we are trying to construct.
echo "create -t default; set zonepath=/local/${zone}" | zonecfg -z ${zone}
A one liner that creates a new zone configuration based on the already existing default. At this point the only thing we need to change is the zonepath, and it should be set to /local/${zone}.
        zoneadm -z ${zone} clone default
We recognize this as a zone cloning operation. The zone root is copied and a /reconfigure is created in the new zone root so that sysidtool performs a complete configuration on first boot. If you happen to be running on a recent release of OpenSolaris, you can put your zoneroot on ZFS and the cloning operating will only take a few seconds and very little additional disk space will be required. Those of us on Solaris 10 11/06 will have to wait for the 160MB or so to be copied. Still better than the 9 minutes to go through a complete zone installation.
        echo "name_service=NONE" > /local/${zone}/root/etc/sysidcfg
        echo "nfs4_domain=dynamic" >> /local/${zone}/root/etc/sysidcfg
        echo "security_policy=NONE" >> /local/${zone}/root/etc/sysidcfg
        echo "root_password=xxxxxxxxxxx" >> /local/${zone}/root/etc/sysidcfg
        echo "system_locale=C" >> /local/${zone}/root/etc/sysidcfg
        echo "network_interface=NONE {hostname=${zone}}" >> /local/${zone}/root/etc/sysidcfg
        echo "terminal=ansi" >> /local/${zone}/root/etc/sysidcfg
        echo "timezone=US/Central" >> /local/${zone}/root/etc/sysidcfg
This step creates a custom sysidcfg file for each zone. Remember to supply your own root password from /etc/shadow in the global zone. This answers all of the sysidtool questions, including the NFSv4 question.
	zoneadm -z {zone} boot
Boot the zone. If we have done everything correctly, the next interaction will be with console login.

Close the for loop in the interactive script. This process will take a few minutes on Solaris 10 11/06, or if we are being clever with OpenSolaris and ZFS - a few seconds.

Now for the hard part - customizing the individual application zones. Well, it's not all that difficult. And if you do this regularly, you probably have scripts to do most of the work. It's just individual application installation and customization.

Here is what I did for my example zones.
The installation instructions for the Solaris 10 MySQL can be found in /etc/sfw/mysql/README.solaris.mysql. There is a typo in the Solaris 10 version of the README. It will cause a lot of grief if you cut and paste without looking at the results. Fortunately it has been corrected in nevada (aka OpenSolaris Community Edition).

Boot the mysql zone and log in as root.
# /usr/sfw/bin/mysql_install_db
# groupadd mysql
# useradd -g mysql mysql
# chgrp -R mysql /var/mysql
# chmod -R 770 /var/mysql       This line is incorrect in the Solaris 10 README - my chmod works better with two arguments
# installf SUNWmysqlr /var/mysql d 770 root mysql
# cp /usr/sfw/share/mysql/my-medium.cnf /var/mysql/my.cnf
The installation instructions continue by linking the start script into /etc/rc3.d. Since we are big SMF fans in these parts, let's do that instead. Feel free to use my MySQL manifest as it contains a couple of cool features (value and action authorizations - more on that later).

Since the mysql zone doesn't have any networking configured, perform this next step from the global zone. If you already have a suitable manifest, or have stashed mine away somewhere in the global zone you can use that instead.
# cd /local/mysql/root/var/svc/manifest/application
# wget
It's probably a good idea to make sure that all of this is working properly. Either reboot the mysql zone, run the manifest-import service manually, or run svccfg import on the new manifest. Your choice. What you should see upon completion is
# svcs mysql
STATE          STIME    FMRI
online         14:41:19 svc:/application/mysql:default

# /usr/sfw/bin/mysqladmin status
Uptime: 459  Threads: 1  Questions: 2  Slow queries: 0  Opens: 6  Flush tables: 1  Open tables: 0  Queries per second avg: 0.004

We're done for now. Unless of course you want to go for some extra credit. In that case
  1. Set up a web server with PHP support. Apache 1 plus the SFWmphp package from the Solaris Companion will do just fine.
  2. Download and unpack phpMyAdmin in the webserver htdocs directory.
  3. Create a user with the mysql.operator authorization
  4. Create a user with the mysql.administrator authorization

Shut down the mysql zone.
This is about as easy as it gets. Boot the web zone and perform the following steps.
# cp /etc/apache2/httpd.conf-example /etc/apache2/httpd.conf
# svcadm enable apache2

A quick check to make sure all is well.
# svcs apache2
STATE          STIME    FMRI
online         17:17:41 svc:/network/http:apache2

# telnet localhost 80
Trying ::1...
telnet: connect to address ::1: Network is unreachable
Connected to localhost.
Escape character is '\^]'.
<title>501 Method Not Implemented</title>

Connection to localhost closed by foreign host.
We're done for now. Shut down the web zone.
This one is a little more complicated. We did this one last time in the zone cloning, but it is worth a second look.

Our task here is to replace the Solaris webmin with the latest download from The technique we are using will allow us to install a custom version of an application into a sparse root zone. Specifically,'s package installs into /opt/webmin, but /opt is a read-only inherited-pkg-dir. The easiest solution for this would be the creation of a symbolic link in the global zone /opt to point to a location that can be safely written by each non-global zone. In my example that would be /local-pkgs.

In the global zone, create the link in /opt, create the local package directory in the webmin zoneroot, and download the latest webmin package.
# ln -s ../local-pkgs/webmin /opt/webmin
# mkdir -p /local/webmin/root/local-pkgs/webmin
# cd /local/webmin/root/var/tmp
# wget
# gunzip webmin-1.330.pkg.gz

Now boot the webmin zone and log in as root.
# zoneadm -z webmin boot
# zlogin webmin
Remove the Solaris webmin packages (SUNWwebminu SUNWwebminr). The usr package needs to be removed twice - the first pkgrm will leave it as a partially installed package, the second will completely remove it - at least as far as our zone (and future patching) is concerned. Once removed, install the version, which should be conveniently located in /var/tmp.
# pkgrm SUNWwebminu SUNWwebminr SUNWwebminu
# pkgadd -d /var/tmp/webmin-1.330.pkg
We are done with this zone. Shut it down.
We have just built four zones: an empty zone suitable for future customizations, one with the Solaris webmin replaced by the community edition, one with a working MySQL database, and one with a webserver. The last task to be performed on these zones in their current state is to be detached, another new feature in Solaris 10 11/06. Zone detach will copy the zone configuration into the zoneroot (to be used with a subsequent zone attach) and sets the current zone state to configured. You can even delete the zone configurations as a final cleanup prior to building a flash archive.
# zoneadm -z default detach
# zoneadm -z webmin detach
# zoneadm -z mysql detach
# zoneadm -z web detach
# zonecfg -z default delete -F
# zonecfg -z webmin delete -F
# zonecfg -z mysql delete -F
# zonecfg -z web delete -F

And flash
Unless the person in 18B wants to be a jumpstart server, we will have to simulate jumpstart/flasharchive process. We can do this by booting into an alternate boot environment and then delivering the detached zoneroots by some sort of shared or removable storage - something like a USB memory stick. When we are done with this exercise, our zoneflashes will still be on the memory device, ready for their next use. Since the zones will never be booted, just cloned, the speed of the memory device really isn't important.

We need to prepare the USB memory stick (currently formatted as FAT16). We will use rmformat -l to locate the device, fdisk to put a proper label on it, finally newfs for installing a proper file system. ZFS would be interesting, but it would just get in our way later.
# rmformat -l
Looking for devices...
     1. Logical Node: /dev/rdsk/c2t0d0p0
        Physical Node: /pci@0,0/pci1179,1@1d,7/storage@4/disk@0,0
        Connected Device:          USB DISK 2.0     PMAP
        Device Type: Removable
        Bus: USB
        Size: 984.0 MB
        Access permissions: 
     2. Logical Node: /dev/rdsk/c1t0d0p0
        Physical Node: /pci@0,0/pci-ide@1f,1/ide@1/sd@0,0
        Connected Device: TEAC     DW-224E-A        7.2A
        Device Type: CD Reader
        Bus: IDE
        Access permissions: 

# fdisk /dev/rdsk/c2t0d0p0
3 (to delete the existing partition)
1 (to create a new Solaris partition)
5 (to exit and write the new label)

# newfs /dev/rdsk/c2t0d0s2
newfs: construct a new file system /dev/rdsk/c2t0d0s2: (y/n)? y
/dev/rdsk/c2t0d0s2:     2009088 sectors in 981 cylinders of 64 tracks, 32 sectors
        981.0MB in 62 cyl groups (16 c/g, 16.00MB/g, 7680 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 32832, 65632, 98432, 131232, 164032, 196832, 229632, 262432, 295232,
 1705632, 1738432, 1771232, 1804032, 1836832, 1869632, 1902432, 1935232,
 1968032, 2000832
# mkdir /tmp/flash
# mount /dev/dsk/c2t0d0s2 /tmp/flash
# cd /local
# find default webmin web mysql -print | cpio -pdum /tmp/flash
# umount /tmp/flash
We are now done with the original system. At this point we would create a flasharchive (with the detached zoneroots in a convenient place in the archive).

The Prestige

The final act in our magic trick is the delivery. Specifically the transport, reattachment, and subsequent cloning of the zoneflashes on a new system. 18B is now asleep and I really don't want to disturb him, so I'll do this part myself. I'll boot my laptop into another boot environment - built from the same media using the same Live Upgrade method as the boot environment that created the zones.

We begin by mounting the removable media (USB memory stick) that contains the zoneflash. Do take a look around, it is quite likely that our friend volfs has already done this for us. Remember - if we were using a flasharchive to deliver the zoneflash this step would be unnecessary.
# mkdir /flash
# mount /dev/dsk/c2t0d0s2 /flash        (we used rmformat -l to derive the device name)
Now that our zoneflashes have arrived, time to reattach them. The first step is to create zone configurations. If you recall, these were stored in the zoneroot when they were detached. The zonecfg command create -a is used to retrieve the stored configuration information and adapt it to the new system - specifically the new location of the zoneroot. Once configured we use zoneadm attach to reconnect them.

The sequence to reattach our default zone, now called flashdefault, would look something like this.
# zonecfg -z flashdefault
flashdefault: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:flashdefault> create -a /flash/default
zonecfg:flashdefault> commit
zonecfg:flashdefault> exit
# zoneadm -z flashdefault attach
We'll be a little more clever attaching the other three zones.
# for zone in webmin web mysql
      echo "create -a /flash${zone}" | zonecfg -z flash${zone}
      zoneadm -z flash${zone} attach
At this point our zoneroots are still on the USB memory device - but don't worry, these zones will never be booted. Their only purpose is to deliver preconfigured zones. We will use zone cloning to create our real application zones.

Which we will now do. It is very convenient to use the flashzone as a template for our new zone in case there were some special attributes like limitpriv that we might want to preserve. We will also need to add items that were not present in the zoneflashes - specifically networking and local file systems. Once we are satisfied with the zone configurations we will clone the zoneflash. If we are only building one of each type of zone we can detach the zoneflash so that other administrators can use it on their systems.

Let's do this for the mysql zone.
# zonecfg -z mysql
mysql: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:mysql> create -t flashmysql
zonecfg:mysql> set zonepath=/zones/mysql
zonecfg:mysql> add net; set physical=e1000g0; set address=; end
zonecfg:mysql> add fs; set dir=/export; set special=/export; set options=[rw,nosuid,nodevices]; set type=lofs; end
zonecfg:mysql> commit
zonecfg:mysql> exit

# zoneadm -z mysql clone flashmysql
Copying /flash/mysql...

# zoneadm -z flashmysql detach

# echo "name_service=NONE" >    /zones/mysql/root/etc/sysidcfg
# echo "nfs4_domain=dynamic" >> /zones/mysql/root/etc/sysidcfg
# echo "security_policy=NONE" >> /zones/mysql/root/etc/sysidcfg
# echo "root_password=xxxxxxxxxxx" >> /zones/mysql/root/etc/sysidcfg
# echo "system_locale=C" >> /zones/mysql/root/etc/sysidcfg
# echo "network_interface=NONE {hostname=mysql}" >> /zones/mysql/root/etc/sysidcfg
# echo "terminal=ansi" >> /zones/mysql/root/etc/sysidcfg
# echo "timezone=US/Central" >> /zones/mysql/root/etc/sysidcfg

And for the finale - boot the newly flashed mysql zone and you should see an enabled and operating mysql service.
# zoneadm -z mysql boot
# zlogin -C mysql
[Connected to zone 'mysql' console]
Hostname: mysql
Creating new rsa public/private host key pair                           
Creating new dsa public/private host key pair
Mar 20 06:15:44 mysql sendmail[1719]: My unqualified host name (mysql) unknown; sleeping for retry
Mar 20 06:15:44 mysql sendmail[1722]: My unqualified host name (mysql) unknown; sleeping for retry

mysql console login: root
Last login: Mon Mar 19 17:10:10 on console
Mar 20 06:15:49 mysql login: ROOT LOGIN /dev/console
Sun Microsystems Inc.   SunOS 5.11      snv_57  October 2007
# svcs mysql
STATE          STIME    FMRI
online          6:31:28 svc:/application/mysql:default
# /usr/sfw/bin/mysqladmin status
Uptime: 8  Threads: 1  Questions: 1  Slow queries: 0  Opens: 6  Flush tables: 1  Open tables: 0  Queries per second avg: 0.125

How cool is that ? Not only did we clone the zone, but since the database is in /var, it was cloned as well. Perhaps not practical for every situation, but still pretty cool.

I will leave the flashing of default, web, and webmin as an exercise to the reader. Follow the sequence we used for the mysql zone and you should have four working zones, built from a flash like mechanism that can be delivered via removable media, flasharchive, or shared storage.

Next time we'll take a closer look at MySQL and explore running it as a less privileged user. We'll also look at the action and value authorizations.

Technocrati Tags:


Great article! One question for you: does this approach serve as a workaround for the current Solaris 10 limitation against flash archiving a multi-zoned server?

We're exploring flash archiving as a disaster recovery mechanism, and it works like a charm for our servers that don't host non-global zones. But we really would like to use the flash archive/restore approach (with minimal manual steps) with our zoned servers as well.

If I'm interpreting your comments correctly, to back up an existing multi-zoned server we should do the following:

- Make sure the apps on each zone are configured as desired and all required services are enabled
- Stop each zone
- Detach each zone via zoneadm -z {zonename} detach
- Delete the configuration for each zone via zoneadm -z {zonename} delete -F
- Create a flash archive of the server, which will automatically capture the zone images that were written by the detach step in a well-known directory

And restoration would involve
- Installation of the flash archive
- Reattaching each zone
- Boot up each zone

Can you verify whether any steps are missing or incorrect?

I've seen some blogs on OpenSolaris that imply that you can flash archive a multi-zone box if certain key hoops are jumped through, but we're running the commercial version of the OS.



Posted by Jon Giron on August 13, 2009 at 10:40 AM CDT #

Post a Comment:
  • HTML Syntax: NOT allowed

Bob Netherton is a Principal Sales Consultant for the North American Commercial Hardware group, specializing in Solaris, Virtualization and Engineered Systems. Bob is also a contributing author of Solaris 10 Virtualization Essentials.

This blog will contain information about all three, but primarily focused on topics for Solaris system administrators.

Please follow me on Twitter Facebook or send me email


« July 2016