X

Oracle Solaris: Zones on Shared Storage

By: Jeff Victor | Principal Systems Engineer


Oracle
Solaris 11.1
has several new features. At oracle.com you can find
a detailed list.


One of the significant new features, and the most significant new feature releated
to Oracle Solaris Zones, is casually called "Zones on Shared Storage" or simply ZOSS (rhymes
with "moss"). ZOSS offers much more flexibility because you can store Solaris Zones
on shared storage (surprise!) so that you can perform quick and easy migration of a zone
from one system to another. This blog entry describes and demonstrates the use of ZOSS.


ZOSS provides complete support for a Solaris Zone that is stored on "shared storage."
In this case, "shared storage" refers to fiber channel (FC) or iSCSI devices,
although there is one lone exception that I will demonstrate soon. The primary intent
is to enable you to store a zone on FC or iSCSI storage so that it can be migrated
from one host computer to another much more easily and safely than in the past.


With this blog entry, I wanted to make it easy for you to try this yourself. I couldn't
assume that you have a SAN available - which is a good thing, because neither do I! :-)
What could I use, instead? [There he goes,
foreshadowing again... -Ed.]


Developing this entry reinforced the lesson that the solution to every lab problem is
VirtualBox. ;-) Oracle VM VirtualBox
(its formal name) helps here in a couple of important ways. It offers the ability to
easily install multiple copies of Solaris as guests on top of any popular system
(Microsoft Windows, MacOS, Solaris,
Oracle Linux
(and other Linuxes) etc.). It also offers the ability to create a separate
virtual disk drive (VDI) that appears as a local hard disk to a guest. This virtual
disk can be moved very easily from one guest to another. In other words, you can
follow the steps below on a laptop or larger x86 system.


Please note that the ability to use ZOSS to store a zone on a local disk is very useful
for a lab environment, but not so useful for production. I do not suggest
regularly moving disk drives among computers. [Update, 2013.01.28: Apparently the previous sentence
caused some confusion. I do recommend the use of
Zones on Shared Storage in production environments, when appropriate storage is used.
"Appropriate storage" would include SAN or iSCSI at this point. I do not recommend using ZOSS
with local disks in production because doing so would require moving the disks between
computers.]


In the method I describe below, that virtual hard disk will contain the zone that will be
migrated among the (virtual) hosts. In production, you would use FC or iSCSI LUNs
instead. The
zonecfg(1M) man page details the syntax for each of the three types of
devices.


Why Migrate?


Why is the migration of virtual servers important? Some of the most common reasons
are:

  • Moving a workload to a different computer so that the original computer can be
    turned off for extensive maintenance.
  • Moving a workload to a larger system because the workload has outgrown its original system.
  • If the workload runs in an environment (such as a Solaris Zone)
    that is stored on shared storage, you can restore the service of the workload on an
    alternate computer if the original computer has failed and will not reboot.
  • You can simplify lifecycle management of a workload by developing it on a laptop,
    migrating it to a test platform when it's ready, and finally moving it to a production
    system.

Concepts



For ZOSS, the important new concept is named "rootzpool". You can read about it
in the
zonecfg(1M) man page, but here's the short version: it's the backing store
(hard disk(s), or LUN(s)) that will be used to make a ZFS zpool - the zpool that will
hold the zone. This zpool:


  • contains the zone's Solaris content, i.e. the root file system
  • does not contain any content not related to the zone
  • can only be mounted by one Solaris instance at a time

Method Overview


Here is a brief list of the steps to create a zone on shared storage and migrate it.
The next section shows the commands and output.

  1. You will need a host system with an x86 CPU (hopefully at least a couple of CPU cores), at least 2GB of RAM, and at least 25GB of free disk space. (The steps below will not actually use 25GB of disk space, but I don't want to lead you down a path that ends in a big sign that says "Your HDD is full. Good luck!")
  2. Configure the zone on both systems, specifying the rootzpool that both will use.
    The best way is to configure it on one system and then copy the output of "zonecfg export"
    to the other system to be used as input to zonecfg. This method reduces the chances of
    pilot error.
    (It is not necessary to configure the zone on both systems before creating it. You
    can configure this zone in multiple places, whenever you want, and migrate it to one
    of those places at any time - as long as those systems all have access to the shared storage.)
  3. Install the zone on one system, onto shared storage.
  4. Boot the zone.
  5. Provide system configuration information to the zone. (In the Real World(tm)
    you will usually automate this step.)
  6. Shutdown the zone.
  7. Detach the zone from the original system.
  8. Attach the zone to its new "home" system.
  9. Boot the zone.

The zone can be used normally, and even migrated back, or to a different system.

Details


The rest of this shows the commands and output. The two hostnames are "sysA" and
"sysB".


Note that each Solaris guest might use a different device name for the VDI that they
share. I used the device names shown below, but you must discover the device name(s)
after booting each guest. In a production environment you would also discover the device
name first and then configure the zone with that name. Fortunately, you can use the command
"zpool import" or "format" to discover the device on the "new" host for the zone.


The first steps create the VirtualBox guests and the shared disk drive. I describe the steps here without demonstrating them.


  1. Download VirtualBox and
    install it using a method normal
    for your host OS
    . You can read the
    complete instructions.
  2. Create two VirtualBox guests,
    each to run Solaris 11.1. Each will use its own VDI as its root disk.
  3. Install Solaris 11.1 in each guest.Install Solaris 11.1 in each guest. To install a Solaris 11.1 guest, you can either
    download
    a pre-built VirtualBox guest
    , and import it,
    or install Solaris 11.1 from the
    "text install" media.
    If you use the latter method, after booting you will not see a windowing system. To
    install the GUI and other important things, login and run "pkg install solaris-desktop"
    and take a break while it installs those important things.
  4. Life is usually easier if you install the VirtualBox Guest Additions
    because then you can copy and paste between the host and guests, etc. You can find the
    guest additions in the folder
    matching the version of VirtualBox you are using
    . You can also read the
    instructions for installing
    the guest additions.

  5. To create the zone's shared VDI in VirtualBox, you can open the storage configuration for
    one of the two guests, select the SATA controller, and click on the "Add Hard Disk" icon
    nearby. Choose "Create New Disk" and specify an appropriate path name for the file that
    will contain the VDI. The shared VDI must be at least 1.5 GB. Note that the guest must be
    stopped to do this.
  6. Add that VDI to the other guest - using its Storage configuration - so that each
    can access it while running. The steps start out the same, except that you choose "Choose Existing Disk" instead of "Create New Disk."
    Because the disk is configured on both of them,
    VirtualBox prevents you from running both guests at the same time.
  7. Identify device names of that VDI, in each of the guests. Solaris chooses the name based on existing devices. The names may be the same, or may be different from each other. This step is shown below as "Step 1."

Assumptions


In the example shown below, I make these assumptions.

  • The guest that will own the zone at the beginning is named sysA.
  • The guest that will own the zone after the first migration is named sysB.
  • On sysA, the shared disk is named /dev/dsk/c7t2d0
  • On sysB, the shared disk is named /dev/dsk/c7t3d0

(Finally!) The Steps


Step 1) Determine the name of the disk that will move back and forth between the systems.

root@sysA:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c7t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c7t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
Specify disk (enter its number): ^D

Step 2) The first thing to do is partition and label the disk. The magic needed to write an EFI
label is not overly complicated.

root@sysA:~# format -e c7t2d0
selecting c7t2d0
[disk formatted]
FORMAT MENU:
...
format> fdisk
No fdisk table exists. The default partition for the disk is:
a 100% "SOLARIS System" partition
Type "y" to accept the default partition, otherwise type "n" to edit the
partition table. n
SELECT ONE OF THE FOLLOWING:
...
Enter Selection: 1
...
G=EFI_SYS 0=Exit? f
SELECT ONE...
...
6
format> label
...
Specify Label type[1]: 1
Ready to label disk, continue? y
format> quit
root@sysA:~# ls /dev/dsk/c7t2d0
/dev/dsk/c7t2d0

Step 3) Configure zone1 on sysA.

root@sysA:~# zonecfg -z zone1
Use 'create' to begin configuring a new zone.
zonecfg:zone1> create
create: Using system default template 'SYSdefault'
zonecfg:zone1> set zonename=zone1
zonecfg:zone1> set zonepath=/zones/zone1
zonecfg:zone1> add rootzpool
zonecfg:zone1:rootzpool> add storage dev:dsk/c7t2d0
zonecfg:zone1:rootzpool> end
zonecfg:zone1> exit
root@sysA:~#
oot@sysA:~# zonecfg -z zone1 info
zonename: zone1
zonepath: /zones/zone1
brand: solaris
autoboot: false
bootargs:
file-mac-profile:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
anet:
...
rootzpool:
storage: dev:dsk/c7t2d0

Step 4) Install the zone. This step takes the most time, but you can wander
off for a snack or a few laps around the gym - or both! (Just not at the same time...)

root@sysA:~# zoneadm -z zone1 install
Created zone zpool: zone1_rpool
Progress being logged to /var/log/zones/zoneadm.20121022T163634Z.zone1.install
Image: Preparing at /zones/zone1/root.
AI Manifest: /tmp/manifest.xml.RXaycg
SC Profile: /usr/share/auto_install/sc_profiles/enable_sci.xml
Zonename: zone1
Installation: Starting ...
Creating IPS image
Startup linked: 1/1 done
Installing packages from:
solaris
origin: http://pkg.us.oracle.com/support/
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 183/183 33556/33556 222.2/222.2 2.8M/s
PHASE ITEMS
Installing new actions 46825/46825
Updating package state database Done
Updating image state Done
Creating fast lookup database Done
Installation: Succeeded
Note: Man pages can be obtained by installing pkg:/system/manual
done.
Done: Installation completed in 1696.847 seconds.
Next Steps: Boot the zone, then log into the zone console (zlogin -C)
to complete the configuration process.
Log saved in non-global zone as /zones/zone1/root/var/log/zones/zoneadm.20121022T163634Z.zone1.install

Step 5) Boot the Zone.

root@sysA:~# zoneadm -z zone1 boot

Step 6) Login to zone's console to complete the specification of system information.

root@sysA:~# zlogin -C zone1

Answer the usual questions and wait for a login prompt. Then you can end the
console session with the usual "~." incantation.


Step 7) Shutdown the zone so it can be "moved."


root@sysA:~# zoneadm -z zone1 shutdown

Step 8) Detach the zone so that the original global zone can't use it.

root@sysA:~# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / solaris shared
- zone1 installed /zones/zone1 solaris excl
root@sysA:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 17.6G 11.2G 6.47G 63% 1.00x ONLINE -
zone1_rpool 1.98G 484M 1.51G 23% 1.00x ONLINE -
root@sysA:~# zoneadm -z zone1 detach
Exported zone zpool: zone1_rpool

Step 9) Review the result and shutdown sysA so that sysB can use the shared disk.

root@sysA:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 17.6G 11.2G 6.47G 63% 1.00x ONLINE -
root@sysA:~# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / solaris shared
- zone1 configured /zones/zone1 solaris excl
root@sysA:~# init 0

Step 10) Now boot sysB and configure a zone with the parameters
shown above in Step 1. (Again, the safest method is to use "zonecfg ... export"
on sysA as described in section "Method Overview" above.) The one difference
is the name of the rootzpool storage device, which was shown in the list of
assumptions, and which you must determine by booting sysB and using the "format"
or "zpool import" command.

When that is done, you should see the output shown next. (I used the same
zonename - "zone1" - in this example, but you can choose any valid zonename you want.)


root@sysB:~# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / solaris shared
- zone1 configured /zones/zone1 solaris excl
root@sysB:~# zonecfg -z zone1 info
zonename: zone1
zonepath: /zones/zone1
brand: solaris
autoboot: false
bootargs:
file-mac-profile:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
anet:
linkname: net0
...
rootzpool:
storage: dev:dsk/c7t3d0

Step 11) Attaching the zone automatically imports the zpool.

root@sysB:~# zoneadm -z zone1 attach
Imported zone zpool: zone1_rpool
Progress being logged to /var/log/zones/zoneadm.20121022T184034Z.zone1.attach
Installing: Using existing zone boot environment
Zone BE root dataset: zone1_rpool/rpool/ROOT/solaris
Cache: Using /var/pkg/publisher.
Updating non-global zone: Linking to image /.
Processing linked: 1/1 done
Updating non-global zone: Auditing packages.
No updates necessary for this image.
Updating non-global zone: Zone updated.
Result: Attach Succeeded.
Log saved in non-global zone as /zones/zone1/root/var/log/zones/zoneadm.20121022T184034Z.zone1.attach
root@sysB:~# zoneadm -z zone1 boot
root@sysB:~# zlogin zone1
[Connected to zone 'zone1' pts/2]
Oracle Corporation SunOS 5.11 11.1 September 2012

Step 12) Now let's migrate the zone back to sysA. Create a file in zone1 so we can verify it exists after we migrate the zone back, then begin migrating it back.

root@zone1:~# ls /opt
root@zone1:~# touch /opt/fileA
root@zone1:~# ls -l /opt/fileA
-rw-r--r-- 1 root root 0 Oct 22 14:47 /opt/fileA
root@zone1:~# exit
logout
[Connection to zone 'zone1' pts/2 closed]
root@sysB:~# zoneadm -z zone1 shutdown
root@sysB:~# zoneadm -z zone1 detach
Exported zone zpool: zone1_rpool
root@sysB:~# init 0

Step 13) Back on sysA, check the status.

Oracle Corporation SunOS 5.11 11.1 September 2012
root@sysA:~# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / solaris shared
- zone1 configured /zones/zone1 solaris excl
root@sysA:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 17.6G 11.2G 6.47G 63% 1.00x ONLINE -

Step 14) Re-attach the zone back to sysA.

root@sysA:~# zoneadm -z zone1 attach
Imported zone zpool: zone1_rpool
Progress being logged to /var/log/zones/zoneadm.20121022T190441Z.zone1.attach
Installing: Using existing zone boot environment
Zone BE root dataset: zone1_rpool/rpool/ROOT/solaris
Cache: Using /var/pkg/publisher.
Updating non-global zone: Linking to image /.
Processing linked: 1/1 done
Updating non-global zone: Auditing packages.
No updates necessary for this image.
Updating non-global zone: Zone updated.
Result: Attach Succeeded.
Log saved in non-global zone as /zones/zone1/root/var/log/zones/zoneadm.20121022T190441Z.zone1.attach
root@sysA:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 17.6G 11.2G 6.47G 63% 1.00x ONLINE -
zone1_rpool 1.98G 491M 1.51G 24% 1.00x ONLINE -
root@sysA:~# zoneadm -z zone1 boot
root@sysA:~# zlogin zone1
[Connected to zone 'zone1' pts/2]
Oracle Corporation SunOS 5.11 11.1 September 2012
root@zone1:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 1.98G 538M 1.46G 26% 1.00x ONLINE -

Step 15) Check for the file created on sysB, earlier.

root@zone1:~# ls -l /opt
total 1
-rw-r--r-- 1 root root 0 Oct 22 14:47 fileA

Next Steps


Here is a brief list of some of the fun things you can try next.

  • Add space to the zone by adding a second storage device to the rootzpool. Make sure that you add it to the configurations of both zones!
  • Create a new zone, specifying two disks in the rootzpool when you first configure
    the zone. When you install that zone, or
    clone it
    from another zone, zoneadm uses those two disks to create a mirrored pool.
    (Three disks will result in a three-way mirror, etc.)

Conclusion


Hopefully you have seen the ease with which you can now move Solaris Zones from one system to another.



Join the discussion

Comments ( 7 )
  • Pavel Anni Thursday, November 15, 2012

    Jeff, great lab! Thanks!

    As an additional fun thing to try I would install and enable an application inside sysA (e.g. apache-22) and check if it's enabled in sysB after migration. I think I will do it right now! :-)


  • Pavel Anni Friday, November 16, 2012

    It works! I mean: installing and enabling Apache inside zone1 in sysA and then attaching that zone in sysB and Apache is up and running! And it's fast--I mean, the detach/attach process. Just a couple of seconds, even on VirtualBox! Really cool!


  • Pavel Anni Friday, November 16, 2012

    Needless to say: when I copied the VirtualBox file (a disk in Solaris terms) from one laptop to another and attached that zone to my Solaris 11.1 installation in another VirtualBox, it worked perfectly and Apache was up and running.


  • guest Monday, November 19, 2012

    This is not a huge improvement over what I can achieve in solaris 10, I could configure zones on FC/iscsi zpool and just shutdown/detach zones, zpool export and then import on sysB, then create/attach the zones in the imported zpool. Your steps just make above procedure a little simpler.

    However, this is nothing compared to vMotion alike technology, and the fact that you could only do it on zpool level (not zfs level) also limited the usage. The ZFS is lacking the shared FS (compare to VMFS)features, as long as this remains, IMHO the live migration (even the one for Ldom or Oracle VM for Sparc) for solaris zone/ldom has little usefulness.


  • JeffV Monday, November 19, 2012

    Hi guest! ZOSS was introduced in Solaris 11.1, which is just an update to Solaris 11.0. One should not expect huge changes in an update. ZOSS is an incremental improvement.

    ZOSS may not seem to be a big improvement in functionality, but it is a significant reduction in risk. Before ZOSS (unless you use OEM Ops Center) if you installed a zone on shared storage, nothing would prevent you from booting the zone on more than one computer (or trying!). ZOSS prevents that problem - and the resulting mess - because ZFS prevents you from importing a ZFS zpool onto multiple computers concurrently.

    Ops Center achieves the same goal by checking to see if the zone is already running. This works very well as long as you only use Ops Center to boot zones.

    Regarding the use of a zpool vs. a file system: ZOSS is safe because you cannot import a zpool on multiple systems, as I mentioned above. Also, because ZOSS is intended to use LUs (via iSCSI or FC) there is no disadvantage to using a zpool compared to a file system. You can always create more zpools, as long as you can create more LUs on your storage system.

    I hope the additional explanation was helpful. Thanks for reading!


  • JohanK Tuesday, November 27, 2012

    Hi, Jeff!

    Thanks for the info! I´m a storage guy, so apart from other normal people, I've Fc SAN equipment at home....

    I'd like to hear with you, or any other reading this blog, about the possibility of using NPIV with zones? Which means, using a virtual HBA ww port name for providing LUN's for the zone.

    I guess you can create an NPIV in a zone, but you also need to move that NPIV when the zone migrates...

    Perhaps you need to create more then one NPIV, one for each system the zone is supposed to migrate to...?

    Anyone can shed some light here...?

    Rgrds Johan


  • guest Thursday, November 29, 2012

    One disadvantage to having to use a separate zpool per zone is that you don't get any of the advantages of deduplication between zones. dedup only happens within a pool.

    How well does this work with live upgrade?


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha