News, tips, partners, and perspectives for the Oracle Solaris operating system

  • August 28, 2018

Zones Delegated Restarter and SMF Goals

Jan Pechanec
Senior Principal Software Engineer

Managing Zones in Oracle Solaris 11.3

In Oracle Solaris 11.3, Zones are managed by the Zones service svc:/system/zones:default.

The service performs the autobooting and shutdown of Zones on system boot and shutdown according to each zone's configuration. The service boots, in parallel, all Zones configured with autoboot=true, and shuts down, also in parallel, all running Zones with their corresponding autoshutdown option: halt, shutdown, or suspend. See the zonecfg(1M) manual page in 11.3 for more information.

The management mechanism existing in 11.3 is sufficient for systems running a small number of Zones. However, the growing size of systems and the increasing number of Zones on them require a more sophisticated mechanism.

The issue is that the following features are missing:

  • Very limited integration with SMF. That also means zones may not depend on other services and vice versa.
  • No threshold tunable for the number of Zones booted in parallel.
  • No integration with FMA.
  • No mechanism to prioritize Zones booting order.
  • No mechanism for providing information when a zone is considered up and running.

This blog post describes enhancements brought in by 11.4 that address existing shortcomings of the Zones service in 11.3.

Zones Delegated Restarter and Goals in Oracle Solaris 11.4

To solve the shortcomings outlined in the previous section, Oracle Solaris 11.4 brings the Zones Delegated Restarter (ZDR) to manage the Zones infrastructure and autobooting, and SMF Goals. Each zone aside from the Global Zone is modeled as an SMF instance of the service svc:/system/zones/zone:<zonename> where the name of the instance is the name of the zone.

Note that the Zones configuration was not moved into the SMF repository.

For an explanation of what is an SMF restarter, see the section "Restarters" in the smf(7) manual page.

Zone SMF Instances

The ZDR replaces the existing shell script, /lib/svc/method/svc-zones, in the Zones service method with the restarter daemon, /usr/lib/zones/svc.zones. See the svc.zones(8) manual page for more information. The restarter runs under the existing Zones service svc:/system/zones:default.

A zone SMF instance is created at the zone creation time. The instance is marked as incomplete for zones in the configured state. On zone install, attach, and clone, the zone instance is marked as complete. Conversely, on zone detach or uninstall, the zone instance is marked incomplete. The zone instance is deleted when the zones is deleted via zonecfg(8).

An example on listing the Zones instances:

$ svcs svc:/system/zones/zone
STATE          STIME    FMRI
disabled       12:42:55 svc:/system/zones/zone:tzone1
online         16:29:47 svc:/system/zones/zone:s10

$ zoneadm list -vi
  ID NAME      STATUS      PATH                    BRAND      IP
   0 global    running     /                       solaris    shared
   1 s10       running     /system/zones/s10       solaris10  excl
   - tzone1    installed   /system/zones/tzone1    solaris    excl

On start-up, the ZDR creates a zone SMF instance for any zone (save for the Global Zone) that does not have one but is supposed to. Likewise, if there is a zone SMF instance that does not have a corresponding zone, the restarter will remove the instance.

The ZDR is responsible for setting up the infrastructure necessary for each zone, spawning a zoneadmd(8) daemon for each zone, and restarting the daemon when necessary. There is a running zoneadmd for each zone in a state greater than configured on the system.

ZDR generated messages related to a particular zone are logged to /var/log/zones/<zonename>.messages, which is where zoneadmd logs as well.

Failures during the infrastructure setup for a particular zone will place the zone to the maintenance state. A svcadm clear on the zone instance triggers the ZDR to re-try.

SMF Goal Services

SMF goals provide a mechanism by which a notification can be generated if a zone is incapable of auto booting to a fully online state (i.e. unless an admin intervenes). With their integration into ZDR they can be used to address one of the shortcomings mentioned above, that we had no mechanism for providing information when a zone is considered up and running.

A goal service is a service with the general/goal-service=true property setting. Such service enters a maintenance state if its dependencies can never be satisfied.

Goal services in maintenance automatically leave that state once their dependencies are satisfiable. The goal services failure mechanism is entirely encompassed in the SMF dependency graph engine. Any service can be marked as a goal service.

We also introduced a new synthetic milestone modeled as a goal service, svc:/milestone/goals:default. The purpose of this new milestone is to provide a clear, unambiguous, and well-defined point where we consider the system up and running. The dependencies of milestone/goals should be configured to represent the mission critical services for the system. There is one dependency by default:

root@vzl-143:~# svcs -d milestone/goals
STATE          STIME    FMRI
online         18:35:49 svc:/milestone/multi-user-server:default

While goals are ZDR agnostic, they are a fundamental requirement of the ZDR which uses the states of the milestone/goals services to determine the state of each non-global zone.

To change the dependency, use svcadm goals:

# svcadm goals milestone/multi-user-server:default network/http:apache24
# svcs -d milestone/goals
STATE          STIME    FMRI
online         18:35:49 svc:/milestone/multi-user-server:default
online         19:53:32 svc:/network/http:apache24

To reset (clear) the dependency service set to the default, use svcadm goals -c.

Zone SMF Instance State

ZDR is notified of the state of the milestone/goals service of each non-global zone that supports it. The zone instance state of each non-global zone will match the state of its milestone/goals.

Kernel Zones that support the milestone/goals service (i.e. those with 11.4+ installed) use internal auxiliary states to report back to the host. Kernel Zones that do not support milestone/goals are considered online when their state is running and auxiliary state is hotplug-cpu.

Zone SMF instances mapping to solaris10(7) branded Zones will have its state driven by the exit code of the zoneadm command.

If the zone's milestone/goals is absent or disabled, the ZDR will treat the zone as not having support for milestone/goals.

The ZDR can be instructed to ignore milestone/goals for the purpose of moving the zone SMF instance to the online state based only on the success of zoneadm boot -- if zoneadm boot fails the zone SMF instance is placed into maintenance. The switch is controlled by the following SMF property under the ZDR service instance, svc:/system/zones:default:

config/track-zone-goals = true | false

For example, this feature might be useful to providers of VMs to different tenants (IaaS) that do not care about what is running on the VMs but only care whether those VMs are accessible to their tenants.

Zone Dependencies

The set of SMF FMRIs that make up the zone dependencies is defined by a new zonecfg resource, smf-dependency. It is comprised of fmri and grouping properties. All SMF dependencies for a zone will be of type service and have restart_on none -- we do not want zones being shutdown or restarted because of a faulty flip-flopping dependency. Example:

An example on setting dependencies for a zone:

add smf-dependency
        set fmri svc:/application/frobnicate:default
add smf-dependency
        set fmri svc:/system/zones/zone:appfirewall
add smf-dependency
        set fmri svc:/system/zones/zone:dataload
        set grouping=exclude_all

The default for grouping is require_all. See the smf(7) manual page for other dependency grouping types.

Zone Config autoboot Configuration

The zone instance SMF property general/enabled corresponds to the zone configuration property autoboot and is stored in the SMF repository. The existing Zones interfaces:

zonecfg(1M) export
zoneadm(1M) detach -n

stay unmodified and contain autoboot in their output. Also, the RAD interfaces accessing the property do not change from 11.3.

Zones Boot Order

There are two ways to establish zones boot order. One is determined by the SMF dependencies of a zone SMF instance (see above for smf-dependency). The other one is an assigned boot priority for a zone. Once the SMF dependencies are satisfied for a zone, the zone is placed in a queue according to its priority. The ZDR then boots zones from the highest to lowest boot priority in the queue. The new zonecfg property is boot-priority.

set boot-priority={ high | default | low }

Note that the boot ordering based on assigned boot priority is best-effort and thus non-deterministic. It is not guaranteed that all zones with higher boot priority will be booted before all zones with lower boot priority. If your configuration requires a deterministic behavior, use SMF dependencies.

Zones Concurrent Boot and Suspend/Resume Limits

The ZDR can limit the number of concurrent zones booting up or shutting down, and suspending or resuming. The max number of concurrent boot and suspend resume will be determined by the following properties on the ZDR service instance:

$ svccfg -s system/zones:default listprop config/concurrent*
config/concurrent-boot-shutdown  count       0
config/concurrent-suspend-resume count       0

0 or absence of value means there is no limit imposed by the restarter. If the value is N, the restarter will attempt to boot in parallel at most N zones.

The booting process of a NGZ will be considered completed when the milestone/goals of a zone is reached. If the milestone/goals cannot be reached, the zone SMF instance will be placed into maintenance and the booting process for that zone will be deemed complete from the ZDR perspective. Kernel Zones that do not support milestone/goals are considered up when the zone auxiliary state hotplug-cpu is set. KZs with the goal support use private auxiliary states to report back to the host. solaris10 branded zones will be considered up when the zoneadm boot command returns.

Integration with FMA

This requirement was automatically achieved with fully integrating the Zones framework with SMF.


Let's have a very simplistic example with zones jack, joe, lady, master, and yesman<0-9>. Now, the master zone depends on lady, lady depends on both jack and joe, and we do not care much about when yesman<0-9> zones boot up.

+------+              +------+               +--------+
| jack |<----------+--| lady |<--------------| master |
+------+          /   +------+               +--------+
+------+        /
| joe  |<------+

+---------+      +---------+
| yesman0 | .... | yesman9 |
+---------+      +---------+

Let's not tax the system excessively when booting, so we set the boot concurrency to 2 for this example. Also, let's assume we need a running web server in jack, so add that one to the goals milestone.

Based on the environment we have, we choose to assign the high boot priority to jack and joe, keep lady and master at the default priority, and put all yesman zones to the low boot priority.

To achieve all of the above, this is what we need to do:

# svccfg -s system/zones:default setprop config/concurrent-boot-shutdown=2
# zlogin jack svcadm goals svc:/milestone/multi-user-server:default apache24
# zonecfg -z jack "set boot-priority=high"
# zonecfg -z joe "set boot-priority=high"
# zonecfg -z lady "add smf-dependency; set fmri=svc:/system/zones/zone:jack; end"
# zonecfg -z lady "add smf-dependency; set fmri=svc:/system/zones/zone:joe; end"
# zonecfg -z master "add smf-dependency; set fmri=svc:/system/zones/zone:lady; end"
# for i in $(seq 0 9); do zonecfg -z yesman$i "set boot-priority=low"; done

During the boot, you may see something like the following. As mentioned above, the boot priority is best effort also given we have dependencies, some yesman zones will boot up before some higher priority zones. You will see that at any given moment during the boot, only two zones are being booted up in parallel (the '*' denotes a service in a state transition, see svcs(1)), as we set the boot concurrency above to 2.

$ svcs -o STATE,FMRI -s FMRI system/zones/*
STATE          FMRI
offline*       svc:/system/zones/zone:jack
online         svc:/system/zones/zone:joe
offline        svc:/system/zones/zone:lady
offline        svc:/system/zones/zone:master
offline        svc:/system/zones/zone:yesman0
offline        svc:/system/zones/zone:yesman1
offline        svc:/system/zones/zone:yesman2
offline*       svc:/system/zones/zone:yesman3
online         svc:/system/zones/zone:yesman4
online         svc:/system/zones/zone:yesman5
offline        svc:/system/zones/zone:yesman6
offline        svc:/system/zones/zone:yesman7
offline        svc:/system/zones/zone:yesman8
online         svc:/system/zones/zone:yesman9


With the Zones Delegated Restarter introduced in 11.4, we resolved several shortcomings of the Zones framework in 11.3. There is always room for additional enhancements, making the boot ordering based on boot priorities more deterministic, for example.

We are open to any feedback you might have on this new Zones Delegated Restarter feature.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.