Friday Aug 14, 2015

Inspecting Scheduled Services

In a recent post, we looked at how to create scheduled services in Oracle Solaris 11.3.  Today we're going to discover how to interact with them.  SMF tracks certain information that applies only to scheduled services, such as the next time the service is scheduled to run.  It is easy to find this information using familiar tools.


I've created a very simple scheduled service that executes once a day.  To keep things simple, the start method is set to /usr/bin/true.

<?xml version="1.0" ?>
<!DOCTYPE service_bundle
  SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type="manifest" name="example/scheduled">
    <service version="1" type="service" name="example/scheduled">
        <scheduled_method recover="false"
        <instance enabled="true" name="default"/>

Once the service is imported, we can start looking at it.


Let's start with the most basic question: When is this service going to run next.  We've added a new column, nrun, to svcs(1) to display that information.

# svcs -o nrun example/scheduled

As you can see, this service is going to run today at 23:02:56.  Recall that the service has an interval of 'day' and no constraints, meaning this time has been randomly generated.

Another interesting question is: When did this service last run?  There's a new column for svcs(1) that shows that as well, lrun.

# svcs -o lrun example/scheduled

The dash indicates that there hasn't been a previous run.  That makes sense given that I only created this service today, so it hasn't had time to execute yet.  So I'm going to do something Evil and advance time a bit just to get a number.

# date 2330.30
# svcs -o lrun example/scheduled

Now that I've advanced time past the first scheduled execution time, you can see that the last run field has been populated with the last scheduled run time.  Of course, I advanced the clock so the service didn't actually run at exactly that time because I compressed most of a day into a few microseconds  On a normal system in normal conditions where Evil operators aren't mucking with clocks, the actual run time and the scheduled run time will match.

Let's go look at nrun again...

# svcs -o nrun example/scheduled

As you can see, there's a new execution time.  Remember that only the next smallest unconstrainted time period is kept consistent, and everything else is randomized again.  In our example service, the interval is set to 'day'.  So it runs in the same hour, 23 both times, but the minute and seconds fields are different.

There's one more interesting question:  Is the method for my service actually running or not?  Periodic and scheduled services are always 'online' so long as their dependencies are met and are enabled, so just looking at the service state doesn't tell you that much.  Predictably, there's a new column for that.  It's called 'astate' and corresponds to any auxiliary state a service might have.  Restarters are free to put whatever they want in that field.  The Periodic Restarter uses it to communicate if the method for the service is running or if it's scheduled to run.

# svcs -o astate example/scheduled

Since our service won't run until 23:44:43, its auxiliarly state is 'scheduled' because it is scheduled to run at a later time.  Since our start method is '/usr/bin/true', it's going to be hard to actually catch the service in a running state.

# svcs -o astate example/scheduled

Those three columns represent the truly interesting information about scheduled and periodic services.  Of course, the standard information you'd expect from SMF services is still available as well.

Changing Your Mind

Scheduled services make changing the schedule as easy as reconfiguring anything in SMF.  All you have to do is create a new schedule and refresh the service.  From there, the periodic restarter will pick up the new schedule and recompute the next run time.  Let's make our service run once a week, on Mondays.

# svccfg -s example/scheduled:default
svc:/example/scheduled:default> setprop scheduled/interval = week
svc:/example/scheduled:default> setprop scheduled/day = astring: Monday
svc:/example/scheduled:default> refresh
# svcs -o nrun example/scheduled

As you can see, the new schedule is in place and instead of running at 23:44:43 it will run on Monday, November 9th, 2015.

A Wider Worldview

Looking at scheduled services one by one is interesting enough in its own right, but what if you want to look at all of them at once.  svcs(1) has supported the -R option to look at services delegated to a certain restarter for some time now.  Combining that with the new columns allows you to get a complete picture of the scheduled and periodic services installed on your system.

# svcs -o state,nrun,lrun,astate,fmri -R svc:/system/svc/periodic-restarter:default
STATE          NRUN     LRUN     ASTATE      FMRI
disabled       -        -        none        svc:/application/pkg/sysrepo-cache-compact:default
online         Nov_09   23:02:56 scheduled   svc:/example/scheduled:default

Thursday Jul 30, 2015

Creating Scheduled Services in Oracle Solaris

In the previous post, we introduced the idea of scheduled services in Oracle Solaris as an alternative to cronjobs.  Here, we'll do a deep dive into the scheduling model and create a basic service.

Scheduling Model

Scheduled Services and cronjobs have fundamentally different scheduling models.  The model that cron(1M) uses is based on pattern matching. Each line in a crontab specifies a set of conditions specifying when a job should run. Each minute, the cron daemon wakes up and checks each job to see if those criteria match the current moment.  If they do, then cron executes the job.

Scheduled Services, on the other hand, use an interval and frequency based model similar to iCalendar. Rather than saying that a job should run on each day that matches a given weekday a la cron, scheduled services are told to run 'weekly'. The interval is a property in the schedule and can have one of the following values: year, month, week, day, hour, minute. If the interval is 'year', the service executes once a year. If it's 'day', the services is executed each day. From there, the schedule can be further specified by adding additional properties know as constraints.Constraints for a given interval can be defined starting from the next smallest unit of time. For a weekly interval, the biggest constraint that can be specified is the day of the week. If more control is required, constraints run all the way down to the minute. Again for a weekly interval, the allowable constraints are day, hour, and minute. For a daily schedule, only hour and minute are allowed. One caveat is that constraints must be contiguous, meaning gaps are not allowed. A weekly interval can have day and hour, for example, but not day and minute. This is because the hour is missing. Any value that is not constrained is randomized by the periodic restarter. To make sure Dumb Things don't happen, the periodic restarter only randomizes the first unconstrained unit once. That way something that executes once a year won't run on December 31st of one year and January 1st of the next. It will run in the same month every time.

The periodic restarter supports three different calendar models for scheduled services. One of them uses the Gregorian model where dates are specified by the year, the month, and the day of the month. February 8th, 1976, for example. Another is the ISO Week Date format. This method defines a date using the year, the number of weeks into that year, and a day of the week. This results in dates such as Tuesday, in the 34th week of 2027. Finally, there is a model that arose out of the need to cleanly represent holidays in the United States. Instead of specifying a day of a month, the model supports specifying a day of the week along with which instance of that day in a given month. For example, the 4th Thursday in November.

Here is the full set of constraints allowed for each schedule type:

Gregorian Dates:






ISO Week Dates:






Month-Week Dates:







Most schedules can be adequately defined using a combination of intervals and constraints. But what if you need to execute your service, say, every 2 weeks? To handle this, scheduled services also have a frequency property. The frequency defines how many intervals must pass before the service executes again. To create a scheduled service that runs every two weeks, set the interval to 'week' and the frequency to '2'.

Schedules that have a frequency other than 1 need to define where to start counting. If your service should run every two years, the periodic restarter needs to know if it should be running in even years or odd years. This is known as a reference point. A reference point is defined using the same properties as constraints. The difference is that constraints start at the next shortest unit of time, whereas reference points always start with a year and end with the same unit of time. For the 2 week case, it is necessary to specify at least enough information to determine a specific week of a specific year.  It doesn't matter which of the 3 scheduling models is used, so long as there is enough information to know from what week to start counting.  It should be noted that reference points are not the same thing as a starting date, and is used only for determing when to schedule an execution. A service that runs every 5 years with a reference point of 2050 will run in 2015, 2020, 2025, and so on.  Scheduled Services do not have a notion of a start nor end date and will run both before and after the reference point.

So know what we know what constitutes a schedule, how do we make one? Each service delegated to the periodic restarter can have an arbitrary number of schedules, each in its own property group with type 'schedule'. All properties are optional except the 'interval' property. The periodic restarter then combines all the schedules and computes when the service should next be executed.

Creating a Scheduled Service

The vast majority of scheduled services are going to have a single schedule. To make that use case easier, we've added a new element to the service bundle DTD, 'scheduled_method'. This element does three things. It creates a schedule property group, creates a start method, and also delegates the instance to the periodic restarter. A manifest for a simple scheduled services looks like:

<?xml version="1.0" ?>

<!DOCTYPE service_bundle

SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>

<service_bundle type="manifest" name="example/scheduled">

<service version="1" type="service" name="example/scheduled">





<instance enabled="true" name="default"/>



This service will execute the method '/my-method' once a week.

svcbundle(1M) fully supports scheduled services. In fact, this example was created using that tool with the invocation:

$ svcbundle -s service-name=example/scheduled -s instance-name=default -s start-method=/my-method -s interval=week

I took the liberty of trimming out some of less-interesting parts of the manifest so that it's easier to see the scheduled_method element. svcbundle(1M) knows all about constraints and reference points, and will try very hard to make sure it won't generate an invalid schedule.

Here are two more scheduled_method elements to illustrate some more interesting schedules:

<scheduled_method exec="/my-method"






This schedule runs on US Thanksgiving once a year.

<scheduled_method exec="/my-method"






This schedule runs every two months, counting from June of 2027.

And that's pretty much it. In a follow-up post, we'll dig into inpsecting scheduled services using the standard suite of SMF tools.  Stay tuned.

Thursday Jul 09, 2015

Cron Begone: Predicable Task Scheduling

cron has had a long reign as the arbiter of scheduled system tasks on
Unix systems.  However, it has some critical flaws that make its use
somewhat fraught.  Delivering cronjobs is Hard.  Inserting fragments
into crontab in a reliable and predictable fashion is tricky.  The
fragment-based delivery provided by /etc/cron.d helps, but it’s not perfect.

It’s also far too easy to create systemic problems on a large scale.  If
your cron fragment is packaged and deployed to multiple systems, the
result is that every system runs that job at exactly the same time.  For
jobs that hit a single server, like checking for package updates, the
result is that every machine connects to the server at the same time,
often overloading it.  Like job delivery, it’s possible to work around
the problem, but it takes some gymnastics to do so.

cron also lacks validation, error handling, dependency management, and a
host of other features.

Enter the Periodic Restarter and Scheduled Services.

The Periodic Restarter is a delegated restarter, at
svc:/system/svc/periodic-restarter:default, that allows the creation of
SMF services that represent scheduled or periodic tasks.  Scheduled
Services have all the benefits that other SMF services have, including
error handling, consistent logging, dependencies, predictable lifecycle
management, and more.  We’ve baked in a some extra goodies on top of
that, which I’ll explain in a bit.

As you can imagine, scheduled services have a slightly different model
than other services.  The most visible difference is that instances of
scheduled services can be in the “online” state even when the task isn’t
running.  For these services, the “online” state means that the service
has all of its dependencies met, is scheduled, and is ready to execute
when the appropriate time comes.  The service will stay in the “online”
state while the job is executing, as well.  Other than that, though,
scheduled services behave like any other SMF service.

So what other features do scheduled services have that make them
superior to ordinary cronjobs?

The first is that scheduled services have a mechanism to execute jobs
that have been missed due to system downtime.  Simply by setting a flag
in the service configuration, the periodic restarter will execute your
task when the system comes back up, once all of its dependencies have
been met.  If your task was set to run at 1:35 in the morning, but it
was rebooted over that time, your task will still execute.

The other nice feature is that scheduled services have built in
randomness.  The task has an execution interval that can be constrained
as necessary, but only as necessary.  Everything else is automatically
randomized.  Maybe a task has to be run once a month, but it doesn’t
matter at all when during the month.  For such a task, the only thing
that must be specified is that the task should execute once a month.
From there, the periodic restarter will pick a time during the month to
execute the job.  Another task, such as log rotation, may need to run
during off hours every day.  In that case, you can constrain the
schedule so that it runs during a particular hour, but the exact time
during that hour will be randomized.  This feature enables the
deployment of scheduled tasks on a wide scale without having to consider
load balancing, both between systems and between tasks on the same system.

In subsuquent updates, we’ll discuss how to create scheduled services as
well as how to inspect some of the extra state that comes along with

Monday Jun 29, 2015

Introduction to svcbundle(1M)

Many people state that one of the roadblocks to creating a new SMF service for Oracle Solaris is the need to write the service manifest in XML. The svcbundle(1M)program is intended to help you get by that roadblock. Based on the command line options svcbundle generates a valid service manifest or profile. In all but the simplist of cases, you will need to hand edit the generated file, but svcbundle gets you off to a good start. This is the first in a series of blogs that discusses what you can do with svcbundle.

svcbundle is driven by specifying multiple -s options on the command line. Each -s is followed by a name=value pair. If you type svcbundle help, you will get a list of items that you use in the name part of the name=value pair. Use of the various names is documented in the man page, or typing the name as an argument of the svcbundle help command gives you more information about it:

$ svcbundle help bundle-type
-s bundle-type=type
    Specifies the type of bundle to generate.  Legal values are "manifest"
    and "profile".  The default is manifest.

A Simple Example

The simplest svcbundle command that you can type requires the service-name and start-method names. Most likely you'll want to use more name=value pairs to get a useful manifest, but I'll discuss that in a future blog. Let's look at what you get with just these two pairs.

$ svcbundle -s service-name=myservice \
        -s start-method="/lib/svc/method/myservice %m"

Note the use of quotes to protect the white space in the start-method command. %m tells svc.startd to pass in the name of method being invoked to the myservice command. See smf_method(5) for details. I've captured the output and added line numbers so that we can talk about the generated manifest.

 1 <?xml version="1.0" ?>
 2 <!DOCTYPE service_bundle
 3   SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
 4 <!--
 5     Manifest created by svcbundle (2015-Jun-17 10:38:04-0700)
 6 -->
 7 <service_bundle type="manifest" name="myservice">
 8     <service version="1" type="service" name="myservice">
 9         <!--
10             The following dependency keeps us from starting until the
11             multi-user milestone is reached.
12         -->
13         <dependency restart_on="none" type="service"
14             name="multi_user_dependency" grouping="require_all">
15             <service_fmri value="svc:/milestone/multi-user"/>
16         </dependency>
17         <exec_method timeout_seconds="60" type="method" name="start"
18             exec="/lib/svc/method/myservice %m"/>
19         <!--
20             The exec attribute below can be changed to a command that SMF
21             should execute to stop the service.  See smf_method(5) for more
22             details.
23         -->
24         <exec_method timeout_seconds="60" type="method" name="stop"
25             exec=":true"/>
26         <!--
27             The exec attribute below can be changed to a command that SMF
28             should execute when the service is refreshed.  Services are
29             typically refreshed when their properties are changed in the
30             SMF repository.  See smf_method(5) for more details.  It is
31             common to retain the value of :true which means that SMF will
32             take no action when the service is refreshed.  Alternatively,
33             you may wish to provide a method to reread the SMF repository
34             and act on any configuration changes.
35         -->
36         <exec_method timeout_seconds="60" type="method" name="refresh"
37             exec=":true"/>
38         <property_group type="framework" name="startd">
39             <propval type="astring" name="duration" value="transient"/>
40         </property_group>
41         <instance enabled="true" name="default"/>
42         <template>
43             <common_name>
44                 <loctext xml:lang="C">
45                     <!--
46                         Replace this comment with a short name for the
47                         service.
48                     -->
49                 </loctext>
50             </common_name>
51             <description>
52                 <loctext xml:lang="C">
53                     <!--
54                         Replace this comment with a brief description of
55                         the service
56                     -->
57                 </loctext>
58             </description>
59         </template>
60     </service>
61 </service_bundle>

Editing the Manifests

Note that svcbundle includes comments in the generated manifest to guide you in edits that you might wish to make. Let's look at some places in the manifest where I suggest you consider making changes.

First consider the dependency declaration at lines 13-16. The default dependency generated by svcbundle makes the service dependent on the multi-user milestone. You may want to edit this so that your service is dependent on a different milestone or a specific service.

Next consider the declaration of the start method in lines 17-18. The default timeout_seconds of 60 seconds may be too short for your service, so you will want to increase it. We quite often find that service developers under estimate the amount of time that it takes to start a service. Think about what might happen if the service is running in a zone where many non-global zones are being started simultaneously.

Finally, you should always edit <common_name> (lines 43-50) and <description> (lines 51-58) of the template section. Replace the comments in the <common_name> and <description> sections of the template to provide meaningful data there.

svcbundle Provided Defaults

In the simple example svcbundle used default values for many entries in the manifest. I'd like to point out a few of these, just so that you will be aware of them. These are not items that you'll need to edit, though. If you don't like the defaults, you can change things with some of the other name options on the command line.

In lines 24-25 the stop method has been set to :true. In lines 38-39 the service has been declared as a transient service.  See svc.startd(1M) for a discussion of service models. Finally, in line 41, the instance name is declared as default, and the instance is enabled. As we'll see in my next blog, you can change all of the default assignments by specifying more name=value pairs on the command line, so no need to edit

Friday Aug 15, 2014

Synchronous Actions

Since the introduction of SMF, svcadm(1M) has had the ability to enable or disable a service instance and wait for that service instance to reach a final state.  With Oracle Solaris 11.2, we’ve expanded the set of administrative actions which can be invoked synchronously. Now all subcommands of svcadm(1M) have synchronous behavior. Let’s take a look at the new usage:

Usage: svcadm [-v] [cmd [args ... ]]

svcadm enable [-rt] [-s [-T timeout]] <service> ...

enable and online service(s)

svcadm disable [-t] [-s [-T timeout]] <service> ...

disable and offline service(s)

svcadm restart [-s [-T timeout]] <service> ...

restart specified service(s)

svcadm refresh [-s [-T timeout]] <service> ...

re-read service configuration

svcadm mark [-It] [-s [-T timeout]] <state> <service> ...

set maintenance state

svcadm clear [-s [-T timeout]] <service> ...

clear maintenance state

svcadm milestone [-d] [-s [-T timeout]] <milestone>

advance to a service milestone

svcadm delegate [-s] <restarter> <svc> ...

delegate service to a restarter

As you can see, each subcommand now has a ‘-s’ flag. That flag tells svcadm(1M) to wait for the subcommand to complete before returning. For enables, that means waiting until the instance is either ‘online’ or in the ‘maintenance’ state. For disable, the instance must reach the ‘disabled’ state. Other subcommands complete when:


A restart is considered complete once the instance has gone offline after running the ‘stop’ method, and then has either returned to the ‘online’ state or has entered the ‘maintenance’ state.


If an instance is in the ‘online’ state, a refresh is considered complete once the ‘refresh’ method for the instance has finished.

mark maintenance

Marking an instance for maintenance completes when the instance has reached the ‘maintenance’ state.

mark degraded

Marking an instance as degraded completes when the instance has reached the ‘degraded’ state from the ‘online’ state.


A milestone transition can occur in one of two directions. Either the transition moves from a lower milestone to a higher one, or from a higher one to a lower one. When moving to a higher milestone, the transition is considered complete when the instance representing that milestone reaches the ‘online’ state. The transition to a lower milestone, on the other hand, completes only when all instances which are part of higher milestones have reached the ‘disabled’ state.

That’s not the whole story. svcadm(1M) will also try to determine if the actions initiated by a particular subcommand cannot complete. Trying to enable an instance which does not have its dependencies satisfied, for example, will cause svcadm(1M) to terminate before that instance reaches the ‘online’ state.

You’ll also notice the optional ‘-T’ flag which can be used in conjunction with the ‘-s’ flag. This flag sets a timeout, in seconds, after which svcadm gives up on waiting for the subcommand to complete and terminates. This is useful in many cases, but in particular when the start method for an instance has an infinite timeout but might get stuck waiting for some resource that may never become available.

For the C-oriented, each of these administrative actions has a corresponding function in libscf(3SCF), with names like smf_enable_instance_synchronous(3SCF) and smf_restart_instance_synchronous(3SCF).  Take a look at smf_enable_instance_synchronous(3SCF) for details.

Thursday Aug 07, 2014

svcs' new -L option

In Solaris 11.2, svcs gained a new option, “-L”.  The -L option allows a user to easily look at the most recent log events for a service.

The addition of the -x option (e.g svcs -xL mysvc:default) will print the normal -x output for the service:instance given, but will additionally print one of the following subsets of log file content :

1. Start at the last occurrence of the log message for a method exit without errors.
2. The last 10 lines of the log file if smaller than (1)
3. If the last line is the successful exit then print the last 5 lines of the log file
4. If the last method exit has errors, the last ten lines of the log file is presented.

In the following example, the start method exited with an error, and the last ten lines
of the log file are shown.

# svcs -xL svc:/mysvc:myinst
 State: maintenance since Mon Aug 04 15:17:51 2014
Reason: Start method failed repeatedly, last exited with status 1.
   See: /var/svc/log/mysvc:myinst.log
Impact: This service is not running.
[ 2014 Aug  4 15:17:51 Leaving maintenance because clear requested. ]
[ 2014 Aug  4 15:17:51 Enabled. ]
[ 2014 Aug  4 15:17:51 Executing start method ("/root/blah.ksh start"). ]
Hello... starting 1
Hello... starting 2
Hello... starting 3
Hello... starting 4
Hello... starting 5
[ 2014 Aug  4 15:17:51 Method "start" exited with status 1. ]
[ 2014 Aug  4 15:18:09 Rereading configuration. ]

Making this the default output of svcs -x is under consideration for a future release of Solaris.

Simply using -L by itself, the name of the log file is given and you could do the following to print the contents of the log file :

# less `svcs -L mysvc:default` 

Or add the -v option (svcs -vL mysvc:default) to display the full content of the log file.

# svcs -vL mysvc:myinst | less
[ 2014 Jul 25 10:10:08 Enabled. ]
[ 2014 Jul 25 10:10:08 Rereading configuration. ]
[ 2014 Jul 25 10:10:08 Executing start method (:true). ]
[ 2014 Jul 25 11:06:11 Enabled. ]
[ 2014 Jul 25 11:06:11 Rereading configuration. ]
[ 2014 Jul 25 11:06:11 Executing start method (:true). ]
[ 2014 Jul 30 12:52:04 Executing stop method ("/root/blah.ksh stop"). ]
Hello... stopping
[ 2014 Jul 30 12:52:04 Method "stop" exited with status 0. ]
[ 2014 Aug  4 15:14:09 Disabled. ]
[ 2014 Aug  4 15:14:59 Enabled. ]

If multiple instances are provided with the -vL option then only the last 10 lines
of the log files are presented.  For example :

svcs -Lv svc:/system/name-service/cache:default svc:/system/fm/smtp-notify:default

svc:/system/name-service/cache:default (name service cache)
[ 2014 Jun 20 09:10:32 Executing start method ("/lib/svc/method/svc-nscd start"). ]
[ 2014 Jun 20 09:10:33 Method "start" exited with status 0. ]
[ 2014 Jul 17 08:34:25 Disabled. ]
[ 2014 Jul 17 08:34:26 Enabled. ]
[ 2014 Jul 17 08:35:07 Executing start method ("/lib/svc/method/svc-nscd start"). ]
[ 2014 Jul 17 08:35:08 Method "start" exited with status 0. ]
[ 2014 Jul 28 05:31:51 Stopping because service restarting. ]
[ 2014 Jul 28 05:31:52 Executing stop method (:kill). ]
[ 2014 Jul 28 05:31:52 Executing start method ("/lib/svc/method/svc-nscd start"). ]
[ 2014 Jul 28 05:31:52 Method "start" exited with status 0. ]

svc:/system/fm/smtp-notify:default (Solaris Email Event Notification Agent)
[ 2014 Apr 30 08:47:06 Method "start" exited with status 0. ]
[ 2014 May 24 13:12:14 Enabled. ]
[ 2014 May 24 13:22:49 Executing start method ("/usr/lib/fm/notify/smtp-notify"). ]
[ 2014 May 24 13:22:49 Method "start" exited with status 0. ]
[ 2014 Jun 20 09:09:43 Enabled. ]
[ 2014 Jun 20 09:21:58 Executing start method ("/usr/lib/fm/notify/smtp-notify"). ]
[ 2014 Jun 20 09:21:58 Method "start" exited with status 0. ]
[ 2014 Jul 17 08:34:30 Enabled. ]
[ 2014 Jul 17 08:45:47 Executing start method ("/usr/lib/fm/notify/smtp-notify"). ]
[ 2014 Jul 17 08:45:47 Method "start" exited with status 0. ]

Tuesday Apr 29, 2014

Introducing SMF Stencils

    As much as we'd like to believe that every application that runs on
Oracle Solaris 11.2 uses SMF directly to maintain configuration, the reality is
that it's a difficult or downright impossible goal for many types of software.
Be it a 3rd party service, some piece of FOSS, or software for which the
maintainers are long forgotten; some services simply must continue to use files
to store their configuration.  Does this mean managing Oracle Solaris machines
eternally dooms you to multiple configuration management tools?  It did, until

    Enter SMF Stencils.

    Simply put, SMF Stencils is a framework for converting configuration
data stored in the repository to the files used to configure particular
applications.  Using a small bit of metadata about the configuration files and
a template describing them, SMF can automatically generate and regenerate those
files at the appropriate times.  So now you can have your cake and eat it too;
your applications don't need to be rewritten to take advantage of SMF directly,
but can still reap the benefits of SMF's configuration management


    svcio(1), which lives at /lib/svc/bin/svcio, is the program which
actually handles file generation.  It reads in both the SMF repository and any
stencil files configured for a service, processes them, and writes any
resultant files to the appropriate path.  The master restarter, in turn,
invokes svcio before either the start or refresh methods for a service are run.
 This way, the configuration files for a service always reflect the latest set
of data stored in the SMF repository.  Generally speaking, it's not necessary
to know that svcio exists let alone how it works, but it's sometimes useful to
invoke it directly (for example, when testing a stencil).  The usage is as

Usage: svcio [-alux] [-f fmri] [-g group] [-i file] [-L opts]
        [-m mode] [-o file] [-O user] [-R dir] [-S dir]

        -a  process all configfile dependencies for fmri
        -f  set instance fmri (default is $SMF_FMRI)
        -g  set the group any output files are associated with
        -i  set input file (default is stdin)
        -l  list referenced properties from input file
        -L  use lofs <opts> to loopback mount the output file(s)
        -m  set output file mode (default is 644)
        -o  set output file (default is stdout)
        -O  set the owner of any output files
        -R  set root prefix for all output files
        -S  set stencil directory (default is /lib/svc/stencils)
        -u  unlink output file(s) and undo any lofs mounts
        -x  terminate svcio on any error

    The vast majority of the options are for esoteric uses that are beyond
the scope of this blog, but three of them, -a, -f, and -i, are useful for
developing a stencil.  The -f switch tells svcio which service instance to
process stencils for, and the -a option makes it process all stencils for the
specified instance.  We'll explore that in more detail shortly.  The -i, on the
other hand, tells svcio to skip any configured stencils for the service
instance and forces it to use a specific stencil file.  Without a -o to tell it
where to write the result, svcio will write the processed filed to stdout.  For
now, it's best to ignore the other options.

Making a Service Stencil-Aware

    Configuring a service to use stencils is quite simple.  All that's
required is to create a property group, with a few properties, for each
configuration file that SMF should manage.  The property group can have any
name, but must be of type 'configfile', and it is possible to have as many
property groups as the application has configuration files.  svcio understands
5 properties in this property group, 3 of which are mandatory:

    stencil        The path to the stencil file, rooted at
            /lib/svc/stencils.  svcio will use the value of this
            property to determine which stencil to read in.
    path        The path to the output file.  svcio will write the
            results of processing the stencil to this path.
    mode        The octal mode for the output file.
    user        The user to create the output file under.  This property
            is optional.
    group        The group to create the output file in.  This property
            is optional.

Our Sandbox Service

    In the interest of providing concrete examples, we'll start by creating
a service to play around in.  Let's assume we're converting a simple payroll
manager that uses an INI style file to store its configuration data.  The
manifest for such a service has been attached to this entry, but a portion of
the svcprop output for the service is provided here for reference:

    svcprop test/payroll-manager:default

    account/account_number integer 12345
    account/routing_number integer 918273645
    payee_1/address astring 1234\ 1st\ St.\ \ Mapleton,\ MA.\ 06033
    payee_1/name astring Robert\ Bruce
    payee_1/salary integer 55000
    payee_2/address astring 6N418\ County\ Lane\ 1.\ \ Cairo,\ WI.\ 60993
    payee_2/name astring Catherine\ Denninger
    payee_2/salary integer 64000
    payroll_stencil/mode integer 600
    payroll_stencil/path astring /etc/payroll.conf
    payroll_stencil/stencil astring payroll.conf.stencil

    There's some basic information here about the account to use to pay the
two payees: Robert Bruce and Catherine Denninger. Ignore the payroll_stencil
property group for the time being, which will be addressed later on.

    To follow along with these examples, just import the manifest for the
service, payroll_manager.xml, and copy the stencil, payroll.conf.stencil, to
/lib/svc/stencils.  Alternately, you can follow along with each iteration of the
stencil file to /lib/svc/stencils/payroll.conf.stencil.  The version of
payroll.conf.stencil attached here is only the final iteration.

Writing a Stencil

    The phrase "stencil file" has been tossed around quite a bit here, but
it hasn't been defined.  So what exactly is a stencil file?

    A stencil file defines how properties and property groups in the SMF
repository relate to the structure of a configuration file.  At the most basic,
stencils use property and property group names to retrieve values of properties
and write them into a file in much the same way as method tokens behave.  A
simple stencil for a simple configuration file might contain only a few property
expressions, whereas a more complicated stencil might have a huge variety of
configuration options that interact in various ways.

    We're going to construct a stencil for our payroll manager.
The configuration file contains a section for basic account information, along
with a section for each payee.  An example file would look like:

        account_number = 12345
        routing_number = 918273645

    [Robert Bruce]
        salary = 55000
        address = 1234 1st St.  Mapleton, MA. 06033

    [Catherine Denninger]
        salary = 64000
        address = 6N418 County Lane 1.  Cairo, WI. 60993

    Retreiving Property Values

    First we'll create the portion of the stencil which handles account
information.  Since this section can appear only once and has only a few
values, we can just hardcode the section into the stencil.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}


    Matching Properties and Property Groups

    Many configuration files can be fully represented using simple
substitution like the example above.  However, many other files have many
sections with duplicate information (Puppet, Samba, Kerberos, to name a few).
It's unreasonable to change the stencil for each new grouping created, so the
stencil markup language has a facility to iterate over properties and property
groups.  To keep things flexible, svcio uses regular expression matching on
property and property group names to determine which properties and property
groups to iterate over.  In our example, each payee in the system is identified
with a property group named "payee_<number>", with a unique number per payee.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}

    $%/payee_([0-9]*)/ {
        salary = $%{payee_$%1/salary}
        address = $%{payee_$%1/address}


    For those familiar with regular expression, it should be relatively
easy to figure out what's going on.  For everyone else, let's go through this
line by line.

    $%/payee_([0-9]*)/ {

    This line defines the regular expression which is matched against all
property and property group names for the instance that svcio(1) is processing
the stencil against.  The regular expression itself is the text between "$%/"
and "/ {".  It's worth noting that the regular expression is terminated with the
character sequence "/<whitespace>{", where arbitrary whitespace is allowed.
This means that it is permissible to use unescaped forward slash characters
within a regular expression, making it much simpler to match on strings that
include both property groups and properties (e.g. "$%/general/(.*)/ {" is a
perfectly legal regular expression).


    Notice the parentheses in the regular expression on the previous line.
These parentheses indicate a submatch within the total match.  Within the
braces, each submatch can be referenced using the sequence
"$%<submatch_index>".  Since there's only one submatch, it must be the first,
and is referenced by $%1.  svcio(1) will substitute any text within the
submatch and continue processing the stencil.  So if we matched on "payee_1",
this line will become "[$%{payee_1/name}]" which will, in turn, be translated
into "[Robert Bruce]".

    salary = $%{payee_$%1/salary}
    address = $%{payee_$%1/address}

    The pattern established on the previous line is continued here.


    This brace terminates the regular expression block.  For each match,
all the text between the two braces will get written to the output file.  In
our example, this means that each payee property group will be translated into
a few lines of text in the output file.

    Composing Regular Expressions

    Since our little payroll manager can't yet read e-mail, it is safe to
assume it will expand over time.  One likely place for expansion is extra
configuration in the payee sections.  With that in mind, it is probably not a
good idea to hard-code the properties printed in each payee section.  The
stencil markup language allows the composition of regular expressions to
further generalize stencils and support cases just like this one.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}

    $%/payee_([0-9]*)/ {
    $%/payee$%1/([^n].*)/ {\t$%2 = $%{payee_$%1/$%2}\n}


    Most of this version is identical to the previous version, just with
the hardcoded entries for salary and address removed in favor of another
regular expression.  Since the second regular expression is embedded in the
first, some adjustment must be done with the submatch numbers.  For embedded
regular expressions, any submatch indices take the outer expressions into
account.  In this example, the outer expression has one submatch, so the inner
expression starts counting at two.  If the outer expression had two submatches,
the inner expression would start counting at three.  This way it is possible to
use submatches from any outer regular expressions within inner expressions
(hence the $%{payee_$%1/$%2}).

    With this modification, any new properties that are added to payee
information will automatically propagate to the configuration file, provided the
name of the property does not start with the letter 'n' (this was done to
prevent the property 'name' from appearing).  Clever readers will be able to
modify this regular expression to include all property names which are not
the word 'name'.

    Putting It All Together

    Now that we have a good stencil, we can add the appropriate
configuration to the service so that the master restarter will run svcio(1)
before running a start method or refreshing.  The first step is to create a
property group with the type set to "configfile", which we'll call
"payroll_stencil".  After that, the following properties are necessary:

    payroll_stencil/stencil = payroll.conf.stencil
    payroll_stencil/path = /etc/payroll.conf
    payroll_stencil/mode = 600

    Using those properties, svcio(1) will read the stencil file from
/lib/svc/stencils/payroll.conf.stencil and use it generate a file at
/etc/payroll.conf with the mode set to 600.

    And that's it!  Whenever the configuration has to be changed, just
refresh the instance (assuming it's online) or enable the instance and go check
/etc/payroll.conf.  It will have changed to reflect the updated configuration.



Friday Feb 03, 2012

Changes to svccfg import and delete

The behavior of svccfg import and svccfg delete fmri has changed in S11 if the manifests are in SMF's standard locations. The standard locations are /lib/svc/manifest and /var/svc/manifest with /lib/svc/manifest being the preferred location. If your manifest is stored under one of these two directories, you shouldn't be using svccfg import at all. You should only use svccfg delete fmri when you are prohibited from removing the manifest for some reason and still want to remove the service from the system. Instead, the preferred action is:

        # svcadm restart manifest-import


The reason is that in S11, SMF keeps the repository in sync with the files in the standard locations, and the manifest-import service is the mechanism for maintaining this synchronization. So instead of using svccfg import copy your manifest to a standard location and type:

        # svcadm restart manifest-import
Instead of using svccfg delete remove your manifest from its location and restart manifest-import.


In each case manifest-import will detect any changes in the standard directories and update the repository accordingly. Note that the manifest-import service runs asynchronously from the svcadm command, so it may take a short amount of time for the changes to take effect.

Also the manifest-import service not only detects file additions and removals. It also detects changes to manifests and profiles. If you are a service provider, this gives you an upgrade path if your manifest changes. Simply deposit your new manifest over the old one and make sure that manifest-import is restarted. Restarting of manifest-import is usually handled by the packaging service.

Let's look at some examples. First, let's get the manifest for our new service imported.

# cp mysvc.xml /lib/svc/manifest/site
# svcadm restart manifest-import
# svcs mysvc
STATE          STIME    FMRI
online         15:19:41 svc:/mysvc:default
Now delete the service:
# rm /lib/svc/manifest/site/mysvc.xml 
# svcadm restart manifest-import
# svcs mysvc
svcs: Pattern 'mysvc' doesn't match any instances
STATE          STIME    FMRI


Now let's look at what happens if you stray from this advice and use svccfg delete. First, reinstall the manifest just as we did before.

# cp mysvc.xml /lib/svc/manifest/site
# svcadm restart manifest-import
# svcs mysvc
STATE          STIME    FMRI
online         15:34:41 svc:/mysvc:default
Now the fun begins.
# svccfg delete -f svc:/mysvc
# svcs mysvc
svcs: Pattern 'mysvc' doesn't match any instances
STATE          STIME    FMRI
It looks as if the service has been removed from the repository, but it really hasn't been. Since the manifest file is still on the file system, the service is merely masked in the repository. This can lead to confusion. Even if you modify the manifest and restart manifest-import, svcs will not find the service. This is because the masking is done at the administrative layer (see Sean Wilcox's discussion of layers). The masking is not removed by changing the manifest, although manifest-import will record the changes in the repository.


How can we find a masked service.

# svccfg listcust -M | grep svc:/mysvc
svc:/mysvc manifest MASKED
svc:/mysvc:default manifest MASKED
The first line of output shows that the service is masked. Masking a service also masks it instances which is why we see the second line.


So if you accidentally mask a service, how can you unmask it? We enter svccfg interactive mode, select the service and then use the delcust command.

# svccfg
svc:> select mysvc
svc:/mysvc> delcust
 Deleting customizations for service: mysvc
svc:/mysvc> quit
# svcs mysvc
STATE          STIME    FMRI
online         15:50:46 svc:/mysvc:default
The svcs command shows that the service is unmasked.


Standard Locations (why?)

The manifest-import service manages importing of manifests that are delivered as part of a package for an application.  This instantiates the service and its instances on the system.  The manifest-import service will then manage re-importing those manifests if they are modified/upgraded in some way.

Also, the manifest-import service manages the application of profiles that are in the standard location for profiles of /etc/svc/profile/site.  If these profiles change or are removed then the support for them can be removed from the services on the system.

Finally once the application has served its purpose and the delivering package is removed from the system along with the removal of the manifest the service will then be cleaned up by the manifest-import service.

With that the manifest-import service needs a well known place to be able to find the manifests for a service, and be able to find those manifests under a certain set of rules.  One, the cleanup side of the service needs to be able to know for sure that a manifest is removed and not that a location is simply temporarily unavailable.

Before the Solaris 11, the manifests were located in /var/svc/manifest.  But this location might or might not be available at boot because /var can be a separate filesystem, that is not mounted early in boot.  With the Solaris 11, the manifests were moved to /lib/svc/manifest so that the manifests would be available at the beginning of system boot. Therefore, manifests are to no longer be placed in /var/svc/manifest as it is strictly supported for backwards compatibility only.

So with this standard location that is guaranteed to be available at boot SMF can now make sure that changes and upgrades to manifests are imported before any services are started.  This way services that need to start early in the boot cycle (even before /var might be mounted, if the manifest is in /lib/svc/manifest) will be guaranteed to start with their new property values.

Also, if the manifest is removed from the system, there is a chance for the service to be removed from the system before it attempts to start and does not find the service binaries and/or other files that may be required for the service to run.

Put your manifests and profiles in the standard location and let SMF manage your import, apply and ultimately the cleanup of your services and instances.

So in summary the benefits are :

1. manifests can be imported early in boot before any services are started that might use the information from the manifests.
2. upgrades of manifests and profiles can be done during this early boot phase as well, so that services get the new information before they start.
3. if manifests are removed, then the manifest-import service that manages these can know for sure that the manifests are removed and clean up the services.
4. manifests in a standard location are the base layer for services and their property group and property sets.

Wednesday Nov 09, 2011

Introducing SMF Layers

SMF Layers, meta data for your services.  With S11 we added a new feature that will allow for better tracking of where services, instances, property groups and properties originate and how they may change over there life span. We store a layer at which the entity exists.  At this time there are 4 layers at which an entity can exist.  These layers are based on the location of its delivering or contributing data.

The layers are :

system profile
site profile

Manifest layer entities are delivered or contributed from a standard location for manifests, /lib/svc/manifest or /var/svc/manifest.

System profile layer entities are delivered in the system profile.  A profile that is delivered by Solaris as part of the operating system itself.  At this time this is the generic.xml manifest and a select few other files included into that file.

Site profile layer entities are delivered in a profile located under the /etc/svc/profile/site directory or a subdirectory created under this directory.

Everything else falls under the administrative layer.  Anything delivered outside of the standard locations for manifests and profiles.  As well as any command line creation of entities whether it be through svccfg or the libscf(3LIB) interfaces.

Starting at the manifest layer (the lowest) each of the next layers will override the previous layer.  Although, each layer's data is stored in the repository so that, if one of the higher layers is removed, it will reveal any underlying layers.  This makes upgrading but not overriding higher layer changes a simple action of putting the new files in place and letting the entities be
added to their appropriate layer.

One of the primary concepts we kept to when designing and implementing this project, was that the repository must represent the filesystem in the standard locations, and all other changes are administrative customizations to the system.

So how do Layers benefit us?  Layers allow us to properly upgrade service configurations while preserving customizations.  They provide better observability into service configurations, and the ability to undo customizations

Solaris Service Management Facility information, tips and tricks.


« October 2015