Friday Aug 15, 2014

Synchronous Actions

Since the introduction of SMF, svcadm(1M) has had the ability to enable or disable a service instance and wait for that service instance to reach a final state.  With Oracle Solaris 11.2, we’ve expanded the set of administrative actions which can be invoked synchronously. Now all subcommands of svcadm(1M) have synchronous behavior. Let’s take a look at the new usage:

Usage: svcadm [-v] [cmd [args ... ]]

svcadm enable [-rt] [-s [-T timeout]] <service> ...

enable and online service(s)

svcadm disable [-t] [-s [-T timeout]] <service> ...

disable and offline service(s)

svcadm restart [-s [-T timeout]] <service> ...

restart specified service(s)

svcadm refresh [-s [-T timeout]] <service> ...

re-read service configuration

svcadm mark [-It] [-s [-T timeout]] <state> <service> ...

set maintenance state

svcadm clear [-s [-T timeout]] <service> ...

clear maintenance state

svcadm milestone [-d] [-s [-T timeout]] <milestone>

advance to a service milestone

svcadm delegate [-s] <restarter> <svc> ...

delegate service to a restarter

As you can see, each subcommand now has a ‘-s’ flag. That flag tells svcadm(1M) to wait for the subcommand to complete before returning. For enables, that means waiting until the instance is either ‘online’ or in the ‘maintenance’ state. For disable, the instance must reach the ‘disabled’ state. Other subcommands complete when:


A restart is considered complete once the instance has gone offline after running the ‘stop’ method, and then has either returned to the ‘online’ state or has entered the ‘maintenance’ state.


If an instance is in the ‘online’ state, a refresh is considered complete once the ‘refresh’ method for the instance has finished.

mark maintenance

Marking an instance for maintenance completes when the instance has reached the ‘maintenance’ state.

mark degraded

Marking an instance as degraded completes when the instance has reached the ‘degraded’ state from the ‘online’ state.


A milestone transition can occur in one of two directions. Either the transition moves from a lower milestone to a higher one, or from a higher one to a lower one. When moving to a higher milestone, the transition is considered complete when the instance representing that milestone reaches the ‘online’ state. The transition to a lower milestone, on the other hand, completes only when all instances which are part of higher milestones have reached the ‘disabled’ state.

That’s not the whole story. svcadm(1M) will also try to determine if the actions initiated by a particular subcommand cannot complete. Trying to enable an instance which does not have its dependencies satisfied, for example, will cause svcadm(1M) to terminate before that instance reaches the ‘online’ state.

You’ll also notice the optional ‘-T’ flag which can be used in conjunction with the ‘-s’ flag. This flag sets a timeout, in seconds, after which svcadm gives up on waiting for the subcommand to complete and terminates. This is useful in many cases, but in particular when the start method for an instance has an infinite timeout but might get stuck waiting for some resource that may never become available.

For the C-oriented, each of these administrative actions has a corresponding function in libscf(3SCF), with names like smf_enable_instance_synchronous(3SCF) and smf_restart_instance_synchronous(3SCF).  Take a look at smf_enable_instance_synchronous(3SCF) for details.

Thursday Aug 07, 2014

svcs' new -L option

In Solaris 11.2, svcs gained a new option, “-L”.  The -L option allows a user to easily look at the most recent log events for a service.

The addition of the -x option (e.g svcs -xL mysvc:default) will print the normal -x output for the service:instance given, but will additionally print one of the following subsets of log file content :

1. Start at the last occurrence of the log message for a method exit without errors.
2. The last 10 lines of the log file if smaller than (1)
3. If the last line is the successful exit then print the last 5 lines of the log file
4. If the last method exit has errors, the last ten lines of the log file is presented.

In the following example, the start method exited with an error, and the last ten lines
of the log file are shown.

# svcs -xL svc:/mysvc:myinst
 State: maintenance since Mon Aug 04 15:17:51 2014
Reason: Start method failed repeatedly, last exited with status 1.
   See: /var/svc/log/mysvc:myinst.log
Impact: This service is not running.
[ 2014 Aug  4 15:17:51 Leaving maintenance because clear requested. ]
[ 2014 Aug  4 15:17:51 Enabled. ]
[ 2014 Aug  4 15:17:51 Executing start method ("/root/blah.ksh start"). ]
Hello... starting 1
Hello... starting 2
Hello... starting 3
Hello... starting 4
Hello... starting 5
[ 2014 Aug  4 15:17:51 Method "start" exited with status 1. ]
[ 2014 Aug  4 15:18:09 Rereading configuration. ]

Making this the default output of svcs -x is under consideration for a future release of Solaris.

Simply using -L by itself, the name of the log file is given and you could do the following to print the contents of the log file :

# less `svcs -L mysvc:default` 

Or add the -v option (svcs -vL mysvc:default) to display the full content of the log file.

# svcs -vL mysvc:myinst | less
[ 2014 Jul 25 10:10:08 Enabled. ]
[ 2014 Jul 25 10:10:08 Rereading configuration. ]
[ 2014 Jul 25 10:10:08 Executing start method (:true). ]
[ 2014 Jul 25 11:06:11 Enabled. ]
[ 2014 Jul 25 11:06:11 Rereading configuration. ]
[ 2014 Jul 25 11:06:11 Executing start method (:true). ]
[ 2014 Jul 30 12:52:04 Executing stop method ("/root/blah.ksh stop"). ]
Hello... stopping
[ 2014 Jul 30 12:52:04 Method "stop" exited with status 0. ]
[ 2014 Aug  4 15:14:09 Disabled. ]
[ 2014 Aug  4 15:14:59 Enabled. ]

If multiple instances are provided with the -vL option then only the last 10 lines
of the log files are presented.  For example :

svcs -Lv svc:/system/name-service/cache:default svc:/system/fm/smtp-notify:default

svc:/system/name-service/cache:default (name service cache)
[ 2014 Jun 20 09:10:32 Executing start method ("/lib/svc/method/svc-nscd start"). ]
[ 2014 Jun 20 09:10:33 Method "start" exited with status 0. ]
[ 2014 Jul 17 08:34:25 Disabled. ]
[ 2014 Jul 17 08:34:26 Enabled. ]
[ 2014 Jul 17 08:35:07 Executing start method ("/lib/svc/method/svc-nscd start"). ]
[ 2014 Jul 17 08:35:08 Method "start" exited with status 0. ]
[ 2014 Jul 28 05:31:51 Stopping because service restarting. ]
[ 2014 Jul 28 05:31:52 Executing stop method (:kill). ]
[ 2014 Jul 28 05:31:52 Executing start method ("/lib/svc/method/svc-nscd start"). ]
[ 2014 Jul 28 05:31:52 Method "start" exited with status 0. ]

svc:/system/fm/smtp-notify:default (Solaris Email Event Notification Agent)
[ 2014 Apr 30 08:47:06 Method "start" exited with status 0. ]
[ 2014 May 24 13:12:14 Enabled. ]
[ 2014 May 24 13:22:49 Executing start method ("/usr/lib/fm/notify/smtp-notify"). ]
[ 2014 May 24 13:22:49 Method "start" exited with status 0. ]
[ 2014 Jun 20 09:09:43 Enabled. ]
[ 2014 Jun 20 09:21:58 Executing start method ("/usr/lib/fm/notify/smtp-notify"). ]
[ 2014 Jun 20 09:21:58 Method "start" exited with status 0. ]
[ 2014 Jul 17 08:34:30 Enabled. ]
[ 2014 Jul 17 08:45:47 Executing start method ("/usr/lib/fm/notify/smtp-notify"). ]
[ 2014 Jul 17 08:45:47 Method "start" exited with status 0. ]

Tuesday Apr 29, 2014

Introducing SMF Stencils

    As much as we'd like to believe that every application that runs on
Oracle Solaris 11.2 uses SMF directly to maintain configuration, the reality is
that it's a difficult or downright impossible goal for many types of software.
Be it a 3rd party service, some piece of FOSS, or software for which the
maintainers are long forgotten; some services simply must continue to use files
to store their configuration.  Does this mean managing Oracle Solaris machines
eternally dooms you to multiple configuration management tools?  It did, until

    Enter SMF Stencils.

    Simply put, SMF Stencils is a framework for converting configuration
data stored in the repository to the files used to configure particular
applications.  Using a small bit of metadata about the configuration files and
a template describing them, SMF can automatically generate and regenerate those
files at the appropriate times.  So now you can have your cake and eat it too;
your applications don't need to be rewritten to take advantage of SMF directly,
but can still reap the benefits of SMF's configuration management


    svcio(1), which lives at /lib/svc/bin/svcio, is the program which
actually handles file generation.  It reads in both the SMF repository and any
stencil files configured for a service, processes them, and writes any
resultant files to the appropriate path.  The master restarter, in turn,
invokes svcio before either the start or refresh methods for a service are run.
 This way, the configuration files for a service always reflect the latest set
of data stored in the SMF repository.  Generally speaking, it's not necessary
to know that svcio exists let alone how it works, but it's sometimes useful to
invoke it directly (for example, when testing a stencil).  The usage is as

Usage: svcio [-alux] [-f fmri] [-g group] [-i file] [-L opts]
        [-m mode] [-o file] [-O user] [-R dir] [-S dir]

        -a  process all configfile dependencies for fmri
        -f  set instance fmri (default is $SMF_FMRI)
        -g  set the group any output files are associated with
        -i  set input file (default is stdin)
        -l  list referenced properties from input file
        -L  use lofs <opts> to loopback mount the output file(s)
        -m  set output file mode (default is 644)
        -o  set output file (default is stdout)
        -O  set the owner of any output files
        -R  set root prefix for all output files
        -S  set stencil directory (default is /lib/svc/stencils)
        -u  unlink output file(s) and undo any lofs mounts
        -x  terminate svcio on any error

    The vast majority of the options are for esoteric uses that are beyond
the scope of this blog, but three of them, -a, -f, and -i, are useful for
developing a stencil.  The -f switch tells svcio which service instance to
process stencils for, and the -a option makes it process all stencils for the
specified instance.  We'll explore that in more detail shortly.  The -i, on the
other hand, tells svcio to skip any configured stencils for the service
instance and forces it to use a specific stencil file.  Without a -o to tell it
where to write the result, svcio will write the processed filed to stdout.  For
now, it's best to ignore the other options.

Making a Service Stencil-Aware

    Configuring a service to use stencils is quite simple.  All that's
required is to create a property group, with a few properties, for each
configuration file that SMF should manage.  The property group can have any
name, but must be of type 'configfile', and it is possible to have as many
property groups as the application has configuration files.  svcio understands
5 properties in this property group, 3 of which are mandatory:

    stencil        The path to the stencil file, rooted at
            /lib/svc/stencils.  svcio will use the value of this
            property to determine which stencil to read in.
    path        The path to the output file.  svcio will write the
            results of processing the stencil to this path.
    mode        The octal mode for the output file.
    user        The user to create the output file under.  This property
            is optional.
    group        The group to create the output file in.  This property
            is optional.

Our Sandbox Service

    In the interest of providing concrete examples, we'll start by creating
a service to play around in.  Let's assume we're converting a simple payroll
manager that uses an INI style file to store its configuration data.  The
manifest for such a service has been attached to this entry, but a portion of
the svcprop output for the service is provided here for reference:

    svcprop test/payroll-manager:default

    account/account_number integer 12345
    account/routing_number integer 918273645
    payee_1/address astring 1234\ 1st\ St.\ \ Mapleton,\ MA.\ 06033
    payee_1/name astring Robert\ Bruce
    payee_1/salary integer 55000
    payee_2/address astring 6N418\ County\ Lane\ 1.\ \ Cairo,\ WI.\ 60993
    payee_2/name astring Catherine\ Denninger
    payee_2/salary integer 64000
    payroll_stencil/mode integer 600
    payroll_stencil/path astring /etc/payroll.conf
    payroll_stencil/stencil astring payroll.conf.stencil

    There's some basic information here about the account to use to pay the
two payees: Robert Bruce and Catherine Denninger. Ignore the payroll_stencil
property group for the time being, which will be addressed later on.

    To follow along with these examples, just import the manifest for the
service, payroll_manager.xml, and copy the stencil, payroll.conf.stencil, to
/lib/svc/stencils.  Alternately, you can follow along with each iteration of the
stencil file to /lib/svc/stencils/payroll.conf.stencil.  The version of
payroll.conf.stencil attached here is only the final iteration.

Writing a Stencil

    The phrase "stencil file" has been tossed around quite a bit here, but
it hasn't been defined.  So what exactly is a stencil file?

    A stencil file defines how properties and property groups in the SMF
repository relate to the structure of a configuration file.  At the most basic,
stencils use property and property group names to retrieve values of properties
and write them into a file in much the same way as method tokens behave.  A
simple stencil for a simple configuration file might contain only a few property
expressions, whereas a more complicated stencil might have a huge variety of
configuration options that interact in various ways.

    We're going to construct a stencil for our payroll manager.
The configuration file contains a section for basic account information, along
with a section for each payee.  An example file would look like:

        account_number = 12345
        routing_number = 918273645

    [Robert Bruce]
        salary = 55000
        address = 1234 1st St.  Mapleton, MA. 06033

    [Catherine Denninger]
        salary = 64000
        address = 6N418 County Lane 1.  Cairo, WI. 60993

    Retreiving Property Values

    First we'll create the portion of the stencil which handles account
information.  Since this section can appear only once and has only a few
values, we can just hardcode the section into the stencil.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}


    Matching Properties and Property Groups

    Many configuration files can be fully represented using simple
substitution like the example above.  However, many other files have many
sections with duplicate information (Puppet, Samba, Kerberos, to name a few).
It's unreasonable to change the stencil for each new grouping created, so the
stencil markup language has a facility to iterate over properties and property
groups.  To keep things flexible, svcio uses regular expression matching on
property and property group names to determine which properties and property
groups to iterate over.  In our example, each payee in the system is identified
with a property group named "payee_<number>", with a unique number per payee.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}

    $%/payee_([0-9]*)/ {
        salary = $%{payee_$%1/salary}
        address = $%{payee_$%1/address}


    For those familiar with regular expression, it should be relatively
easy to figure out what's going on.  For everyone else, let's go through this
line by line.

    $%/payee_([0-9]*)/ {

    This line defines the regular expression which is matched against all
property and property group names for the instance that svcio(1) is processing
the stencil against.  The regular expression itself is the text between "$%/"
and "/ {".  It's worth noting that the regular expression is terminated with the
character sequence "/<whitespace>{", where arbitrary whitespace is allowed.
This means that it is permissible to use unescaped forward slash characters
within a regular expression, making it much simpler to match on strings that
include both property groups and properties (e.g. "$%/general/(.*)/ {" is a
perfectly legal regular expression).


    Notice the parentheses in the regular expression on the previous line.
These parentheses indicate a submatch within the total match.  Within the
braces, each submatch can be referenced using the sequence
"$%<submatch_index>".  Since there's only one submatch, it must be the first,
and is referenced by $%1.  svcio(1) will substitute any text within the
submatch and continue processing the stencil.  So if we matched on "payee_1",
this line will become "[$%{payee_1/name}]" which will, in turn, be translated
into "[Robert Bruce]".

    salary = $%{payee_$%1/salary}
    address = $%{payee_$%1/address}

    The pattern established on the previous line is continued here.


    This brace terminates the regular expression block.  For each match,
all the text between the two braces will get written to the output file.  In
our example, this means that each payee property group will be translated into
a few lines of text in the output file.

    Composing Regular Expressions

    Since our little payroll manager can't yet read e-mail, it is safe to
assume it will expand over time.  One likely place for expansion is extra
configuration in the payee sections.  With that in mind, it is probably not a
good idea to hard-code the properties printed in each payee section.  The
stencil markup language allows the composition of regular expressions to
further generalize stencils and support cases just like this one.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}

    $%/payee_([0-9]*)/ {
    $%/payee$%1/([^n].*)/ {\t$%2 = $%{payee_$%1/$%2}\n}


    Most of this version is identical to the previous version, just with
the hardcoded entries for salary and address removed in favor of another
regular expression.  Since the second regular expression is embedded in the
first, some adjustment must be done with the submatch numbers.  For embedded
regular expressions, any submatch indices take the outer expressions into
account.  In this example, the outer expression has one submatch, so the inner
expression starts counting at two.  If the outer expression had two submatches,
the inner expression would start counting at three.  This way it is possible to
use submatches from any outer regular expressions within inner expressions
(hence the $%{payee_$%1/$%2}).

    With this modification, any new properties that are added to payee
information will automatically propagate to the configuration file, provided the
name of the property does not start with the letter 'n' (this was done to
prevent the property 'name' from appearing).  Clever readers will be able to
modify this regular expression to include all property names which are not
the word 'name'.

    Putting It All Together

    Now that we have a good stencil, we can add the appropriate
configuration to the service so that the master restarter will run svcio(1)
before running a start method or refreshing.  The first step is to create a
property group with the type set to "configfile", which we'll call
"payroll_stencil".  After that, the following properties are necessary:

    payroll_stencil/stencil = payroll.conf.stencil
    payroll_stencil/path = /etc/payroll.conf
    payroll_stencil/mode = 600

    Using those properties, svcio(1) will read the stencil file from
/lib/svc/stencils/payroll.conf.stencil and use it generate a file at
/etc/payroll.conf with the mode set to 600.

    And that's it!  Whenever the configuration has to be changed, just
refresh the instance (assuming it's online) or enable the instance and go check
/etc/payroll.conf.  It will have changed to reflect the updated configuration.



Friday Feb 03, 2012

Changes to svccfg import and delete

The behavior of svccfg import and svccfg delete fmri has changed in S11 if the manifests are in SMF's standard locations. The standard locations are /lib/svc/manifest and /var/svc/manifest with /lib/svc/manifest being the preferred location. If your manifest is stored under one of these two directories, you shouldn't be using svccfg import at all. You should only use svccfg delete fmri when you are prohibited from removing the manifest for some reason and still want to remove the service from the system. Instead, the preferred action is:

        # svcadm restart manifest-import


The reason is that in S11, SMF keeps the repository in sync with the files in the standard locations, and the manifest-import service is the mechanism for maintaining this synchronization. So instead of using svccfg import copy your manifest to a standard location and type:

        # svcadm restart manifest-import
Instead of using svccfg delete remove your manifest from its location and restart manifest-import.


In each case manifest-import will detect any changes in the standard directories and update the repository accordingly. Note that the manifest-import service runs asynchronously from the svcadm command, so it may take a short amount of time for the changes to take effect.

Also the manifest-import service not only detects file additions and removals. It also detects changes to manifests and profiles. If you are a service provider, this gives you an upgrade path if your manifest changes. Simply deposit your new manifest over the old one and make sure that manifest-import is restarted. Restarting of manifest-import is usually handled by the packaging service.

Let's look at some examples. First, let's get the manifest for our new service imported.

# cp mysvc.xml /lib/svc/manifest/site
# svcadm restart manifest-import
# svcs mysvc
STATE          STIME    FMRI
online         15:19:41 svc:/mysvc:default
Now delete the service:
# rm /lib/svc/manifest/site/mysvc.xml 
# svcadm restart manifest-import
# svcs mysvc
svcs: Pattern 'mysvc' doesn't match any instances
STATE          STIME    FMRI


Now let's look at what happens if you stray from this advice and use svccfg delete. First, reinstall the manifest just as we did before.

# cp mysvc.xml /lib/svc/manifest/site
# svcadm restart manifest-import
# svcs mysvc
STATE          STIME    FMRI
online         15:34:41 svc:/mysvc:default
Now the fun begins.
# svccfg delete -f svc:/mysvc
# svcs mysvc
svcs: Pattern 'mysvc' doesn't match any instances
STATE          STIME    FMRI
It looks as if the service has been removed from the repository, but it really hasn't been. Since the manifest file is still on the file system, the service is merely masked in the repository. This can lead to confusion. Even if you modify the manifest and restart manifest-import, svcs will not find the service. This is because the masking is done at the administrative layer (see Sean Wilcox's discussion of layers). The masking is not removed by changing the manifest, although manifest-import will record the changes in the repository.


How can we find a masked service.

# svccfg listcust -M | grep svc:/mysvc
svc:/mysvc manifest MASKED
svc:/mysvc:default manifest MASKED
The first line of output shows that the service is masked. Masking a service also masks it instances which is why we see the second line.


So if you accidentally mask a service, how can you unmask it? We enter svccfg interactive mode, select the service and then use the delcust command.

# svccfg
svc:> select mysvc
svc:/mysvc> delcust
 Deleting customizations for service: mysvc
svc:/mysvc> quit
# svcs mysvc
STATE          STIME    FMRI
online         15:50:46 svc:/mysvc:default
The svcs command shows that the service is unmasked.


Standard Locations (why?)

The manifest-import service manages importing of manifests that are delivered as part of a package for an application.  This instantiates the service and its instances on the system.  The manifest-import service will then manage re-importing those manifests if they are modified/upgraded in some way.

Also, the manifest-import service manages the application of profiles that are in the standard location for profiles of /etc/svc/profile/site.  If these profiles change or are removed then the support for them can be removed from the services on the system.

Finally once the application has served its purpose and the delivering package is removed from the system along with the removal of the manifest the service will then be cleaned up by the manifest-import service.

With that the manifest-import service needs a well known place to be able to find the manifests for a service, and be able to find those manifests under a certain set of rules.  One, the cleanup side of the service needs to be able to know for sure that a manifest is removed and not that a location is simply temporarily unavailable.

Before the Solaris 11, the manifests were located in /var/svc/manifest.  But this location might or might not be available at boot because /var can be a separate filesystem, that is not mounted early in boot.  With the Solaris 11, the manifests were moved to /lib/svc/manifest so that the manifests would be available at the beginning of system boot. Therefore, manifests are to no longer be placed in /var/svc/manifest as it is strictly supported for backwards compatibility only.

So with this standard location that is guaranteed to be available at boot SMF can now make sure that changes and upgrades to manifests are imported before any services are started.  This way services that need to start early in the boot cycle (even before /var might be mounted, if the manifest is in /lib/svc/manifest) will be guaranteed to start with their new property values.

Also, if the manifest is removed from the system, there is a chance for the service to be removed from the system before it attempts to start and does not find the service binaries and/or other files that may be required for the service to run.

Put your manifests and profiles in the standard location and let SMF manage your import, apply and ultimately the cleanup of your services and instances.

So in summary the benefits are :

1. manifests can be imported early in boot before any services are started that might use the information from the manifests.
2. upgrades of manifests and profiles can be done during this early boot phase as well, so that services get the new information before they start.
3. if manifests are removed, then the manifest-import service that manages these can know for sure that the manifests are removed and clean up the services.
4. manifests in a standard location are the base layer for services and their property group and property sets.

Wednesday Nov 09, 2011

Introducing SMF Layers

SMF Layers, meta data for your services.  With S11 we added a new feature that will allow for better tracking of where services, instances, property groups and properties originate and how they may change over there life span. We store a layer at which the entity exists.  At this time there are 4 layers at which an entity can exist.  These layers are based on the location of its delivering or contributing data.

The layers are :

system profile
site profile

Manifest layer entities are delivered or contributed from a standard location for manifests, /lib/svc/manifest or /var/svc/manifest.

System profile layer entities are delivered in the system profile.  A profile that is delivered by Solaris as part of the operating system itself.  At this time this is the generic.xml manifest and a select few other files included into that file.

Site profile layer entities are delivered in a profile located under the /etc/svc/profile/site directory or a subdirectory created under this directory.

Everything else falls under the administrative layer.  Anything delivered outside of the standard locations for manifests and profiles.  As well as any command line creation of entities whether it be through svccfg or the libscf(3LIB) interfaces.

Starting at the manifest layer (the lowest) each of the next layers will override the previous layer.  Although, each layer's data is stored in the repository so that, if one of the higher layers is removed, it will reveal any underlying layers.  This makes upgrading but not overriding higher layer changes a simple action of putting the new files in place and letting the entities be
added to their appropriate layer.

One of the primary concepts we kept to when designing and implementing this project, was that the repository must represent the filesystem in the standard locations, and all other changes are administrative customizations to the system.

So how do Layers benefit us?  Layers allow us to properly upgrade service configurations while preserving customizations.  They provide better observability into service configurations, and the ability to undo customizations

Solaris Service Management Facility information, tips and tricks.


Top Tags
« April 2015