News, tips, partners, and perspectives for the Oracle Solaris operating system

  • April 29, 2014

Introducing SMF Stencils

    As much as we'd like to believe that every application that runs on
Oracle Solaris 11.2 uses SMF directly to maintain configuration, the reality is
that it's a difficult or downright impossible goal for many types of software.
Be it a 3rd party service, some piece of FOSS, or software for which the
maintainers are long forgotten; some services simply must continue to use files
to store their configuration.  Does this mean managing Oracle Solaris machines
eternally dooms you to multiple configuration management tools?  It did, until

    Enter SMF Stencils.

    Simply put, SMF Stencils is a framework for converting configuration
data stored in the repository to the files used to configure particular
applications.  Using a small bit of metadata about the configuration files and
a template describing them, SMF can automatically generate and regenerate those
files at the appropriate times.  So now you can have your cake and eat it too;
your applications don't need to be rewritten to take advantage of SMF directly,
but can still reap the benefits of SMF's configuration management


    svcio(1), which lives at /lib/svc/bin/svcio, is the program which
actually handles file generation.  It reads in both the SMF repository and any
stencil files configured for a service, processes them, and writes any
resultant files to the appropriate path.  The master restarter, in turn,
invokes svcio before either the start or refresh methods for a service are run.
 This way, the configuration files for a service always reflect the latest set
of data stored in the SMF repository.  Generally speaking, it's not necessary
to know that svcio exists let alone how it works, but it's sometimes useful to
invoke it directly (for example, when testing a stencil).  The usage is as

Usage: svcio [-alux] [-f fmri] [-g group] [-i file] [-L opts]
        [-m mode] [-o file] [-O user] [-R dir] [-S dir]

        -a  process all configfile dependencies for fmri
        -f  set instance fmri (default is $SMF_FMRI)
        -g  set the group any output files are associated with
        -i  set input file (default is stdin)
        -l  list referenced properties from input file
        -L  use lofs <opts> to loopback mount the output file(s)
        -m  set output file mode (default is 644)
        -o  set output file (default is stdout)
        -O  set the owner of any output files
        -R  set root prefix for all output files
        -S  set stencil directory (default is /lib/svc/stencils)
        -u  unlink output file(s) and undo any lofs mounts
        -x  terminate svcio on any error

    The vast majority of the options are for esoteric uses that are beyond
the scope of this blog, but three of them, -a, -f, and -i, are useful for
developing a stencil.  The -f switch tells svcio which service instance to
process stencils for, and the -a option makes it process all stencils for the
specified instance.  We'll explore that in more detail shortly.  The -i, on the
other hand, tells svcio to skip any configured stencils for the service
instance and forces it to use a specific stencil file.  Without a -o to tell it
where to write the result, svcio will write the processed filed to stdout.  For
now, it's best to ignore the other options.

Making a Service Stencil-Aware

    Configuring a service to use stencils is quite simple.  All that's
required is to create a property group, with a few properties, for each
configuration file that SMF should manage.  The property group can have any
name, but must be of type 'configfile', and it is possible to have as many
property groups as the application has configuration files.  svcio understands
5 properties in this property group, 3 of which are mandatory:

    stencil        The path to the stencil file, rooted at
            /lib/svc/stencils.  svcio will use the value of this
            property to determine which stencil to read in.
    path        The path to the output file.  svcio will write the
            results of processing the stencil to this path.
    mode        The octal mode for the output file.
    user        The user to create the output file under.  This property
            is optional.
    group        The group to create the output file in.  This property
            is optional.

Our Sandbox Service

    In the interest of providing concrete examples, we'll start by creating
a service to play around in.  Let's assume we're converting a simple payroll
manager that uses an INI style file to store its configuration data.  The
manifest for such a service has been attached to this entry, but a portion of
the svcprop output for the service is provided here for reference:

    svcprop test/payroll-manager:default

    account/account_number integer 12345
    account/routing_number integer 918273645
    payee_1/address astring 1234\ 1st\ St.\ \ Mapleton,\ MA.\ 06033
    payee_1/name astring Robert\ Bruce
    payee_1/salary integer 55000
    payee_2/address astring 6N418\ County\ Lane\ 1.\ \ Cairo,\ WI.\ 60993
    payee_2/name astring Catherine\ Denninger
    payee_2/salary integer 64000
    payroll_stencil/mode integer 600
    payroll_stencil/path astring /etc/payroll.conf
    payroll_stencil/stencil astring payroll.conf.stencil

    There's some basic information here about the account to use to pay the
two payees: Robert Bruce and Catherine Denninger. Ignore the payroll_stencil
property group for the time being, which will be addressed later on.

    To follow along with these examples, just import the manifest for the
service, payroll_manager.xml, and copy the stencil, payroll.conf.stencil, to
/lib/svc/stencils.  Alternately, you can follow along with each iteration of the
stencil file to /lib/svc/stencils/payroll.conf.stencil.  The version of
payroll.conf.stencil attached here is only the final iteration.

Writing a Stencil

    The phrase "stencil file" has been tossed around quite a bit here, but
it hasn't been defined.  So what exactly is a stencil file?

    A stencil file defines how properties and property groups in the SMF
repository relate to the structure of a configuration file.  At the most basic,
stencils use property and property group names to retrieve values of properties
and write them into a file in much the same way as method tokens behave.  A
simple stencil for a simple configuration file might contain only a few property
expressions, whereas a more complicated stencil might have a huge variety of
configuration options that interact in various ways.

    We're going to construct a stencil for our payroll manager.
The configuration file contains a section for basic account information, along
with a section for each payee.  An example file would look like:

        account_number = 12345
        routing_number = 918273645

    [Robert Bruce]
        salary = 55000
        address = 1234 1st St.  Mapleton, MA. 06033

    [Catherine Denninger]
        salary = 64000
        address = 6N418 County Lane 1.  Cairo, WI. 60993

    Retreiving Property Values

    First we'll create the portion of the stencil which handles account
information.  Since this section can appear only once and has only a few
values, we can just hardcode the section into the stencil.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}


    Matching Properties and Property Groups

    Many configuration files can be fully represented using simple
substitution like the example above.  However, many other files have many
sections with duplicate information (Puppet, Samba, Kerberos, to name a few).
It's unreasonable to change the stencil for each new grouping created, so the
stencil markup language has a facility to iterate over properties and property
groups.  To keep things flexible, svcio uses regular expression matching on
property and property group names to determine which properties and property
groups to iterate over.  In our example, each payee in the system is identified
with a property group named "payee_<number>", with a unique number per payee.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}

    $%/payee_([0-9]*)/ {
        salary = $%{payee_$%1/salary}
        address = $%{payee_$%1/address}


    For those familiar with regular expression, it should be relatively
easy to figure out what's going on.  For everyone else, let's go through this
line by line.

    $%/payee_([0-9]*)/ {

    This line defines the regular expression which is matched against all
property and property group names for the instance that svcio(1) is processing
the stencil against.  The regular expression itself is the text between "$%/"
and "/ {".  It's worth noting that the regular expression is terminated with the
character sequence "/<whitespace>{", where arbitrary whitespace is allowed.
This means that it is permissible to use unescaped forward slash characters
within a regular expression, making it much simpler to match on strings that
include both property groups and properties (e.g. "$%/general/(.*)/ {" is a
perfectly legal regular expression).


    Notice the parentheses in the regular expression on the previous line.
These parentheses indicate a submatch within the total match.  Within the
braces, each submatch can be referenced using the sequence
"$%<submatch_index>".  Since there's only one submatch, it must be the first,
and is referenced by $%1.  svcio(1) will substitute any text within the
submatch and continue processing the stencil.  So if we matched on "payee_1",
this line will become "[$%{payee_1/name}]" which will, in turn, be translated
into "[Robert Bruce]".

    salary = $%{payee_$%1/salary}
    address = $%{payee_$%1/address}

    The pattern established on the previous line is continued here.


    This brace terminates the regular expression block.  For each match,
all the text between the two braces will get written to the output file.  In
our example, this means that each payee property group will be translated into
a few lines of text in the output file.

    Composing Regular Expressions

    Since our little payroll manager can't yet read e-mail, it is safe to
assume it will expand over time.  One likely place for expansion is extra
configuration in the payee sections.  With that in mind, it is probably not a
good idea to hard-code the properties printed in each payee section.  The
stencil markup language allows the composition of regular expressions to
further generalize stencils and support cases just like this one.


        account_number = $%{account/account_number}
        routing_number = $%{account/routing_number}

    $%/payee_([0-9]*)/ {
    $%/payee$%1/([^n].*)/ {\t$%2 = $%{payee_$%1/$%2}\n}


    Most of this version is identical to the previous version, just with
the hardcoded entries for salary and address removed in favor of another
regular expression.  Since the second regular expression is embedded in the
first, some adjustment must be done with the submatch numbers.  For embedded
regular expressions, any submatch indices take the outer expressions into
account.  In this example, the outer expression has one submatch, so the inner
expression starts counting at two.  If the outer expression had two submatches,
the inner expression would start counting at three.  This way it is possible to
use submatches from any outer regular expressions within inner expressions
(hence the $%{payee_$%1/$%2}).

    With this modification, any new properties that are added to payee
information will automatically propagate to the configuration file, provided the
name of the property does not start with the letter 'n' (this was done to
prevent the property 'name' from appearing).  Clever readers will be able to
modify this regular expression to include all property names which are not
the word 'name'.

    Putting It All Together

    Now that we have a good stencil, we can add the appropriate
configuration to the service so that the master restarter will run svcio(1)
before running a start method or refreshing.  The first step is to create a
property group with the type set to "configfile", which we'll call
"payroll_stencil".  After that, the following properties are necessary:

    payroll_stencil/stencil = payroll.conf.stencil
    payroll_stencil/path = /etc/payroll.conf
    payroll_stencil/mode = 600

    Using those properties, svcio(1) will read the stencil file from
/lib/svc/stencils/payroll.conf.stencil and use it generate a file at
/etc/payroll.conf with the mode set to 600.

    And that's it!  Whenever the configuration has to be changed, just
refresh the instance (assuming it's online) or enable the instance and go check
/etc/payroll.conf.  It will have changed to reflect the updated configuration.



Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha

Integrated Cloud Applications & Platform Services