Tips on deploying and managing Oracle Solaris, especially in clouds

Building an OpenStack Cloud for Solaris Engineering, Part 3

Dave Miner
Sr. Principal Software Engineer

At the end of Part 2, we built the infrastructure needed to deploy the undercloud systems into our network environment.  However, there's more configuration needed on these systems than we can completely express via Automated Installation, and there's also the issue of how to effectively maintain the undercloud systems.  We're only running a half dozen initially, but expect to add many more as we grow, and even at this scale it's still too much work, with too high a probability of mistakes, to do things by hand on each system.  That's where a configuration management system such as Puppet shows its value, providing us the ability to define a desired state for many aspects of many systems and have Puppet ensure that state is maintained.  My team did a lot of work to include Puppet in Solaris 11.2 and extend it to manage most of the major subsystems in Solaris, so the OpenStack cloud deployment was a great opportunity to start working with another shiny new toy.

Configuring the Puppet Master

One feature of the Puppet integration with Solaris is that the Puppet configuration is expressed in SMF, and then translated by the new SMF Stencils feature to settings in the usual /etc/puppet/puppet.conf file.  This makes it possible to configure Puppet using SMF profiles at deployment time, and the examples in Part 2 showed this for the clients.  For the master, we apply the profile below:

<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
This profile configures the Puppet master
<service_bundle type="profile" name="puppet">
<service version="1" type="service" name="application/puppet">
<instance enabled="true" name="master">
<property_group type="application" name="config">
<propval name="server" value="puppetmaster.example.com"/>
<propval name="autosign" value="/etc/puppet/autosign.conf"/>

The interesting setting is the autosign configuration, which allows new clients to have their certificates automatically signed and accepted by the Puppet master.  This isn't strictly necessary, but makes operation a little easier when you have a reasonably secure network and you're not handing out any sensitive configuration via Puppet.  We use an autosign.conf that looks something like:


This means that we're accepting any system that identifies as being in the example.com domain.  The main pain with autosigning is that if you reinstall any of the systems and you're using self-generated certificates on the clients, you need to clean out the old certificate before the new one will be accepted; this means issuing a command on the master like:

# puppet cert clean client.example.com

There are lots of options in Puppet related to certificates and more sophisticated ways to manage them, but this is what we're doing for now.  We have filed some enhancement requests to implement ways of integrating Puppet client certificate delivery and signing with Automated Installation, which would make using the two together much more convenient.

Writing the Puppet Manifests

Next, we implemented a small Mercurial source repository to store the Puppet manifests and modules.  Using a source control system with Puppet is a highly recommended practice, and Mercurial happens to be the one we use for Solaris development, so it's natural for us in this case.  We configure /etc/puppet on the Puppet master as a child repository of the main Mercurial repository, so when we have new configuration to apply it's first checked into the main repository and then pulled into Puppet via hg pull -u, then automatically applied as each client polls the master.  Our repository presently contains the following:


An example tar file with all of the above is available for download.

The site manifest  starts with:

include ntp
include nameservice

The ntp module is the canonical example of Puppet, and is really important for the OpenStack undercloud, as it's necessary for the various nodes to have a consistent view of time in order for the security certificates issued by Keystone to be validated properly.  I'll describe the nameservice module a little later in this post.

Since most of our nodes are configured identically, we can use a default node definition to configure them.  The main piece is configuring Datalink Multipathing (DLMP), which provides us additional bandwidth and higher availability than a single link.  We can't yet configure this using SMF, so the Puppet manifest:

  • Figures out the IP address the system is using with some embedded Ruby
  • Removes the net0 link and creates a link aggregation from net0 and net1
  • Enables active probing on the link aggregation, so that it can detect upstream failures on the switches that don't affect link state signaling (which is also used, and is the only means unless probing is enabled)
  • Configures an IP interface and the same address on the new aggregation link
  • Restricts Ethernet autonegotiation to 1 Gb to work around issues we have with these systems and the switches/cabling we're using the in the lab; without this, we get 100 Mb speeds negotiated about 50% of the time, and that kills performance.
You'll note several uses of the require and before statements to ensure the rules are applied in the proper order, as we need to tear down the net0 IP interface before it can be moved into the aggregation, and the aggregation needs to be configured before the IP objects on top of it.
node default {

$myip = inline_template("<% _erbout.concat(Resolv::DNS.open.getaddress('$fqdn').to_s) %>")

# Force link speed negotiation to be at least 1 Gb

link_properties { "net0":

ensure => present,

properties => { en_100fdx_cap => "0" },


link_properties { "net1":

ensure => present,

properties => { en_100fdx_cap => "0" },


link_aggregation { "aggr0" :

ensure => present,

lower_links => [ 'net0', 'net1' ],

mode => "dlmp",


link_properties { "aggr0":

ensure => present,

require => Link_aggregation['aggr0'],

properties => { probe-ip => "+" },


ip_interface { "aggr0" :

ensure => present,

require => Link_aggregation['aggr0'],


ip_interface { "net0":

ensure => absent,

before => Link_aggregation['aggr0'],


address_object { "net0":

ensure => absent,

before => Ip_interface['net0'],


address_object { 'aggr0/v4':

require => Ip_interface['aggr0'],

ensure => present,

address => "${myip}/24",

address_type => "static",

enable => "true",


The controller node declaration includes all of the above functionality, but also adds these elements to keep rabbitmq running and install the mysql database.

    service { "application/rabbitmq" :
ensure => running,
package { "database/mysql-55":
ensure => installed,

The database installation could have been part of the AI derived manifest as well, but it works just as well here and it's convenient to do it this way when I'm setting up staging systems to test builds before we upgrade.

The nameservice Puppet module is shown below.  It's handling both nameservice and RBAC (Role-based Access Control) configuration:

class nameservice {
dns { "openstack_dns":
search => [ 'example.com' ],
nameserver => [ ', '' ],
service { "dns/client":
ensure => running,
svccfg { "domainname":
ensure => present,
fmri => "svc:/network/nis/domain",
property => "config/domainname",
type => "hostname",
value => "example.com",
# nameservice switch
nsswitch { "dns + ldap":
default => "files",
host => "files dns",
password => "files ldap",
group => "files ldap",
automount => "files ldap",
netgroup => "ldap",
# Set user_attr for administrative accounts
file { "user_attr" :
path => "/etc/user_attr.d/site-openstack",
owner => "root",
group => "sys",
mode => 644,
source => "puppet:///modules/nameservice/user_attr",
# Configure zlogin access
file { "site-zlogin" :
path => "/etc/security/prof_attr.d/site-zlogin",
owner => "root",
group => "sys",
mode => 644,
source => "puppet:///modules/nameservice/prof_attr-zlogin",
file { "zlogin-exec" :
path => "/etc/security/exec_attr.d/site-zlogin",
owner => "root",
group => "sys",
mode => 644,
source => "puppet:///modules/nameservice/exec_attr-zlogin",
file { "policy.conf" :
path => "/etc/security/policy.conf",
owner => "root",
group => "sys",
mode => 644,
source => "puppet:///modules/nameservice/policy.conf",

You may notice that the nameservice configuration here is exactly the same as what we provided in the SMF profile in part 2.  We include it here because it's configuration we anticipate changing someday and we won't want to re-deploy the nodes.  There are ways we could prevent the duplication, but we didn't have time to spend on it right now and it also demonstrates that you could use a completely different configuration in operation than at deployment/staging time.

What's with the RBAC configuration?

The RBAC configuration is doing two things, the first being configuring the user accounts of the cloud administrators for administrative access on the cloud nodes.  The user_attr file we're distributing confers the System Adminstrator and OpenStack Management profiles, as well as access to the root role (oadmin is just an example account in this case):

oadmin::::profiles=System Administrator,OpenStack Management;roles=root

As we add administrators, I just need to add entries for them to the above file and they get the required access to all of the nodes.  Note that this doesn't directly provide administrative access to OpenStack's CLI's or its Dashboard, that's configured within OpenStack.

A limitation of the OpenStack software we include in Solaris 11.2 is that we don't provide the ability to connect to the guest instance consoles, an important feature that's being worked on.  The zlogin User profile is something I created to work around this problem and allow our cloud users to get access to the consoles, as this is often needed in Solaris development and testing.  First, the profile is defined by a prof_attr file with the entry:

zlogin User:::Use zlogin:auths=solaris.zone.manage

We also need an exec_attr file to ensure that zlogin is run with the needed uid and privileges:

 zlogin User:solaris:cmd:RO::/usr/sbin/zlogin:euid=0;privs=ALL

Finally, we modify the RBAC policy file so that all users are assigned to the zlogin User profile:

PROFS_GRANTED=zlogin User,Basic Solaris User

The result of all this is that a user can obtain access to their specific OpenStack guest instance via login to the compute node on which the guest is running, and runing a command such as:

$ pfexec zlogin -C instance-0000abcd

At this point we have the undercloud nodes fully configured to support our OpenStack deployment.  In part 4, we'll look at the scripts used to configure OpenStack itself.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha