Thursday Aug 14, 2014

VXLAN in Solaris 11.2

What is a VXLAN?

VXLAN, or Virtual eXtensible LAN, is essentially a tunneling mechanism used to provide isolated virtual Layer 2 (L2) segments that can span multiple physical L2 segments. Since it is a tunneling mechanism it uses IP (IPv4 or IPv6) as its underlying network which means we can have isolated virtual L2 segments over networks connected by IP. This allows Virtual Machines (VM) to be in the same L2 segment even if they  are located on systems that are in different physical networks. Some of the benefits of VXLAN include:

  • Better use of resources, i.e. VMs can be provisioned on systems, that span different geographies, based on system load.
  • VMs can be moved across systems without having to reconfigure the underlying physical network.
  • Fewer MAC address collision issues, i.e. MAC address may collide as long as they are in different VXLAN segments.
Isolated L2 segments can be supported by existing mechanisms such as VLANs, but VLANs don't scale; the number of VLANs are limited to 4094 (0 and 1 are reserved), but VXLAN can provide upto 16 million isolated L2 networks.

Additional details, including protocol working, can be found in the VXLAN draft IETF RFC. Note that Solaris uses the IANA specified UDP port number of 4789 for VXLAN. 

The following is a quick primer on administering VXLAN in Solaris 11.2 using the Solaris administrative utility dladm(1m). Solaris Elastic Virtual Switch (EVS) can be used to manage VXLAN deployment automatically in a cloud environment - this will be the subject of a  future discussion.

The following illustrates how VXLANs are created on Solaris:

where IPx is an IP address (IPv4 or IPv6) and VNIs y and z are different VXLAN segments. VM1, VM2 and VM3 are guests with interfaces configured on VXLAN segments y and z. vxlan1 and vxlan2 are VXLAN links, represented by a new class called VXLAN.

Creating VXLANs

To begin with we need to create  VXLAN links in the segments that we want to use  for guests - let's assume we want to create segments 100 and 101. Additionally, we also want to create the VXLAN links on IP (remember VXLANs are overlay over IP networks), so we need the IP address over which we want to create the VXLAN links - let's assume our endpoint on this system is (in the following example this IP address resides on net4).

# ipadm show-addr net4                                      
ADDROBJ           TYPE     STATE        ADDR
net4/v4                 static        ok 

Create VXLAN segments 100 and 101 on this IP address.

# dladm create-vxlan -p addr=,vni=100 vxlan1 
# dladm create-vxlan -p addr=,vni=101 vxlan2    


  • In the above example we explicitly provide the IP address, however, you could also:
    • provide a prefix and prefixlen to use an IP address that matches it, e.g:
# dladm create-vxlan -p addr=,vni=100 vxlan1
    • provide an interface (say net4 in our case) to pick an active address on that interface, e.g:
# dladm create-vxlan -p interface=net4,vni=100 vxlan1
(you can't provide interface and addr together)

  • VXLAN links can be created on an IP address over any interface, including IPoIB link, except IPMP, loopback or VNI (Virtual Network Interface).
  • The IP address may belong to a VLAN segment.

Displaying VXLANs

Check if we have our VXLAN links:

# dladm show-vxlan                                          
LINK                ADDR                     VNI   MGROUP
vxlan1                   100
vxlan2                   101

One thing  we haven't talked about so far is the MGROUP. Recall from the RFC that VXLAN links use IP multicast for broadcast. So, we can assign a multicast address to each  VXLAN segment that we create. If we don't specify a multicast address, we assign the all-host multicast address (or all nodes for IPv6) to the VXLAN segments. In the above case since we didn't specify the multicast address both vxlan1 and vxlan2 will use the all-host multicast address.

The VXLAN links created, vxlan1 and vxlan2, are just like other datalinks (physical, VNIC, VLAN, etc.) and can be displayed using 

# dladm show-link
LINK                CLASS     MTU    STATE    OVER
vxlan1              vxlan     1440         up            --
vxlan2              vxlan     1440         up            --

The STATE reflects that state of the VXLAN links which is based on the status of the IP address ( in this case). Note that the MTU is reduced because of the VXLAN encapsulation for each packet, on this VXLAN link.

Now that we have our VXLAN links, we can create Virtual Links (VNICs) over these  VXLAN links. Note, the VXLAN links themselves not active links, i.e. you can't plumb IP address or create Flows on them, but they can be snooped.

# dladm create-vnic  -l vxlan1 vnic1                    
# dladm create-vnic  -l vxlan1 vnic2    
# dladm create-vnic  -l vxlan2 vnic3            
# dladm create-vnic  -l vxlan2 vnic4  

# dladm show-vnic                                           
LINK                OVER              SPEED  MACADDRESS        MACADDRTYPE VIDS
vnic1               vxlan1            10000     2:8:20:d9:df:5f            random                   0
vnic2               vxlan1            10000     2:8:20:72:9a:70          random                   0
vnic3               vxlan2            10000     2:8:20:19:c7:14          random                   0
vnic4               vxlan2            10000     2:8:20:88:98:6d         random                    0

You can see from the above that the process of creating a VNIC on a VXLAN link  is no different from creating one any other link  such as physical, aggregation, etherstub etc.  This means that the VNICs created may belong to a VLAN and properties (such as maxbw and priority) can be set on them.

Once created, these VNICs can be assiged explicitly to Solaris zones. Alternatively, the VXLAN links can be set as the lower-link for configuring anet (automatic VNIC) links in Solaris Zones.

For Logical Domains on SPARC, the virtual switch (add-vsw) can be created on the VXLAN device which means the vnets created on the virtual switch will be part of the VXLAN segment.

Deleting VXLANs

A VXLAN can be deleted once all the VNICs over the VXLAN links have been deleted. Thus in our case:

# dladm delete-vnic vnic1   
# dladm delete-vnic vnic2 
# dladm delete-vnic vnic3     
# dladm delete-vnic vnic4  

# dladm delete-vxlan vxlan1
# dladm delete-vxlan vxlan2  

Additional Notes:
  • VXLAN for Solaris Kernel zone and LDom guests are not supported with direct I/O.
  • Hardware capabilities such as checksum and LSO are not available for the encapsulated (inner) packet.
  • Some earlier implementations (e.g. Linux) might use a pre-IANA assigned port number. If so, such implementations might have to be configured to use the IANA port number to interoperate with Solaris VXLAN. 
  • IP multicast must be available in the underlying network and if communicating  across different IP subnets, multicast routing should be available as well.
  • Modifying properties (IP address, multicast address or VNI) on a VXLAN link is currently not supported; you'd have to delete the VXLAN and re-create it.

Tuesday Jul 29, 2014

DTrace improvements in Oracle Solaris 11.2

There have been a few improvements to DTrace in Solaris 11.2.


DTrace has quantize() and lquantize() aggregating actions to give you, respectively, a power-of-two distribution and a linear distribution of data points that you're interested in.  While these are both useful, there may be instances in which you want to examine events whose latencies span multiple orders of magnitude, but for which you want relatively fine-grained information about the data within each order of magnitude.  With quantize(), there's likely to be insufficient detail. You could use lquantize(), but you'd need multiple aggregations to cover the multiple orders of magnitude.

In 11.2, we have added a log-linear quantize aggregation, llquantize().  This aggregating action allows you to specify a base and a range of exponents for the data, but it also allows you to specify a number of steps (or buckets, if you will) per order of magnitude.  For example, the following line will create an aggregation covering the values from 103 through 106 - 1 with 10 steps per order of magnitude.  (The buckets for 105 will include the values from 105 through 106 - 1.):

@ = llquantize(foo, 10, 3, 5, 20);

We can use this in a script to examine system call latencies:

                self->ts = timestamp;

        / self->ts /
                @ = llquantize(timestamp - self->ts, 10, 3, 5, 10);
                self->ts = 0;

Because the timestamp is measured in nanoseconds, this script reports system call latencies in the microseconds range.  Here's sample output from this script:

           value  ------------- Distribution ------------- count
          < 1000 |@@@@@@                                   12899
            1000 |@@@@@@@@@@@@                             26357
            2000 |@                                        3202
            3000 |@                                        1869
            4000 |@                                        2110
            5000 |@@                                       4716
            6000 |@@                                       3998
            7000 |@                                        1617
            8000 |@@                                       4924
            9000 |@                                        2515
           10000 |@@@@@@@                                  15307
           20000 |@                                        2240
           30000 |@                                        1327
           40000 |@                                        1369
           50000 |                                         990
           60000 |                                         1057
           70000 |                                         631
           80000 |                                         453
           90000 |                                         434
          100000 |@                                        1570
          200000 |                                         228
          300000 |                                         45
          400000 |                                         59
          500000 |                                         60
          600000 |                                         52
          700000 |                                         30
          800000 |                                         22
          900000 |                                         17
      >= 1000000 |                                         513


When DTrace was first conceived, 100 CPUs was a large machine.  Now, the largest machines contain over 1,000 CPUs, and the original DTrace architecture is starting to show its age.  dtrace(1M) (or specifically, libdtrace(3LIB)) was originally written to process data from the CPUs on a server in a single thread.  Unfortunately, dtrace(1M) was unable to to keep up on newer, larger servers with just a single thread.

In Solaris 11.2, we've modified libdtrace to perform this task with multiple threads.  On x86 servers, dtrace(1M) will use one thread per 8 CPUs.  On SPARC servers, it will use one thread per 16 CPUs.  We've also included an option, nworkers, to allow you to request a specific number of threads.

What does this mean for you?  The main benefit of doing this is that dtrace(1M) will actually be able to keep up on larger servers.  During testing on a 256-CPU system, generating 6,000 records per second per CPU, we were seeing hundreds of aggregation drops per second per CPU without this multi-threading framework.  Running the same test with a multi-threaded dtrace(1M), we saw no aggregation drops.

errexit option

Another minor enhancement in Solaris 11.2 is the errexit option.  This option causes dtrace(1M) to exit when it first hits an error.

As a trivial example, consider the following script:

        this->i = 0;
        this->j = 5 / this->i;

If run normally, this script would run until termination reporting an error once per second:

# dtrace -q -s divide-by-zero.d
dtrace: error on enabled probe ID 1 (ID 5148: profile:::tick-1s): divide-by-zero in action #2 at DIF offset 2
dtrace: error on enabled probe ID 1 (ID 5148: profile:::tick-1s): divide-by-zero in action #2 at DIF offset 2
dtrace: error on enabled probe ID 1 (ID 5148: profile:::tick-1s): divide-by-zero in action #2 at DIF offset 2


When run with the errexit option, the script terminates after the first error:

# dtrace -x errexit -q -s divide-by-zero.d
dtrace: error on enabled probe ID 1 (ID 5148: profile:::tick-1s): divide-by-zero in action #2 at DIF offset 2


tracemem() enhancements

We've added an optional third argument to tracemem().  Where the second argument specifies how many bytes of memory to trace, the optional third argument specifies how many bytes of memory to display.  This can be useful in cases where the size of the data you care about is variable.  (The DTrace architecture requires that you trace a constant amount of data, but in some cases, the amount of data you're interested in is variable.  What you would currently do is trace enough to capture what you're interested in and ignore the rest.  This optional argument lets dtrace(1M) ignore the garbage for you.)

As an example, consider tracing the beginning of an SSH connection, as seen from the server side.  (This is a simplified example, so we'll just look at what the server writes.):

/execname == "sshd"/
        tracemem(copyin(arg1, arg2), 1024, arg2);

While running this script and opening a connection from a remote server, we see this:

 CPU     ID                    FUNCTION:NAME
   0   5900                      write:entry
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 53 53 48 2d 32 2e 30 2d 53 75 6e 5f 53 53 48 5f  SSH-2.0-Sun_SSH_
        10: 32 2e 32 0a                                      2.2.

   0   5900                      write:entry
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 25 54 00 00                                      %T..

   1   5900                      write:entry
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 90 39 c4 53 d7 82 01 00 25 54 00 00              .9.S....%T..

   1   5900                      write:entry
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 90 39 c4 53 49 84 01 00 25 54 00 00              .9.SI...%T..

   1   5900                      write:entry
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 90 39 c4 53 d2 84 01 00 25 54 00 00              .9.S....%T..

[ ... ]

This is certainly much better than seeing this:

 CPU     ID                    FUNCTION:NAME
   0   5900                      write:entry
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 53 53 48 2d 32 2e 30 2d 53 75 6e 5f 53 53 48 5f  SSH-2.0-Sun_SSH_
        10: 32 2e 32 0a 00 00 00 00 18 00 00 00 00 00 00 00  2.2.............
        20: 98 f6 d3 08 25 1f d3 08 01 21 d3 08 00 00 00 00  ....%....!......
        30: 00 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00  ................
        40: c0 23 d2 08 0d 21 d3 08 10 00 00 00 01 00 00 00  .#...!..........
        50: 9d 11 00 00 00 00 00 00 18 00 00 00 00 00 00 00  ................
        60: c0 24 d2 08 25 1f d3 08 0d 21 d3 08 00 00 00 00  .$..%....!......
        70: 00 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00  ................
        80: 80 25 d2 08 01 00 00 00 02 00 00 00 03 00 00 00  .%..............
        90: ff ff ff ff 04 00 00 00 18 00 00 00 00 00 00 00  ................
        a0: a0 22 d2 08 2e 21 d3 08 0e 00 00 00 01 00 00 00  ."...!..........
        b0: ab 11 00 00 00 00 00 00 18 00 00 00 00 00 00 00  ................
        c0: 00 23 d2 08 2e 21 d3 08 00 00 00 00 25 1f d3 08  .#...!......%...
        d0: 00 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00  ................
        e0: b8 f6 d3 08 00 00 00 00 00 00 00 00 00 00 00 00  ................
        f0: 00 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  ................
       100: d8 f6 d3 08 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ ... ]
       3d0: 00 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  ................
       3e0: b8 f9 d3 08 00 00 00 00 00 00 00 00 00 00 00 00  ................
       3f0: 00 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  ................

tracemem() output consistency

Another enhancement we've made to tracemem() is to modify the behavior of tracemem() for certain traced memory sizes.  Previously, when tracing 1, 2, 4 or 8 bytes, the traced memory would be treated as a signed decimal integer.  When tracing any other amount below 32 bytes, the traced memory was treated as a string if the buffer contained only printable ASCII characters.  For example:

# dtrace -qn 'BEGIN {tracemem(&`initname, 1); exit(0)}'
# dtrace -qn 'BEGIN {tracemem(&`initname, 4); exit(0)}'
# dtrace -qn 'BEGIN {tracemem(&`initname, 32); exit(0)}'
# dtrace -qn 'BEGIN {tracemem(&`initname, 64); exit(0)}'

             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 2f 75 73 72 2f 73 62 69 6e 2f 69 6e 69 74 00 00  /usr/sbin/init..
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 80 00 00 00 01 00 00 00 84 2b ab fb ff ff ff ff  .........+......
        30: ac 2e ab fb ff ff ff ff ac 31 ab fb ff ff ff ff  .........1......


We've modified this behavior to be consistent across traced memory sizes.  For example:

# dtrace -qn 'BEGIN {tracemem(&`initname, 1); exit(0)}'

             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 2f                                               /

# dtrace -qn 'BEGIN {tracemem(&`initname, 4); exit(0)}'

             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 2f 75 73 72                                      /usr

# dtrace -qn 'BEGIN {tracemem(&`initname, 32); exit(0)}'

             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 2f 75 73 72 2f 73 62 69 6e 2f 69 6e 69 74 00 00  /usr/sbin/init..
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

# dtrace -qn 'BEGIN {tracemem(&`initname, 64); exit(0)}'

             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 2f 75 73 72 2f 73 62 69 6e 2f 69 6e 69 74 00 00  /usr/sbin/init..
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 80 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 88 32 b6 fb ff ff ff ff 00 36 b6 fb ff ff ff ff  .2.......6......


Structure alignment

DTrace has had a longstanding issue with how it calculates alignment for structures, unions, and bit-fields.  At best, this provided an annoyance requiring users to modify their scripts to manually add padding to reflect correct behavior, or at worst, would provide wrong information when using things like the sizeof() action or produce alignment errors where there shouldn't be any issue.  We have made several modifications such that structures, unions, and bit-fields are now more ABI compliant.

In the case of bit fields, depending on where the break is between the bits, some would get lost or placed where they shouldn't.  For example, having a DTrace script written as:

union u1tag
        unsigned char c;
                unsigned int b1:1;
                unsigned int b2:7;
        } s1;
} u1;

        u1.c = 255;
        printf ("%d %d %d\n", u1.c, u1.s1.b1, u1.s1.b2);
        u1.c = 0;
        u1.s1.b1 = 1;
        u1.s1.b2 = 127;
        printf ("%d %d %d\n", u1.c, u1.s1.b1, u1.s1.b2);

Would produce a result of:

255 1 0
128 1 127

If you were to write the same thing in C, you would see:

255 1 127
255 1 127

For structs and unions, the problem is more pronounced.  Suppose we have a DTrace script:

typedef struct _my_data {
        uint32_t a;
        uint64_t c;
} my_data;

typedef struct _more_data {
        uint32_t x;
        my_data y;
} more_data;

more_data a;

        a.y.c = 30;
        printf("%lu\n", a.y.c);

When we run it, DTrace will report that this otherwise valid structure will result in an invalid alignment.

 # dtrace -64 -s /tmp/test.d
dtrace: script '/tmp/test.d' matched 1 probe
dtrace: error on enabled probe ID 1 (ID 1: dtrace:::BEGIN): invalid alignment
(0xffffff08bf603a0c) in action #1 at DIF offset 24

What would be done in the past to work around this issue is to add a padding variable so that the struct _more_data would look like:

typedef struct _more_data {
        uint32_t x;
        uint32_t padding;
        my_data y;
} more_data;

This means that without the padding, not only did we run into alignment issues that would produce runtime errors, but we ran the risk of reporting wrong data within the members of the struct.  This would also impact the size being reported by actions such as sizeof().

For instance suppose we have two structs:

struct s1 {
        int x;          /* 4 bytes */
        short a;        /* 2 bytes */

struct s2 {
        struct s1 b;    /* 6 bytes */
        short c;        /* 2 bytes */
        int d;          /* 4 bytes */

In this case, sizeof(s1) would report a size of 6, and sizeof(s2) would report a size of 12 instead of 8 and 16 respectively.

One last alignment issue is being addressed with this release, and that deals with the memory alignment restrictions between SPARC and x86 architectures.  In the past, DTrace would force a 4-byte alignment regardless of the architecture the script happened to be running on. With 11.2, alignment is based on the platform since SPARC still requires 4-byte alignment for memory accesses, but the x86 architecture is more lenient and doesn't require such restrictions.

Summing up, by insuring ABI compliance each structure will match their C counterpart, and when you copyin a struct from your program, and rely on the structs being defined inside a header file or your own D scripts, you can rest assured the data is reliable.

Wednesday May 07, 2014

Solaris-specific Providers for Puppet

As I mentioned in my previous post about Puppet, there are some new Solaris-specific Resource Types for Puppet 3.4.1 in Oracle Solaris 11.2.  All of these new Resource Types and Providers have been available on since integration into the FOSS projects gate.  I am actively working with Puppet Labs to get this code pushed back upstream so that it's available for anybody to work with.

Here's a small description of a few (of 23) of the new Resource Types:

  • boot_environment
    • name - The boot_environment name (#namevar)
    • description - Description for the new boot environment
    • clone_be - Create a new boot environment from an existing inactive boot environment
    • options - Create the datasets for a new boot environment with specific ZFS properties.  Specified as a hash
    • zpool - Create the new boot environment in the specified zpool
    • activate - Activate the specified boot environment
  • pkg_publisher
    • name - The publisher name (#namevar)
    • origin - Which origin URI(s) to set.  For multiple origins, specify them as a list
    • enable - Enable the publisher
    • sticky - Set the publisher 'sticky'
    • searchfirst - Set the publisher first in the search order
    • searchafter - Set the publisher after the specified publisher in the search order
    • searchbefore - Set the publisher before the specified publisher in the search order
    • proxy - Use the specified web proxy URI to retrieve content for the specified origin or mirror
    • sslkey - The client SSL key
    • sslcert - The client SSL certificate
  • vnic
    • name - The name of the VNIC (#namevar)
    • temporary - Optional parameter that specifies that the VNIC is temporary
    • lower_link - The name of the physical datalink over which the VNIC is operating
    • mac_address - Sets the VNIC's MAC address based on the specified value
  • dns
    • name - A symbolic name for the DNS client settings to use.  This name is used for human reference only
    • nameserver - The IP address(es) the resolver is to query.  A maximum of 3 IP addresses may be specified.  Specify multiple addresses as a list
    • domain - The local domain name
    • search - The search list for host name lookup.  A maximum of 6 search entries may be specified.  Specify multiple search entries as a list
    • sortlist - Addresses returned by gethostbyname() to be sorted.  Entries must be specified in IP 'slash notation'.  A maximum of 10 sortlist entries may be specified.  Specify multiple entries as an array.
    • options - Set internal resolver variables.  Valid values are debug, ndots:n, timeout:n, retrans:n, attempts:n, retry:n, rotate, no-check-names, inet6.  For values with 'n', specify 'n' as an integer.  Specify multiple options as an array.

Other Resource Types are:

  • Datalink Management:   etherstub, ip_tunnel, link_aggregation, solaris_vlan
  • IP Network Interfaces:  address_object, address_property, interface_properties, ip_interface, ipmp_interface,                                             link_properties, protocol_properties, vni_interface
  • pkg(5) Management:  pkg_facet, pkg_mediator, pkg_variant
  • Naming Services:  nis, nsswitch, ldap

The zones Resource Type has been updated to provide Kernel Zone and archive support as well.

Tuesday May 06, 2014

OpenSSL on Oracle Solaris 11.2

I'm sure you all wonder which version of OpenSSL is delivered with Oracle Solaris 11.2?
The answer is the latest and greatest OpenSSL 1.0.1h!

Now that I answered 80% of the questions you may have with regard to OpenSSL, I would like to announce three major features added to the Oracle Solaris 11.2 which I'm sure you'll all be excited to hear :-)

Inlined T4/T4+ instructions support and Engines

Background: S11.1 and earlier

Years and years ago, I worked on the SPARC T2/T3 crypto drivers.  On the SPARC T2/T3 processors, the crypto instructions are privileged; and therefore, the drivers are needed to access those instructions.  Thus, to make use of T2/T3 crypto hardware, OpenSSL had to use pkcs11 engine which adds lots of cycles going through the thick PKCS#11 session/object management layer, Solaris kernel layer, hypervisor layer to the hardware, and all the way back.  However, on SPARC T4/T4+ processors, crypto instructions are no longer privileged; and therefore, you can access them directly without drivers.  Valerie Fenwick has a nice article explaining the lower level specifics of the T4 hardware.

What does that means to you?  Much improved performance!  No more PKCS#11 layer, no more copy-in/copy-out of the data from the userland to the kernel space, no more scheduling, no more hypervisor, NADA!   As much as I enjoyed working on the crypto drivers, I'm happy to see this driver-less transition! ;-)

Dan Anderson has a great blog entry describing the difference between the T3 and T4 based hardware.  As he described, on Solaris 11 and 11.1, we made the T4 instructions available to OpenSSL via OpenSSL engine mechanism.  It was great for the time being, but to make T4 instruction support available directly from the OpenSSL website and to even bypass the engine layer all together, I was assigned to assassinate the t4 engine (Sorry, Dan) and make T4 instructions embedded to the OpenSSL's internal crypto module (a.k.a adding inlined T4 instruction support).

S11.2 and beyond

As I was learning how OpenSSL development worked, I learned OpenSSL upstream engineers had already committed the inlined T4 instruction support to the OpenSSL 1.0.2 branch.  (Thanks for making my life easier, OpenSSL team!)  I was job-less for a second, but since OpenSSL 1.0.2 won't be available in time for Solaris 11.2 delivery, we decided to patch the inlined T4 instruction support to our OpenSSL 1.0.1g delivery bundled with Solaris 11.2.

With this change, you'll get the T4/T4+ instruction support without engines; and therefore, you get as great performance as the t4 engine and even better performance for some algorithms (i.e. SHA-1, MD5) by default.

Other Engines

Oracle Solaris 11.2  killed not only the t4 engine, but also the aesni engine and the devcrypto engine.   The story for the aesni engine is pretty much similar to the one for the t4 engine.   It was introduced in Solaris 11 as Dan Anderson described in his article, and killed in Solaris 11.2.  AES-NI instruction support is now embedded in the OpenSSL upstream implementation (OpenSSL 1.0.1); and therefore, the separate engine is no longer needed.  The devcrypto engine was removed simply due to the lack of use.

With all this change, Oracle Solaris 11.2 OpenSSL is left with the one and only pkcs11 engine. pkcs11 engine is still necessary on the T2/T3 platforms and on any platform with the hardware keystore (i.e. SCA 6000). However, be sure to leave the pkcs11 engine disabled on T4/T4+ if you want max performance.  Again, I would like to emphasize that the OpenSSL performance on T4/T4+ platforms are looking MUCH better compared to the one on T2/T3 platforms!  It's time to move onto T4/T4+ platform, Y'all!!

OpenSSL FIPS-140 version support

It is important for many federal and financial service customers to have their cryptographic products being FIPS-140 validated. Oracle Solaris Cryptographic Framework recently achieved a FIPS 140-2 validation(yay!!), and it was very important to deliver the FIPS-140 validated OpenSSL with Solaris 11.2.

At the time Solaris 11 was released, OpenSSL 1.0.0 was the latest OpenSSL version available, and since OpenSSL 1.0.0 was not FIPS-140 validated, we only delivered non-FIPS-140 version of OpenSSL with Solaris 11.

Thanks to the OpenSSL upstream team (again), the best and greatest OpenSSL 1.0.1 can be compiled with a FIPS-140 validated module, and we are now delivering the FIPS-140 version of OpenSSL in addition to the non-FIPS-140 version of OpenSSL with Solaris 11.2.

When do you want to use FIPS-140 version of OpenSSL?

It's probably important to mention that the FIPS-140 version of OpenSSL is not for everybody.  The FIPS-140 validated version of cryptographic products come with a price tag.  Enabling FIPS-140 mode adds a lot of cycles to satisfy the FIPS-140 verification requirement (i.e. POST, pair-wise consistency test, contiguous RNG test, etc) at run time.  In addition, inlined T4/T4+ instruction support is not available in the FIPS-140 version of OpenSSL, and you won't get the best performance when the FIPS-140 mode is enabled.

That said, I would recommend you to enable FIPS-140 mode *only if* you need to.  The good news is that you will get the FIPS-140 compatible implementation even when the FIPS-140 mode is disabled.  It's just that it runs much faster!
That's one of the reasons why non-FIPS-140 version of OpenSSL is activated by default.

How to enable FIPS-140 version of OpenSSL

If you decided to enable FIPS-140 mode, here is how you can switch to the FIPS-140 version of OpenSSL.

Make sure you have the FIPS-140 version of the OpenSSL installed on the system.

# pkg mediator -a openssl
openssl  vendor            vendor     default
openssl  system            system     fips-140

To activate the fips-140 implementation
# pkg set-mediator -I fips-140 openssl

To check the currently activated OpenSSL implementation
# pkg mediator openssl

To change back to the default (non-FIPS-140) implementation
# pkg set-mediator -I default openssl

OpenSSL Thread and Fork Safety

OpenSSL provides an interface CRYPTO_set_locking_callback() for you (any application or library) to set your own locking callback function with the mutexes of your choice.
That sounds reasonable if the OpenSSL library is used only by applications.  However, when the OpenSSL library is used by another library, such design is asking for trouble.

We've seen a case where an OpenSSL application used a library which set a locking callback function, and the library got unloaded while the application continued using the OpenSSL library.  The application got a segfault because OpenSSL tried to reference the invalid locking callback function set by the unloaded library.  Whose fault is this?

You can argue that the library should have set the locking callback to NULL when it was unloaded.
Well, not quite.  Once the locking callback is set to NULL, the application is no longer thread-safe.

OpenSSL needed some changes to make applications and libraries thread and fork safe.

To fix this issue, the OpenSSL library ( delivered with Solaris 11.2 sets up mutexes and a locking callback internally, and it ignores an attempt to set/change the locking callback.

What does that mean to you?
OpenSSL is now thread and fork safe by default.  You don't need to make any modification to your application nor library.  You can relax and have a margarita or two.

That's all I have for now.

Note:  The version number delivered with Solaris 11.2 was updated from 1.0.1g to 1.0.1h on Jun 05, 2014. OpenSSL version 1.0.1g was delivered with Solaris 11.2 Beta.

Puppet Configuration in Solaris

What is Puppet?

Puppet is IT automation software that helps system administrators manage IT infrastructure. It automates tasks such as provisioning, configuration, patch management and compliance. Repetitive tasks are easily automated, deployment of critical applications occurs rapidly, and required system changes are proactively managed. Puppet scales to meet the needs of the environment, whether it is a simple deployment or a complex infrastructure, and works on-premise or in the cloud.

Puppet is now available as part of Oracle Solaris 11.2!

Use ntpdate or ntpd -q to set the date

Puppet can error out with some very strange messages if the clocks on both the master and agent aren't synchronized.  You can use ntpdate or ntpd -q to set the date just once if you'd like to manage the NTP service with Puppet, or you can configure NTP.

Install the required packages on both systems 

# pkg install puppet

This will install the puppet, facter and ruby-19 packages.

Configure the Puppet SMF instances

master # svccfg -s puppet:master setprop config/server =
master # svccfg -s puppet:master refresh
master # svcadm enable puppet:master

agent # svccfg -s puppet:agent setprop config/server =
agent # svccfg -s puppet:agent refresh

Test the connection to the master and configure authentication

Before enabling the puppet:agent service, you'll want to test the connection first in order to set up authentication

agent # puppet agent --test --server

Info: Creating a new SSL key for
Info: Caching certificate for ca
Info: Creating a new SSL certificate request for
Info: Certificate Request fingerprint (SHA256):
**Exiting; no certificate found and waitforcert is disabled**

Now that the agent has created a new SSL key, authorization needs approval on the master.

Sign the SSL certificate on the master

master # puppet cert list
  "" (SHA256)

master # puppet cert sign
Notice: Signed certificate request for
Notice: Removing file Puppet::SSL::CertificateRequest at

Retest the agent to ensure it can connect

agent # puppet agent --test --server
Info: Caching certificate for
Info: Caching certificate_revocation_list for ca
Info: Retrieving plugin
Info: Caching catalog for
Info: Applying configuration version '1371232699'
Notice: Finished catalog run in 0.65 seconds

Enable the agent service

agent # svcadm enable puppet:agent

Additional configuration of /etc/puppet/puppet.conf on both master and agent (optional) 

Further customizations can be made in /etc/puppet/puppet.conf.  See Puppet's Configurables page for more details.

NOTE:  Puppet's configuration is completely done via  SMF stencils.  /etc/puppet/puppet.conf should not be directly edited as any edits will be lost when the Puppet SMF service (re)starts.  Setting a new value should be done via svccfg(1M):

# svccfg -s puppet:agent setprop config/<option> = <value>

# svccfg -s puppet:agent refresh

(substitute :master as needed)

Tuesday Apr 29, 2014

New in SMF Documentation for Oracle Solaris 11.2

The Service Management Facility guide is all new for the Oracle Solaris 11.2 release, with much more information including an example of creating a pair of services that start and stop an Oracle Database instance and an examination of the Puppet stencil service.

For more information about stencil services, see Solaris SMF Weblog, and see the svcio.1 and smf_stencil.4 man pages below.

Managing System Services in Oracle Solaris 11.2

Chapter 1, "Introduction to the Service Management Facility"

Chapter 2, "Getting Information About Services"
- Service states and contract processes
- Service dependencies and dependents
- New -L option to show service log files
- Property values in layers, snapshots, and customizations

Chapter 3, "Administering Services"
- Starting, restarting, stopping
- Re-reading configuration
- Configuring notification

Chapter 4, "Configuring Services"
- Setting and adding property values
- Adding service instances
- Using profiles to configure multiple systems

Chapter 5, "Using SMF to Control Your Application"
- Creating a service to start or stop an Oracle Database instance
- Using a stencil to create a configuration file

Appendix A, "SMF Best Practices and Troubleshooting"
- Repairing an instance that is in maintenance
- Diagnosing and repairing repository problems
- How to investigate problems starting services at system boot

User Commands                                            svcio(1)

     svcio - create text files  based  on  service  configuration

     /lib/svc/bin/svcio [-alux] [-f FMRI-instance] [-g group]
          [-i file] [-m mode] [-o file] [-O owner]
          [-R dir [-L opts [-p]]] [-S dir]

     The svcio utility reads a template known as  a  stencil  and
     emits  text  based on that file in conjunction with the pro-
     perties from a service instance.  In the typical case, svcio
     is used to generate application-specific configuration files
     for services that are managed by, but are not able  to  read
     their configurations from, SMF.

     If the stencil itself contains any errors, svcio  will  pro-
     vide  a  snippet  of  text  along with a line number and the
     cause of the error.  Unless the error would prevent  further
     progress,  each  error  is printed in the order it occurs in
     the file.

     Error messages are printed to the standard error stream.

     The following options are supported:


         Process all files configured for an instance.

         Specifically, svcio will look  at  all  property  groups
         with  the  type "configfile" and determine which stencil
         to use and where to write the resulting file by  examing
         the values of the properties "path" and "stencil" within
         that property group.  For  example,  if  property  group
         "conf1"  is  of the appropriate type then svcio will use
         the value of "conf1/stencil" as the path of the  stencil
         file  and  "conf1/path" as the path of the file to which
         to write the output.  Additionally, the optional proper-
         ties  "owner"  and  "group" can be used to set the owner
         and group of the output file respectively. If  the  pro-
         perty  group  name  or property name contains a reserved
         character (see smf(5)) then it must be encoded.

     -f FMRI-instance

         The FMRI of  a  service  instance  to  run  the  stencil

     -g group

         The group to associate the output files with

     -i file

         The path to the stencil file (default is  stdin).   This
         option cannot be used with -a.


         Rather than outputting a text file, simply list all pro-
         perties  that would be referenced were a file to be out-

     -L opts

         Specify options to be passed to mount(2)  when  loopback
         mounting output files.  If this option is not specified,
         output files will  not  be  loopback  mounted.   The  -R
         switch  is  required  with  this option.  A regular file
         will be written to the specified output path, rooted  at
         prefix. This file will be loopback mounted to the speci-
         fied output path, rooted at / or the value of the  -R-fR

     -m mode

         Set the mode for any output file (default is 644).

     -o file

         The path to the output file (default is  stdout).   This
         option cannot be used with -a.

     -O owner

         Set the owner of the output files

     -R prefix

         Set the root prefix for all output files.


         Create nonexistent  intermediate directories in the out-
         put  file  path  rooted  at  the value of the -R option.
         Note:  This option will not create directories that  are
         missing in the path to the mount point.

     -S dir

         Look for stencils in  this  directory  rather  than  the


         Unlink output files and undo loopback mounting.  No out-
         put files will be created.


         Terminate svcio on the first error rather than continu-
         ing to the next stencil.

     The following operands are supported:


         A  fault  management  resource  identifier  (FMRI)  that
         specifies  one or more instances (see smf(5)). FMRIs can
         be abbreviated by specifying the instance name,  or  the
         trailing portion of the service name. For example, given
         the FMRI:


         The following are valid abbreviations:


         The following are invalid abbreviations:


         If the  FMRI  specifies  a  service,  then  the  command
         applies  to  all  instances of that service, except when
         used with the -D option.

         Abbreviated forms of FMRIs are unstable, and should  not
         be used in scripts or other permanent tools.


         An FMRI that specifies an instance.

     Example 1 Processing All Configuration Files for an Instance

     This example processes all  configured  configuration  files
     for an instance:

       example% svcio -a -f svc:/service:instance

     Example 2 Removing All Configuration Files for an Instance

     This example unlinks and unmounts all configured  configura-
     tion files for an instance:

       example% svcio -au -f svc:/service:instance

     Example 3 Using an Unconfigured Stencil for an Instance

     This example produces an output  file  based  on  a  stencil
     that has not been configured:

       example% svcio -o /etc/svc.conf -i ~/svc.stencil \
       -f svc:/service1:instance

     The following exit values are returned:


         Successful command invocation.


         A fatal error occurred as a result of a failed system


         Invalid command line options were specified.


         A fatal error occurred as a result of an unexpected SMF


         An error occurred parsing a stencil.

     See attributes(5) for descriptions of the following attri-

    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    | Availability                | system/core-os              |
    | Interface Stability         | Committed                   |

     smf_stencil(4), svcs(1), svcprop(1), svcadm(1M), svccfg(1M),
     svc.startd(1M), stat(2), libscf(3LIB), smf(5)

File Formats                                       smf_stencil(4)

     smf_stencil - defines the relationship between  SMF  proper-
     ties and a flat configuration file

     A stencil file defines a mapping between SMF properties  and
     flat text files.  The Service Management Facility, described
     in smf(5),  uses  stencil  files  in  conjunction  with  the
     svcio(1)  utility to generate text-based configuration files
     from SMF properties by invoking svcio(1)  before  the  start
     and  refresh  methods  of  a property configured service are

     The language understood by svcio(1) is comprised of a  small
     set  of  expressions  that  can  be  combined  to  concisely
     describe the structure of a configuration file  and  how  to
     populate  that  file with data from the SMF repository.  The
     expressions comprising the language are listed below:

     I.    $%{property_fmri[:<transform><transform_expression>]}

       Retrieve and emit the value(s) associated with a property.

       <transform> can be one of the following characters,  which
       define how to handle <transform_expression>:

       -   emit <transform_expression> if  the  property  is  not

       +   emit <transform_expression> if the property is defined

       ?   <transform_expression>    is    of     the     form
           "<true>[/<false>]".  If the boolean property is true,
           then emit <true>, otherwise emit <false>.

       ,   emit <transform_expression>  as  a  delimiter  between
           values in multi-valued properties

       ^   <transform_expression>  is  of  the  form  "<p>[/<s>]"
           where  <p>  is  used as a prefix and <s> is used as a
           suffix when emitting property values

       ^*  Same as '^', but nothing is emitted if the property is
           undefined or empty

       '   <transform_expression>     takes      the      form
           "<pattern>/<replace>",  where  <pattern>  is  a shell
           pattern style glob (for details, see  the  File  Name
           Generation section of sh(1)).  The first substring to
           match <pattern> is replaced with <replace>

       ''  Same as ', but every substring that matches  <pattern>
           is replaced with <replace>

     II.   $%/regular_expression/ { <sub_elements> }

       Process <sub_elements> for each property FMRI and property
       group  FMRI  that  matches regular_expression. As the pro-
       perty group and property is specified as an FMRI they must
       be  encoded  if  they  contain  reserved  characters  (see

     III.  $%<number>

       Retrieve a marked subexpression from a regular expression.

       Retrieve a marked subexpression from a regular expression.

     IV.   $%define name /regular_expression/ { <sub_elements> }

       Name a regular expression such that it can be  used  else-
       where in the stencil.

     V.    $%[regex_name[:<transform><transform_expression]]

       Recall a previously defined regular expression (as in IV).
       In  this  case, the set of transform characters is limited
       to ^, ', and ''.

     VI.   $%define name arg 1 arg 2 ... argN { <sub_elements> }

       Name a macro such that it can be  used  elsewhere  in  the

       Note: In the text above, '[' and ']' denote the macro del-
       imiters  rather  than  optional parameters as they do in I
       and V.

     VII.  $%<arg_name>

       Retrieve the text associated with a macro argument.

     VIII. $%[name foo bar ... baz]

       Recall a previously defined macro (as in VI).

     IX.   $%<method_token>

       Retrieve the value of an environment variable  represented
       by a method token describe in smf_method(5).

     X.    Literal Text

       Arbitrary text can  be  freely interspersed throughout the
       stencil without any denotative markers.

     XI.   ;comments

       A line that starts  with  a  ';',  ignoring  leading  whi-
       tespace,  is  considered a comment and not processed along
       with the rest of the file.

     Any of the  special  characters  above  can  be  escaped  by
     preceding  them  with a blackslash (\) character.  Addition-
     ally, the '\n' and '\t' sequences are expanded into endlines
     and  tab characters respectively.  Any non-special character
     preceded by '\' will emit only the character  following  the
     slash.  Thus '\g' will be translated to 'g'.

     I. $%{property_fmri[:<transform><transform_string>]}

       Example: $%{general/enabled:?on/off}

       This element will fetch the value (or values)  of  a  pro-
       perty  and  emit  a  string  subject to the transform, the
       transform string, and the values themselves.   <transform>
       is  a one- or two- character identifier that indicates how
       to modify a property value before emitting it, subject  to
       <transform_string>, as explained above.

       Note that nesting is allowed.  Imagine we  want  to  print
       the value of foo/b if foo/a is defined, but 'blueberry' if
       it is not.  This could be accomplished via the following:

       it is not.  This could be accomplished via the following:


       For the purposes of resolving FMRIs  into  values,  a  few
       shortcuts  are allowed.  Since svcio is always run against
       a specific instance, properties from that instance can  be
       shortened to "pg/prop" rather than a fully qualified FMRI.
       To  reference  properties  that  are  not  part   of   the
       instance,                     the                     full
       "svc:/service:instance/:properties/pg/prop" is required.

     II. $%/regular_expression/ { sub_elements> }

       Example: $%/pg/(.*)/ {lorem ipsum}

       This element defines a regular expression to match against
       the  entire  set  of property FMRIs on a system.  For each
       property FMRI that matches, the subelements are evaluated.
       When evaluating subelements, svcio(1) iterates over match-
       ing properties in lexicographical  order.   svcio(1)  uses
       the  POSIX extended regular expression set (see regex(5)),
       and  supports  saving  subexpressions   via   parentheses.
       Finally,  as a convenience svcio will surround the regular
       expression with ^ and $ characters.  Should you want  your
       expression  to  match  the  middle of strings, prepend and
       append '".*".

       Since  both  properties  associated  with  the   operating
       instance  as  well  as  properties  from other services or
       instances, regular expressions are only matched against  a
       subset  of  FMRIs  on the system.  If a regular expression
       includes the substring ":properties",  the  expression  is
       parsed for the service and/or instance where those proper-
       ties reside.  Once those properties are fetched, the regu-
       lar  expression  is matched only against that set.  If the
       regular expression does not contain  that  substring,  the
       only  properties  matched  are  those  associated with the
       operating instance.

       Note that the end of a regular expression is denoted by '/
       {'  so  it  is  not  necessary  to escape slash characters
       within the regular expression.

     III.  $%<number>

       Example: $%3

       This element emits the value from a  stored  subexpression
       in  a  preceding  regular  expression.  Using this element
       outside the context of a regular expression is  an  error.
       A valid use would be as follows:

       $%/foo/(.*)/ {
            $%1 = $%{foo/$%1}

       In the preceding example, every property in property group
       foo    would    be    emitted    as   "<property_name>   =

       Since arbitrary subelements are allowed within  a  regular
       expression  block,  nested  regular expressions have their
       subexpression indices adjusted relative to  the  index  of
       the  last subexpression of the containing expression.  For

       ;([a-zA-Z_-]*) is $%1
       $%/([a-zA-Z_-]*)/ {


       ;([a-zA-Z_-]*) is $%1
       $%/([a-zA-Z_-]*)/ {
            ;(.*) becomes $%2
            $%/$%1/(.*)/ {
                 $%2 = $%{$%1/$%2}

       In the preceding example,  every  property  group  for  an
       instance would be emitted in blocks as follows:

            prop1 = <prop1_value>
            prop2 = <prop2_value>

     IV.  $%define name /regular_expression/ { <sub_elements> }

       Example: $%define getProp //(.*)/ {dolor sit amet}

       This element follows the same basic rules as  element  II,
       but stores the element as a named regular expression  that
       can be invoked later in the stencil file.   Named  regular
       expressions are  not matched unless they are referenced as
       per element V, which immediately  follows.   Additionally,
       This element cannot be a child to any other.

     V. $%[regex_name:<transform><transform_string>]

       Example: $%[getProp:^restarter]

       This inserts  a  previously  defined  regular  expression,
       along  with all its subelements into the stencil as though
       the definition were copy and pasted.  Since the  insertion
       is  performed literally, there are some special rules that
       govern how the insertion is done in order to allow such an
       element  to  be  meaningful  at  many levels of expression
       nesting.  First of  all,  all  subexpression  indices  are
       interally  adjusted  so  that they do not collide with the
       outer regular expression context.  Second, a subset of the
       transformations   from   element  I  are  allowed.   These
       transforms operate on relative FMRIs within  the  inserted
       element.   Absolute FMRIs are left untouched.  This allows
       a stencil author to do useful things like prepend  to  the
       FMRI in order to express logical property nesting.  Here's
       an example:

       $%define PROPERTY /(.*)/ { $%1 = $%{$%1} }

       $%/([a-zA-Z_-]*)/ {

       When the insertion is done, the expression  will  function
       as follows:

       $%/([a-zA-Z_-]*)/ {
            $%/$%1/(.*)/ {
                 $%2 = $%{$%1/$%2}

       This is equivalent to the example in element III.

       It ends up this way because the rebasing during  substitu-

       This is equivalent to the example in element III.

       It ends up this way because the rebasing during  substitu-
       tion changes the $%1 to $%2, since $%1 occurs in the outer
       expression.  And as a  result  of  the  prepend  transform
       applied   during   substitution,   the  string  "$%1/"  is
       prepended to both the regular  expression  (since  regular
       expressions match FMRIs) as well as to the element of type
       II, allowing it resolve to a full  pg/property  specifier.
       The  subset  of allowed transforms is ^,',''.  Using other
       transforms is an error.

     VI. $%define macroName arg1 arg2 ... argN { <sub_elements> }

       Example: $%define defaultHost { myMachine }
                $%define getGeneral prop { $%{general/$%prop} }

       Macros provide simple text substitution  with  respect  to
       the  arguments  defined for the macro.  When called subse-
       quent to definition, the text of the sub-elements is emit-
       ted  with  the  text  of  the  arguments substituted where
       appropriate.  See the elements below for more details.

     VII. $%<argName>

       Example: $%prop

       This element emits the corresponding value passed into the
       macro that uses argName as an argument.  For example:

       $%define someMacro someArg someOtherArg {
               $%someArg = $%{pg/$%someOtherArg}

     VIII.  $%[macroName arg1 arg2 ... argN]

       Example: $%[getGeneral enabled]

       After a macro has been defined, the sub-elements  in  con-
       tains  can  be substituted into other parts of the stencil
       by using the form above.  When invoking  a  macro,  spaces
       are  used  to  delimit arguments.  In order to use a space
       within the value of an argument, it is necessary to escape
       that space with a ''.  For example, if we have the macro:

       $%define theMacro variable value {
               $%variable = $%value

       We can then use this form to substitute  that  text  else-
       where in the stencil.  For example, we can call it as fol-

       $%[theMacro ciphers elGamal\ 3DES\ AES\ Blowfish]

       And the resulting text in the output file would be:

       ciphers = elGamal 3DES AES Blowfish

     IX.  $%<method_token>

       Example: $%s

       Each of the single-character method  tokens  described  in
       smf_method(5)  are  available  in stencils.  In particular
       $%r, $%m, $%s,  $%i,  $%f,  and  $%%  are  understood  and
       expanded.   Due to the high chance of collision with macro
       variables (element VII), macro variables  have  precedence
       over method tokens when expansion occurs.  This means that

       variables (element VII), macro variables  have  precedence
       over method tokens when expansion occurs.  This means that
       if the macro variable $%someVar is encountered, it will be
       expanded  to  the value of $%someVar rather than 'service-
       nameomeVar'.  If output such  as  'service-nameomeVar'  is
       desired,  simply  escape a character in the macro variable
       as in $%s\omeVar.

     X.  Literal text

       Example: Lorem ipsum dolor sit amet, consectetur adipisic-
                ing  elit,  sed  do  eiusmod tempor incididunt ut
                labore et dolore magna aliqua.

       Literal text can be freely interspersed within the stencil
       and  is emitted  without modification.  The examples above
       make limited use of literal text.  Text appearing inside a
       regular  expression  is emitted for each match, but is not
       emitted if there are no matches.  Text  appearing  outside
       all  the  preceding  expression  types  is  emitted in all

     XI.  Comments

       Example: ;this is a comment
                     ;so is this
                \;this text will appear in the output file
                so will this, even with the ';' character

       To begin a comment, start the line with  a  ';'  character
       (not  including  whitespace).  The comment continues until
       the end of the line.  If having comments in the  resulting
       output  file  is  desired, simply escape the ';' with a ''

       ;The following example creates a 'configuration file'
       ;that lists some details of the service
       $%define author {Alice}
       $%define reviewer {Bob}

       This file  was  written  by  $%[author]  and  verified  by

       Preferences are $%{preferences/validated:+validated!}

       The following is a .ini style listing of all  the  proper-
       ties of service $%s and instance $%i:

       ;display a property in the form
       ;'   prop_name = prop_value'
       $%define display_property prop
       {\t$%prop = $%{/$%prop}\n}

       ;invokes display_property macro for each
       ;property matched
       $%define property //(.*)/ {$%[display_property $%1]}

       ;matches all property groups (lack of '/' prevents
       ;matching properties) and emits the property group
       ;name in brackets, with each property listed underneath.
       ;The expression '^$%1' means prepend all relative FMRIS
       ;in the regular expression named 'property' with the
       ;property group that satisfies this regular expression
       $%/([a-zA-Z0-9_-]*)/ {



     Suppose we have  a  service  'Foo'  with  just  the  default
     instance and the following properties:

       pg1/prop1 = val1
       pg1/prop2 = va2
       pg2/prop1 = val3 val4
       pg2/prop2 = val5
       preferences/validated = yes

     Using svcio(1) to the example  stencil  with  service  'Foo'
     would result in the following text:

       This file was written by Alice and verified by Bob

       Preferences are validated!

       The following is a .ini style listing of all  the  proper-
       ties of service Foo and instance default:

            prop1 = val1
            prop2 = val2

            prop1 = val3 val4
            prop2 = val5

            validated = yes

     It is also possible to  rewrite  the  example  stencil  more
     tersely, as shown below:

     $%define author {Alice}
     $%define reviewer {Bob}

     This  file  was  written  by  $%[author]  and  verified   by

     Preferences are $%{preferences/validated:+validated!}

     The following is a .ini style listing of all the  properties
     of service $%s and instance $%i:

     $%/([a-zA-Z0-9_-]*)/ {
     $%/$%1/(.*)/ {\t$%2 = $%{$%1/$%1}\n}

     See attributes(5) for descriptions of the  following  attri-

    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    | Availability                | system/core-os              |
    | Stability                   | Committed                   |

     svcio(1), sh(1), regex(5), svcs(1), svcprop(1),  svcadm(1M),
     svccfg(1M), svc.startd(1M), libscf(3LIB), smf(5)

New in IPS Documentation for Oracle Solaris 11.2

Documentation of the Image Packaging System on is in three books. All three books contain new information for the Oracle Solaris 11.2 release.

See also Tim Foster's Web Log


  • New pkg/mirror service
  • New pkg/depot service
  • New chapter about web server configuration, including a new section about configuring https access
  • New pkgrecv --clone option
  • New pkg install and pkg update troubleshooting section
  • New chapter about updating an image
  • New options for pkg subcommands:
    • -r: perform operation recursively on specified non-global zones
    • --sync-actuators: do not return until all actuators have finished
    • --ignore-missing: when updating or uninstalling, ignore packages that are not installed
  • New pkg exact-install command
  • New file attribute for setting system attributes

Copying and Creating Package Repositories in Oracle Solaris 11.2

Chapter 1, "Image Packaging System Package Repositories"
- New section about best practices

Chapter 2, "Copying IPS Package Repositories"
- Copying from a zip file (see also Release Engineering's blog) or iso file
- Using the pkgrecv command
- Using the new pkg/mirror service to automatically periodically update a repository

Chapter 3, "Providing Access To Your Repository"
- Using a ZFS share
- Using the pkg/server service

Chapter 4, "Maintaining Your Local IPS Package Repository"
- New repository update procedure
- Using the pkgrecv --clone option to clone a repository
- Using the new pkg/depot service to serve multiple repositories from a single location

Chapter 5, "Running the Depot Server Behind a Web Server"
- Caching, load balancing
- New section about configuring HTTPS repository access

Adding and Updating Software in Oracle Solaris 11.2

Chapter 1, "Introduction to the Image Packaging System"
- Incorporations and group packages, FMRIs, images

Chapter 2, "Getting Information About Software Packages"
- Packages that can be installed
- Package descriptions, licenses, dependencies, dependents
- Searching for packages

Chapter 3, "Installing and Updating Software Packages"
- New options for pkg subcommands regarding non-global zones, SMF actuators, and ignoring missing packages in a pkg update or uninstall
- New pkg exact-install command (see also Bart's blog)
- Updated information about non-global zones

Chapter 4, "Updating or Upgrading an Oracle Solaris Image"
- Ways to control the version to which to upgrade, including creating a custom incorporation package

Chapter 5, "Configuring Installed Images"
- Configuring publishers
- Variants and facets
- Freezing
- Incorporation constraints
- Mediations
- Groups

Appendix A, "Troubleshooting Package Installation and Update"
- All new - Begins with steps you should always do and then is organized by error message

Appendix B, "IPS Graphical User Interfaces"
- Package Manager and Package Update

Packaging and Delivering Software With the Image Packaging System in Oracle Solaris 11.2

Chapter 1, "IPS Design Goals, Concepts, and Terminology"
- General information about software self-assembly and package lifecycle
- Definitions, package components
- New file attribute, sysattr, for setting system attributes

Chapter 2, "Packaging Software With IPS"
- Updated procedures for publishing and delivering your package

Chapter 3, "Installing, Removing, and Updating Software Packages"
- How this works in the Image Packaging System

Chapter 4, "Specifying Package Dependencies"
- New firmware value for the fmri attribute of the origin dependency for specifying driver firmware compatibility

Chapter 5, "Allowing Variations"
- Variants and facets

Chapter 6, "Modifying Package Manifests Programmatically"
- Using pkgmogrify

Chapter 7, "Automating System Change as Part of Package Installation"
- Specifying actuators on package actions
- Delivering SMF services in IPS packages
- New or updated examples of a run-once service and a self-assembly service

Chapter 8, "Advanced Topics For Package Updating"
- Renaming, merging, splitting, obsoleting packages
- New or updated examples of preserving editable packaged content, preserving unpackaged content, sharing content across boot environments, overlaying files, and delivering a mediation

Chapter 9, "Signing IPS Packages"

Chapter 10, "Handling Non-Global Zones"

Chapter 11, "Modifying Published Packages"

Appendix A, "Classifying Packages"

Appendix B, "How IPS Is Used To Package the Oracle Solaris OS"


Saturday Apr 19, 2014

The Technical Details: April 29 Oracle Solaris 11.2 Launch

You may have already heard that we're going to hold the Oracle Solaris 11.2 launch in New York City in a few days, and that there will also be a live webcast of the event.

One of the things that the webcast will feature that won't be part of the live event will be additional technical presentations where Solaris engineers will go into more detail about some of the new features that are being added. VP for Solaris core engineering Markus Flierl gives a quick rundown:

If this sounds interesting to you, you should register now. The event starts at 1 PM ET / 10 AM PT, with Mark Hurd and John Fowler. Markus then moves on to the more technical part of the in-person event, which will then be followed by the web-only deep-dive presentations.

During the live event, we'll have engineering folks and others on Twitter, tracking hashtag #solaris (apologies in advance to Stanislaw Lem fans).

Webcast: Announcing Oracle Solaris 11.2
Tuesday April 29, 2014
1 PM (ET) / 10:00am (PT)

Friday Jan 10, 2014

Next OTN Virtual Sysadmin Day: January 28th, 2014

Glynn Foster notes that another OTN Virtual Sysadmin Day is coming up in just a couple of weeks, and talks about what's in store for the Oracle Solaris 11 track.

If you're not familiar with these, they're half-day, online, proctored hands-on labs, so you can learn more about various system administration technologies. They're also free--but you do need to register, and there's also some prep work to be done ahead of the event, so take a look at Glynn's blog post, and sign up today.

Thursday Feb 28, 2013

Building a "developer cloud" with Oracle Solaris 11

Orgad Kimchi describes how the new cloud technologies built into Oracle Solaris 11 can be used to build a virtualized "developer cloud."

By the way, this is the story of the "Oracle Solaris Remote Lab," an Oracle PartnerNetwork resource for developers which you'll be hearing more about.

After you read this, check out Orgad's earlier posts on other aspects of Solaris and virtualization.

Friday Feb 22, 2013

Security Experts Spill the Beans on Oracle Solaris 11.1

There have been several recent blog posts about new security / crypto features in Oracle Solaris 11.1:

Glenn Faden has written on a few topics:

Dan Anderson writes about SPARC T4 Digest and Crypto Optimizations.

Darren Moffat shows how to run up a quick Python script to generate a crypt_sha256 hash from the command line.

Monday Feb 11, 2013

Oracle Solaris 11.1: Compliance Reporting with SCAP

Not "scap"; SCAPDarren Moffat writes about the new security compliance framework added in Oracle Solaris 11.1, the emerging Security Content Automation Protocol.

"One of the important parts is the OVAL (Open Vulnerability Assessment Language) standard [which] allows us to write a checkable security policy and verify it against running systems."

[Read More]

Friday Feb 08, 2013

Oracle Solaris 10 1/13: Improved Secure File Copy Performance

We released Oracle Solaris 10 1/13 today, and one of the things we mentioned was performance improvements.

On the Oracle Performance and Best Practices blog, there's a discussion of one of those improvements:

"Oracle Solaris 10 1/13 enabling a TCP receive window size up to 1 MB has up to 8 times faster transfer times over the latency range 50 - 200 msec compared to the previous Oracle Solaris 10 8/11."

This improved performance capability was previously introduced in Oracle Solaris 11.

[Read More]

Wednesday Dec 12, 2012

Oracle Solaris 11 pkg fix

Bob Netherton explains why Solaris 11 pkg fix is his new friend.

"So far so good. Then comes an oops... This is where you generally say a few things to yourself, and then promise to quit deleting configuration files and directories when you don't know what you are doing. Then you recall that the new Solaris 11 packaging system has some ability to correct common mistakes (like the one I just made)."

[Read More]

Thursday Oct 25, 2012

Oracle Solaris 11.1

Jeff Victor writes a brief note about what he likes about Oracle Solaris 11.1 (which just became available for download today).

The Observatory is a blog for users of Oracle Solaris. Tune in here for tips, tricks and more as we explore the Solaris operating system from Oracle.


« July 2016