Monday Jan 21, 2013

Solaris 11 IPS Concepts, Issues, and Workarounds


Image Packaging System (IPS) is a single tier packaging architecture which in Oracle Solaris 11, and other Oracle Sun products such as Oracle Solaris Cluster 4.x, replaces the previous SVR4-based dual tier packaging and patching architecture.

IPS and its implementation in Solaris 11 has a number of significant advantages over the old SVR4-based architecture, including:

  • Monthly consolidated bug fix Support Repository Updates (SRUs) providing a regular predictable cadence of engineered together, tested-together bug fix release baselines in contrast to the almost daily ad hoc patch release previously, eliminating the need to manage which patches out of a population of thousands should be downloaded and installed on which systems.  Simply 'pkg update entire' to an SRU baseline.
  • The package systems is a first class citizen of the Operating System, deeply integrated and leveraging ZFS Root (/), Boot Environments, SMF, Zones, and other Solaris technologies.  For example, the significant advantages in Boot Environment cloning (snapshots) enabling low-overhead, rapid, backups/restores, thanks to the mandatory ZFS Root (/) filesystem and associated commands such as 'beadm'.  This is like a much more integrated, slicker, version of the old Live Upgrade technology used in Solaris 10 and below.
  • The consolidation of multiple packaging and patching commands into a single, functionally rich, 'pkg' command.
  • Install groups defining the packages needed for common Use Cases, currently: solaris-small-server, solaris-large-server, and solaris-desktop
  • Incorporations defining a functional surface or baseline, specifying the versions of particular packages which were engineered together and tested together to provide a defined set of feature and bug fix functionality
  • The replacement of free format patch install scripts, which were a common source of error, with predefined actions

As we get used to Solaris 11 and IPS, it's natural that users will encounter some issues.

As a novice user myself, I've documented here some of the more common Solaris 11 / IPS issues which I've come across over the past year.  I plan to update it with additional items as they arise.

This is not designed to be an exhaustive list, but rather the "gottchas" which temporarily stumped either myself (easy to do!) or other non-IPS-expert colleagues.

Some of the "issues" are more to do with users getting used to conceptual changes. 

Some are Caveats resulting from bugs or sub-optimal choices made in early releases.  While these have been fixed, their residual impact may still be felt on systems with the affected software installed.

Much of the solutions knowledge below is thanks to two Solaris 11 IPS-expert colleagues of mine, Pete Dennis and Albert White, who I've been pestering unmercifully about IPS issues over the past year.  It was either that, or I'd have to RTFM!

If you're looking to update from Solaris 11/11 to Solaris 11.1 or later, please read this article.


Repositories, Publishers, and the 'pkg' Command

The 'pkg' command is functionally rich.  See 'man pkg' and other documentation.  When installing or updating packages, it dynamically analyzes the constraints on the target system, including dependencies and other factors defining what may be installed.

IPS is network repository based.

It is expected that most production customers will set-up their own repository behind their firewall and update it periodically with content from the Support Repository published by Oracle.

Many issues where 'pkg' is unable resolve all constraints imposed on a system, is because the required package versions are not available from the Repositories specified.

Sometimes, it is not immediately obvious why a particular package version is required to resolve a constraint, which can leave users scratching their heads.

Therefore, when a 'pkg install' or 'pkg update' command does not provide the anticipated results, check the specified Publishers (i.e. which Repositories are available to that system) and the content of those Repositories.

For example, Solaris 11 bug fix updates are provided by Support Repository Updates (SRUs) which are released monthly.  They contain only the incremental changes relative to their base release, e.g. Solaris 11.1.  They are designed to be used in conjunction with a Repository containing that base release. 

If the system is already installed with that base release, and the user is just updating existing installed packages, as opposed to installing additional packages, then the user can often get away with just using the SRU on its own.

However, if a bug fix in the SRU has added a dependency on a package which is not installed on the target system, and that package is in the base release rather than the SRU, then an update to that SRU will fail if the base release is not available to enable the dependent package to be pulled in and installed.

For example, a bug fix to the 'thunderbird' package in Solaris 11 11/11 SRU4 to fix font displays resulted in a new dependency being added to the Solaris 11 11/11 'fonts' package.  Since the 'fonts' package hadn't changed since the initial Solaris 11 11/11 release, it wasn't included in the SRU, so access to the base Solaris 11 11/11 release in a Repository was required to resolve the dependency.   There was a similar dependency addition in a later Solaris 11 11/11 SRU.

Similarly, if a Publisher is specified but is unavailable, or is not specified but is needed because that Repository contains a required package, then 'pkg' will be unable to resolve the constraints and will fail.

Making sure the correct Repository Publishers are defined and accessible, and the content of those Repositories is complete will resolve many package install and update issues.

Install Groups and Incorporations

The concept I've had the most difficulty getting straight in my own head is the relationship between Install Groups and Incorporations.

Install Groups simply specify a list of packages to be installed for common Use Cases.  They do not specify the versions of packages to install.  Currently, the following Install Groups exist in Solaris 11:

  • solaris-small-server - the current minimum Install Group, for use by security conscious customers
  • solaris-large-server - a superset of solaris-small-server which includes additional useful Sys Admin utilities and network protocols
  • solaris-desktop - for use where Solaris will be providing a desktop environment to users

Note the Install Group names 'solaris-small-server' and 'solaris-large-server' have nothing to do with the size of the server, rather it's the size of the solaris footprint on the server.  Note also, that 'solaris-desktop' is not a superset of the other two.  See here for more information.

The use of Install Groups is not mandatory.  They are simply provided for ease of use. Additional packages can be specified in addition to these Install Groups, for example, to resolve application dependencies.

Incorporations specify the versions of packages which should be installed together to provide a set of functionality, called a surface.  Incorporations exist for various consolidated sub-components of Solaris 11, such as the 'osnet-incorporation' for the core Operating System and Networking:

gerryh@dublin:~$ pkg info osnet-incorporation                           
Name: consolidation/osnet/osnet-incorporation
Summary: OS/Net consolidation incorporation
Description: This incorporation constrains packages from the OS/Net
Category: Meta Packages/Incorporations
State: Installed
Publisher: solaris
Version: 0.5.11
Build Release: 5.11
Packaging Date: Wed Jan 02 19:28:00 2013
Size: 6.22 kB
FMRI: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-

The 'entire' Incorporation defines what constitutes the version of the entire Solaris Operating System, for example the 'entire' Solaris 11.1 SRU3.4 release:

gerryh@dublin:~$ pkg info entire                                        
Name: entire
Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.1 SRU 3.4).
Description: This package constrains system package versions to the same
                build.  WARNING: Proper system update and correct package
                selection depend on the presence of this incorporation.
                Removing this package will result in an unsupported system.  For
                more information see
Category: Meta Packages/Incorporations
State: Installed
Publisher: solaris
Version: 0.5.11 (Oracle Solaris 11.1 SRU 3.4)
Build Release: 5.11
Packaging Date: Wed Jan 02 19:31:02 2013
Size: 5.46 kB
FMRI: pkg://solaris/entire@0.5.11,5.11-

Removal of the Solaris 'entire' Incorporation is not supported.  Removing it would remove contraints on other Incorporations, allowing an untested mix of Solaris software versions on the system, potentially leading to unnecessary issues.

When installing a Solaris system, it is common to specify both an Install Group - i.e. which packages to install - and a version of the 'entire' Incorporation - i.e. which versions of those packages to install. 

For example, this could be specified in an AI (Automated Installer) manifest, along with any additional IPS products or packages required.  Here's part of an AI manifest my team uses to install SPARC SuperClusters with Solaris 11 11/11 SRU12.4 as well as other tools from a separate Exa-family tools Repository which is specifically for Engineered Systems:

<software type="IPS">
<publisher name="solaris">
<origin name=""/>
<publisher name="exa-family">
<origin name=""/>

<software_data action="install">


Now for the bit which always confuses me.  Strong coffee helps!:

Installing an Incorporation does not, by itself, install any packages.  Rather, the Incorporation specifies the constraints on package versions if they are present on the system.

So 'pkg install entire' on a bare metal system does nothing, unless other packages are specified upon which the constraints specified in the Incorporation are to operate - e.g. an Install Group package such as 'solaris-large-server'.  To show this:

# create a bare metal image to play with
$ pkg image-create -p bare_metal
# what is in this image:
$ cd bare_metal
$ pkg -R `pwd` list
pkg: no packages installed
# Install 'entire'
$ pkg -R `pwd` install --accept entire
Packages to install: 28
# It installed 28 packages! What are they ?
$ pkg -R `pwd` list
NAME (PUBLISHER)                                  VERSION      IFO
consolidation/SunVTS/SunVTS-incorporation 0.5.11-    i--
consolidation/X/X-incorporation 0.5.11- i--
consolidation/admin/admin-incorporation           0.5.11-     i--
consolidation/xvm/xvm-incorporation               0.5.11-     i--
entire 0.5.11-    i--
# These are all Incorporations specified by 'entire'.  There is no software payload installed at all.  

But once Solaris is installed on a system, updating an Incorporation, for example, using 'pkg update entire', updates the constraints, causing the relevant packages which are installed to be updated by IPS to the later functional 'surface' specified by the Incorporation.

So if, for example, the new version of the Incorporation specifies 'foo@1.24' and specifies it has a new dependency on 'bar@1.13' and package 'foo' is already installed, say @ Version 1.20, then updating the Incorporation tells IPS to update 'foo' to Version 1.24 and install 'bar' at Version 1.13 if it hasn't already been installed (from whichever specified Repository/Repositories contains these packages at these versions).

Ask and You Shall Receive - In Abundance!

You can have too much of a good thing.  Like information.  Which can make it hard to see the wood for the trees when trying to debug a 'pkg' issue.

When issues occur, 'pkg' is verbose in its output about the problem.

Packages will have dependencies upon other packages.  These dependencies may not only be satisfied by the explicit version mentioned but also by any later version of a package.

This means that if 'pkg' is unable to solve for all dependencies given the available Publishers specified, the contents of those Repositories, and the constraints specified for the target system, then 'pkg' will produce a list of all the dependencies that could not be satisfied. While these errors are all true, due to the amount of them, they can freak out the user (they do me!), obscuring the underlying issue.

One way to reduce the amount of errors is to specify the version of the packages that you wish to update to.

This is because, by default, 'pkg' will attempt to move to the latest set of packages. If this update fails then it will recurse through all other permutations, producing errors for each possible set of packages for which it attempted to resolve constraints.

By specifying an explicit version of the packages to update then the errors produced will be just for that particular version.

Therefore, rather than just saying:

$ pkg update entire

...explicitly state the FMRI string of the SRU you want to update to...

$ pkg update entire@0.5.11,5.11-

...which specifies Solaris 11 11/11 SRU 12.4.

There's a couple of other good reasons to explicitly specify which SRU or package version you want to update to.

Firstly, if you don't specify a version, 'pkg' will try to update to the latest version which satisfies the constraints on the target system.

If the repository has been updated, this could produce a different result than the same command issued prior to the repository been updated. This may be undesirable if you are trying to update a number of systems to a homogeneous SRU level.

Secondly, if 'pkg' is unable to update to the latest available release due to the constraints on the target system, it will recursively try to update to a version higher than what is already installed.

For example, one of my team issued a 'pkg update entire' to update a test system to SRU4. Only days later when he realized that the test system didn't appear to have the expected bug fixes, did he discover it had actually updated the system to SRU3 as there was a constraint which prevented IPS updating the system to SRU4.

Since IPS is not telepathic, it's best to explicitly state what version you want it to update a system to.

All 'pkg' commands are logged.

The use of 'pkg history' is useful to examine the history of the system.  Additionally, it can be used to print out the previous errors messages without having to rerun a command that you know is going to fail.

Since 'pkg history' can be verbose, it's best to first identify when the error occurred and drill down on that specific invocation. 

For example, if you think the failure occurred within the last 5 invocations of the 'pkg' command then run:

# pkg history -n 5

START                    OPERATION                      CLIENT OUTCOME
2013-01-18T10:28:04      update                           pkg   Succeeded
2013-01-18T10:28:07      refresh-publishers         pkg   Succeeded
2013-01-18T10:28:24      rebuild-image-catalogs  pkg   Succeeded
2013-01-22T14:39:55      install                            pkg   Failed
2013-01-22T14:40:51      install                            pkg   Succeeded

Now look at the command that failed using the -l and -t options:

# pkg history -l -t 2013-01-22T14:39:55
Operation: install
Outcome: Failed
Reason: Constrained
Client: pkg
Version: 907fe02baa47
User: root (0)
Boot Env.: s11.1sru341-reprise
Boot Env. UUID: 6c841d3c-7d0a-c42f-b480-b53bfb0c265e
New Boot Env.: None
New Boot Env. UUID: (None)
Snapshot: (None)
Start Time: 2013-01-22T14:39:55
End Time: 2013-01-22T14:40:06
Total Time: 0:00:11
Command: /usr/bin/pkg install pkg://solaris/system/kernel@0.5.11,5.11-
Release Notes: No
Start State: None
End State: None
Traceback (most recent call last):
File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 1079, in __plan_op self._img.make_install_plan(**kwargs)
File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 4288, in make_install_plan reject_list=reject_list) File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 4249, in __make_plan_common ip.plan_install(**kwargs)
File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 419, in plan_install reject_list=reject_list)
File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 395, in __plan_install reject_list=reject_list) File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 370, in __plan_install_solver ignore_inst_parent_deps=ignore_inst_parent_deps)
File "/usr/lib/python2.6/vendor-packages/pkg/client/", line 442, in solve_install no_version=ret, solver_errors=solver_errors)

PlanCreationException: No matching version of system/kernel can be installed:

Reject: pkg://solaris/system/kernel@0.5.11,5.11-

Reason:  Newer version pkg://solaris/system/kernel@0.5.11,5.11- is already installed

This version is excluded by installed incorporation pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-

All errors are indented with, in most cases, the significant error being indented furthest to the right hand side.

In the above example, the requested Kernel can't be installed because a later revision is already installed and it's constrained by the 'osnet-incorporation' so it can't be "down-rev'd" to an earlier version.

Therefore, when investigating an issue, use the 'pkg history' command to print out the previous errors and look at the errors that are indented to the right.

Note that the errors themselves may be repeated for each and every package that the update has failed on, so the output may still be verbose.

But the errors are typically caused by just one or two issues, such as having the incorrect Publishers specified (too few, too many, or not accessible) or insufficient content in the Repositories. 

Solaris 11 Release and Support Repository Relationship

The Release Repository contains just the Solaris Releases such as the original Solaris 11 11/11 release and the Solaris 11.1 update release.

The Support Repository is only available to customers with a valid support contract.  It contains all releases, including all monthly Support Repository Updates (SRUs) providing bug fix updates to support contract customers.

As discussed in my previous blog posting, we've implemented a process improvement in Solaris 11 to remove any 'blackout' period on the release of bug fixes by tweaking the relationship between Update releases and bug fix releases, compared to Solaris 10 and older.

We still produce periodic Update releases such as Solaris 11.1, containing support for new hardware and enhanced software features (e.g. VM2.0). 

Update releases also contain a significant number of bug fixes for issues found internally during Solaris testing and more complex customer reported issues which required more test "soak" time than is possible in an SRU. 

Update releases are intensely tested and hence provide high quality Solaris Baselines. 

The Solaris Binary Compatibility Guarantee applies, so users should not experience any compatibility issues crossing a Solaris Update boundary. 

The Release Notes for the Update will give at least 12 months notice of any interfaces which will be deprecated.

Support Repository Updates (SRUs) primarily deliver bug fixes, although they may include some feature enhancements. 

They too, are intensely tested prior to release. 

SRUs go through several internal builds prior to release. 

Once released, additional critical bug fixes can be "back-published" to SRUs. 

The build number of the SRU is now included in its name to uniquely identify it, e.g. Solaris 11 11/11 SRU13.4 is Build 4 of SRU13 on top of Solaris 11 11/11. 

Earlier SRUs were documented with a letter suffix to denote "back-published" additional content, e.g. Solaris 11 11/11 SRU2a.

We've improved the process in Solaris 11 so that we can continue to deliver bug fixes for critical issues in SRUs while the content for an Update release is being finalized.

This implies that an Update release may not be a superset of the SRU(s) immediately preceding it. 

Rather, it is the SRU after the Update release which is effectively the superset of both the Update the SRUs preceding the Update release.

The relationship between Update releases and SRUs can be drawn as follows:

Solaris 11 11/11                              Solaris 11.1                                                                    Solaris 11.2...

     \                                                                  \                                                                                 \

SRU1, SRU1a, SRU2, ... SRU12.4, SRU 13.4, SRU1.4, SRU2.5, SRU3.4, SRU3.4.1, ...

Installed systems with a valid support contract should always be updated using SRUs from the Support Repository.

The SRUs are contiguous, just as Kernel patches were contiguous in Solaris 10 and earlier.  That is, the next SRU after Solaris 11 11/11 SRU13.4 is Solaris 11.1 SRU1.4. 

It is important to understand that this is no different to the Kernel PatchID progression in Solaris 10 and earlier releases whereby Kernel patches released after an Update release depended upon the Kernel patch from the Update, which contained feature code from that Update.

The only difference is that that lineage is a little more transparent in Solaris 11 due to the naming of the SRUs.

Known Caveats

As developers, reviewers, and release engineers have become used to Image Packaging System and the Solaris 11 eco-system, the number of bugs, "features", and caveats caused by inexperience continue to diminish.

Nevertheless, users may be impacted by the residual effects of some of these items.

Here's a non-exhaustive list of "features", potential issues, and their workarounds:

Need to accept Java 7 license

Oracle's Legal department insist that users explicitly accept the revised license terms in Java 7.

This means that users must add "--accept" to 'pkg install' or 'pkg update' commands when moving to versions with revised license terms.  For example:

$ pkg install --accept entire

Incorrect architecture packages may be present in early Solaris 11 11/11 installations

An IPS 'pkg' bug in early Solaris 11 11/11 versions can result in some x86 packages being installed on SPARC systems and vice versa due to the incorrect resolution of indirect dependencies.  This is now fixed.

There are several methods to remove the residual effects of the issue on early Solaris 11 11/11 installations.

Until the erroneous architecture packages are removed, errors similar to the following may be displayed when updating:

Plan Creation: Package solver is unable to compute solution.
Dependency analysis is unable to determine exact cause.
Try specifying expected results to obtain more detailed error messages.
Include specific version of packages you wish installed.

Note, the above is a rather generic error indicating the package solver couldn't compute a solution, so not all instances of the above error message may be due to this particular issue.  But for those which are, here's the options to resolve it:

Option 1:

The 'pkg' version delivered in Solaris 11 11/11 SRU 10.5, SRU 11.4, SRU 12.4, and SRU 13.4 contain functionality to remove the residual effects of the issue - i.e. remove the incorrect architecture packages.

Users can perform a "bunny hop" update to SRU 10.5, SRU 11.4, SRU 12.4, or SRU 13.4 prior to updating to a Solaris 11.1 SRU.

Indeed, simply updating the 'pkg' package itself is sufficient:

# pkg update
WARNING: pkg(5) appears to be out of date, and should be updated before
running update.  Please update pkg(5) by executing 'pkg install
pkg:/package/pkg' as a privileged user and then retry the update.

# pkg update package/pkg
Packages to remove:  1
Create boot environment: No
Create backup boot environment: No

PHASE                                        ACTIONS
Removal Phase                                  13/13

PHASE                                          ITEMS
Package State Update Phase               1/1
Package Cache Update Phase             1/1
Image State Update Phase                  2/2

The first command shows that the 'pkg' client has detected an error and outputs a message to fix it by running 'pkg update package/pkg'.

Running this command removes the incorrectly installed packages - in the above example, it was the 'ldoms-incorporation' on an x86 system. 

Option 2:

On SPARC systems, the 'xsvc' and 'nvidia-incorporation' x86 packages may be installed.  Since they were introduced via indirect requirements from other packages, such as the optional 'hmp-tools' package on SPARC or 'ldoms-incorporation' on x86, an alternative resolution is to remove the package with the dependency which will also remove the erroneous packages if nothing else depends upon them.  For example:

root@foobar:~# pkg -R /a/test2 uninstall -v

Packages to remove: 3
Estimated space available: 103.67 GB
Estimated space to be consumed: 19.20 MB
Create boot environment: No
Create backup boot environment: No
Rebuild boot archive: Yes

Changed packages:
system/management/hmp/hmp-tools,5.11-1:20120314T235822Z -> None
0.5.11,5.11- -> None
0.5.11,5.11- -> None

Removal Phase 89/89

Package State Update Phase 3/3
Package Cache Update Phase 3/3
Image State Update Phase 2/2

Once the erroneous architecture packages have been removed, you can update the system as normal.

Multiple versions of 'cacao'

Oracle Solaris delivers the 'cacao' package.  It's version is constrained by the 'cacao-incorporation'.

Oracle Enterprise Manager Ops Center also delivers the 'cacao' package.  Rather than working with Solaris to update its 'cacao' version, or delivering its own version to a private location, early Ops Center versions on Solaris 11 updated the Solaris 'cacao' package to a level later than that contained in any Solaris release.

This was sub-optimal as it had the unintended consequence of effectively breaking Solaris updates as IPS found a version of 'cacao' installed on the target system which was later than any version available from the Solaris publisher.

The Solaris 'entire' Incorporation includes the Solaris 'cacao-incorporation' and that constrained 'cacao' to an earlier version than that installed by Ops Center, meaning IPS could not resolve the constraints, and hence could not update Solaris without user intervention.

The workaround for this, and other such issues, is to "unlock" the offending package(s) from their incorporation, allowing them to float free.  This is done by toggling the IPS 'facet.version-lock' facility to 'false':

gerryh@dublin:~$ pkg contents -m cacao-incorporation                     
set name=pkg.fmri value=pkg://solaris/consolidation/cacao/cacao-incorporation@0.5.11,5.11-
set name=pkg.summary value="cacao consolidation incorporation"
set name=pkg.description value="This incorporation constrains packages from the cacao consolidation."
set name=pkg.depend.install-hold value=core-os.cacao
set name=info.classification value="org.opensolaris.category.2008:Meta Packages/Incorporations"
set name=org.opensolaris.consolidation value=cacao
set name=variant.arch value=sparc value=i386
depend fmri=SUNWcacaort@0.5.11-0.133 type=incorporate
depend fmri=SUNWcacaodtrace@0.5.11-0.133 type=incorporate
depend facet.version-lock.library/cacao=true fmri=library/cacao@,5.11- type=incorporate
depend facet.version-lock.library/cacao/cacao-dtrace=true fmri=library/cacao/cacao-dtrace@,5.11- type=incorporate
depend fmri=SUNWcacaowsvr@0.5.11,5.11-0.166 type=incorporate
depend fmri=library/cacao/web-server@0.5.11,5.11-0.166 type=incorporate
signature 235c7674d821032ae3eeda280c7837d1f1f4fdb5 algorithm=rsa-sha256 chain="8e422c1bb80b05f08f7a849f3d7ae90a976e048e 754665e03bd28ef63b05a416073eb6d649624781" chain.chashes="083e40bb50e6964834ebfd3c66b8720b46028068 f85dabbb0d56b37de3c3de98663dd8f27a12ff8e" chain.csizes="1273 1326" chain.sizes="1773 2061" chash=05654e46fc5cac3b9b9bd11c39512bc92bc85089 pkg.csize=1281 pkg.size=1753 value=3d4ef4de2458c2f6f34e6e8b6fac08312e7b556e70b2af881e30e70912c335e5a39e526c9c662b71e88c8e293cd42d3e688d515fa961ceb876fbc9af5339123c923dc0fd81c8252c9d6098b5bb8bb886733e2b827445266d67ef888bc4f9b814d1b22eaea8758e767f21f2828bd7239797b503333d931f3a2fa2d8cf63ea560fb634228611ad61fc8a5401e0b49f3a54c6198cd712f3d6a97efdc5b11d8a94a160b47d30fd24da303a43bf500111b0bc13bc28d5d66440d50ce434e9002404b103c6e8c23f7a29bbe650002ccb9edffeced85b97656c1e06beb28f937ce054aaf3bb753ed84632a9956c30cbcb67fca1fef52d3c8b0af8a97c60aa85e871d725 version=0

Setting 'facet.version-lock' to 'false' tells IPS that the constraint on that package version can be ignored.  This enables the rest of the packages to be updated. 

Once the rest of Solaris has been updated, the package should typically be re-locked - i.e. set 'facet.version-lock' back to 'true', so that it will be updated along with the rest of the packages when future updates are performed (assuming the issue is transitory).  Failure to re-lock the package will leave it floating independently.

Only use the 'facet.version-lock' feature when you have good cause to do so and you are confident you understand what you are doing.

Some core packages do not have a 'facet.version-lock' and cannot be unlocked from their Incorporation as their version is considered integral to the correct operation of Solaris.

Boot-management Packages

Due to an unfortunate sequence of missteps in development, and in a rare set of circumstances only, users may experience issues updating pre-Solaris 11 11/11 SRU10.4 versions due to 'boot-management' package issues.

The 'boot-management' package was originally part of the 'install-incorporation' and needed to be moved to the 'osnet-incorporation' as part of the GRUB2 boot project.

In preparation for that work the 'boot-management' package was unincorporated from the 'install-incorporation'.

Unfortunately, it didn't get incorporated into the 'osnet-incorporation' until Solaris 11 11/11 SRU 10.4, which effectively allowed it to float free unconstrained in the interim.

Thus, if you have a system installed with an early Solaris 11 version, such as Solaris 11 11/11 SRU 2a, and want to update it to another pre-SRU 10.4 version such as SRU 5.5, but a later SRU version, e.g. SRU 13.4, is also in your Repository, then the latest GRUB2 boot-management package version available in your Repository will be installed, as there's no incorporation in SRU 5.5 constraining it to an earlier version:

/usr/bin/pkg update --accept --be-name=Solaris11-sru5.5

If you subsequently try to update to an SRU between SRU 10.4 and any SRU with an earlier version of the 'boot-management' package than has been installed, say, SRU 11.4, it'll fail, because from SRU 10.4 onwards the version of the 'boot-management' package is constrained.  In SRU 11.4, the 'boot-management' package version is constrained to an earlier version in the 'osnet-incorporation' than the SRU 13.4 version installed.  The error message will be similar to the following:


Reject: pkg://solaris/system/library/boot-management@0.5.11,5.11-

Reason: Excluded by proposed incorporation 'consolidation/osnet/osnet-incorporation'

Newer version pkg://solaris/system/library/boot-management@0.5.11,5.11- is already installed

Reject: pkg://solaris/system/library/boot-management@0.5.11,5.11-

Reason: Newer version pkg://solaris/system/library/boot-management@0.5.11,5.11- is already installed


The solution is actually quite simple. 

Since the 'boot-management' package on the target system installed with SRU 5.5 is not constrained by any incorporation in that SRU, simply "down-rev" the 'boot-management' package to the version in the SRU you wish to update to, e.g. SRU 11.4:

pkg update system/library/boot-management@0.5.11,5.11-

Now perform the update to the desired SRU, e.g. SRU 11.4, again:

pkg update entire@0.5.11,5.11-

Tuesday Oct 23, 2012

Solaris 11 SRU / Update relationship explained, and blackout period on delivery of new bug fixes eliminated

Relationship between SRUs and Update releases

As you may know, Support Repository Updates (SRUs) for Oracle Solaris 11 are released monthly and are available to customers with an appropriate support contract.  SRUs primarily deliver bug fixes.  They may also deliver low risk feature enhancements.

Solaris Updates are typically released once or twice a year, containing support for new hardware, new software feature enhancements, and all bug fixes available at the time the Update content was finalized.  They also contain a significant number of new bug fixes, for issues found internally in Oracle and complex customer bug fixes which  require significant "soak" time to ensure their efficacy prior to release.

Changes to SRU and Update Naming Conventions

We're changing the naming convention of Update releases from a date based format such as Oracle Solaris 10 8/11 to a simpler "dot" version numbering, e.g. Oracle Solaris 11.1. Oracle Solaris 11 11/11 (i.e. the initial Oracle Solaris 11 release) may be referred to as 11.0.

SRUs will simply be named as "" releases, e.g. Oracle Solaris 11.1.1, for SRU1 after Oracle Solaris 11.1.

Many Oracle products and infrastructure tools such as BugDB and MOS are tailored towards this "" style of release naming, so these name changes align Oracle Solaris with these conventions.

No Blackout Periods on Bug Fix Releases

The Oracle Solaris 11 release process has been enhanced to eliminate blackout periods on the delivery of new bug fixes to customers.

Previously, Oracle Solaris Updates were a superset of all preceding bug fix deliveries.  This made for a very simple update message - that which releases later is always a superset of that which was delivered previously.

However, it had a downside.  Once the contents of an Update release were frozen prior to release, the release of new bug fixes for customer issues was also frozen to maintain the Update's superset relationship.

Since the amount of change allowed into the final internal builds of an Update release is reduced to mitigate risk, this throttling back also impacted the release of new bug fixes to customers.

This meant that there was effectively a 6 to 9 week hiatus on the release of new bug fixes prior to the release of each Update.  That wasn't good for customers awaiting critical bug fixes.

We've eliminated this hiatus on the delivery of new bug fixes in Oracle Solaris 11 by allowing new bug fixes to continue to be released in SRUs even after the contents of the next Update release have been frozen.

The release of SRUs will remain contiguous, with the first SRU released after the Update release effectively being a superset of both the the Update release and all preceding SRUs*. 

That is, later SRUs are supersets of the content of previous SRUs.

Therefore, the progression path from the final SRUs prior to the Update release is to the first SRU after the Update release, rather than to the Update release itself.

The timeline / logical sequence of releases can be shown as follows:

Updates: 11.0                                                11.1                               11.2     etc.

                 \                                                         \                                    \

SRUs:       11.0.1, 11.0.2,...,11.0.12, 11.0.13, 11.1.1, 11.1.2,...,11.1.x, 11.2.1, etc.

For example, for systems with Oracle Solaris 11 11/11 SRU12.4 or later installed, the recommended update path is to Oracle Solaris 11.1.1 (i.e. SRU1 after Solaris 11.1) or later rather than to the Solaris 11.1 release itself.  This will ensure no bug fixes are "lost" during the update*.

If for any reason you do wish to update from SRU12.4 or later to the 11.1 release itself - for example to update a test system - the instructions to do so are in the SRU12.4 README,

For systems with Oracle Solaris 11 11/11 SRU11.4 or earlier installed, customers can update to either the 11.1 release or any 11.1 SRU as both will be supersets of their current version.  My colleague, Pete Dennis, explains the step-by-step process here.

Please do read the README of the SRU you are updating to, as it will contain important installation instructions which will save you time and effort.

*Nerdy details:

  • SRUs only contain the latest change delta relative to the Update on which they are based.  Their dependencies will, however, effectively pull in the Update content.  Customers maintaining a local Repo (e.g. behind their firewall), need to add both the 11.1 content and the relevant SRU content to their Repo, to enable the SRU's dependencies to be resolved.  Both will be available from the standard Support Repo and from MOS.  This is no different to existing SRUs for Oracle Solaris 11.0, whereby you may often get away with using just the SRU content to update, but the original 11.0 content may be needed in the Repo to resolve dependencies.
  • The following bug fixes in SRU12.4 are not in Oracle Solaris 11.1.  They'll be available in 11.1.1 (SRU1 for Oracle Solaris 11.1):
7166132 vim should be able to run its test suite
7190213 libibmad and associated files need to be delivered in an NGZ
7191495 mkisofs install is incomplete
7195687 Update fetchmail to version 6.3.22
7195704 Problem with utility/fetchmail
7196234 Problem with network/dns
7197223 vim shows high CPU usage when editing dtrace script with syntax
        highlighting enabled
7071362 tcp_icmp_source_quench and other tunables may no longer be field
7181137 sol_umad should allow userland MAD operations in NGZs
7196540 After 7174929 integration 0.9.0 is shown for first disk in
        secondRAID volume

Tuesday Jun 19, 2012

Setting Up and Managing Local IPS Repositories

My colleague, Albert White, has published a useful article detailing how to set up local IPS repositories for use within an enterprise: How to Create Multiple Internal Repositories for Oracle Solaris 11

This is useful as most servers will not be directly connected to the Internet and most customers will want to control which Oracle Solaris SRUs (Support Repository Updates) are "qualified" for deployment within their organization.  Setting up and managing Internal IPS (Image Packaging System) Repositories is the way to do this.

The concept can naturally be extended and adapted.  For example, Albert talks about a "Development" Repo containing the latest Oracle Solaris 11 deliverables.  When qualifying a software level for deployment across the enterprise, a copy of a specific level could be taken, e.g. "GoldenImage2012Q3" or "SRU8.5", and once it passes testing, be used to deploy across the enterprise.

Best Wishes,


Thursday Apr 12, 2012

How To Update Oracle Solaris 11

My colleague, Glynn Foster, has published a nice article on how to update Oracle Solaris 11 which I think you may find interesting.

This blog is to inform customers about Solaris 11 maintenance best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. Gerry Haskins, Director, Software Lifecycle Engineering


« April 2014