Friday Sep 26, 2014

Solaris SRUs, patches, and IDRs available on MOS for bash vulnerabilities CVE-2014-6271, CVE-2014-7169, CVE-2014-7186, CVE-2014-7187

SRUs, Patches, and IDRs (Interim Diagnostics & Relief) are available from My Oracle Support, for all supported Solaris releases to address the recent critical bash vulnerabilities, CVE-2014-6271, CVE-2014-7169.

Newer IDR revisions are available on MOS which additionally address the less critical "mop up" vulnerabilities, CVE-2014-7186, CVE-2014-7187.  Patches and SRUs will follow for these too.

See MOS Doc ID 1930090.1 for details.

Many thanks to the folks around the globe who have been working tirelessly over the last 48 hours to code, test, and release these SRUs, patches, and IDRs - from Australia to India to the Czech Republic to Ireland and the US.

I sincerely apologise for the delay in proactively communicating these fixes to you.   That was outside of my control.

Best Wishes,


Friday Feb 07, 2014

Heads up! Regression in Solaris 10 Kernel patch 15040[01]-0[67] - now fixed in 15040[01]-09

Update:  Bug 17628036 is fixed in Solaris 10 patch 15040[01]-09 and Solaris 11.1 SRU16, both of which are now available on MOS.  Customers with 15040[01]-0[78] installed are recommended to install 15040[01]-09.  The Solaris 10 January CPU (Critical Patch Update) has been respun to included 15040[01]-09 and is available from MOS: (SPARC) (x86)
The issue is far less common on Solaris 11, so you only need to update to SRU16 if you experience the issue.  There will be a further "mop-up" fix in 15040[01]-10 and a later SRU, but 15040[01]-09 resolves the main issue and should be sufficient for most customers (addresses 95%+ of the instances reported).

Original post:

Please note that there's an issue with revs -06 and -07 of the Solaris 10 Kernel patches 15040[01].

Please see Sun Alert 1619580.1 on MOS for further details.  A number of Solaris 10 customers have hit the 2nd of the reported issues. 

We've respun the Solaris 10 January CPU (Critical Patch Update) to revert to rev-05 (now available from MOS), we're expediting a fixed rev-09 (rev-08 won't be released), and have withdrawn revs -06 and -07 from release.  Update: The Solaris 10 January CPU and Recommended patchset have been respun to include the fixed rev-09.

15040[01]-09 will address Bug 17628036 and the current ETA for expedited release is Feb 21.

The Solaris 11 fix is also being expedited in Solaris 11.1 SRU16, but seems to be less prone to the issue.

I apologize for the inconvenience caused.

Thursday Sep 12, 2013

Solaris 10 Patches Now On Monthly Release Cadence

(Updated Nov 25, 2013)

We've recently moved to a monthly release cadence for Solaris 10 OS patches.

New Solaris 10 OS patches are now available from MOS by the Tuesday closest to 17th of each month. 

The updated Solaris 10 OS Recommended Patchset will be available by the next day, Wednesday, assuming there are new patches released which meet its inclusion criteria - that is, patches which address security or other critical issues.

This enables customers to predict patch release dates and schedule maintenance windows.

This is similar to the monthly release cadence for Solaris Repository Updates (SRUs) for Solaris 11.

Please note that the Solaris 10 OS Recommended Patchset may not be updated every month.  This is because in some months there may be no new patches meeting the inclusion criteria.  That is, patches which address security, availability, data corruption, or other critical issues.

Thursday Jun 06, 2013

Next Solaris 10 Kernel PatchIDs, 150400 (SPARC) & 150401 (x86)

As I've noted in an update to my previous blog posting, Murphy's Law strikes again!

No sooner had I written that Solaris 10 Kernel PatchIDs 148888-xx (SPARC) and 148889-xx (x86) were here to stay for the foreseeable future, than the integration of the SR-IOV feature into rev-04 of these patches made it prudent to rejuvenate them. 

So from July 2013, the Solaris 10 Kernel PatchIDs will change to be 150400-xx (SPARC) and 150401-xx (x86).

See here for the full Solaris 10 Kernel PatchID sequence.

Tuesday Nov 08, 2011

Solaris 11 released, 2 days early

Today, we launch Solaris 11 in New York City.

Work on Solaris 11 started 7 years ago, as soon as Solaris 10 reached "code freeze".

About 6 years ago in a Solaris P-Team (Product Team) meeting, someone raised the repeatedly asked question as to when we planned to release Solaris 11.

A slightly exasperated Jeff Jackson said 11/11/11, half jokingly, half seriously.  It made sense.  It was in the right ballpark considering all the radical changes the architects wanted to make in Solaris 11.  And what better date to launch Solaris 11 ?

175 bi-weekly builds and two release candidate respins later, and we're releasing Solaris "Nevada" build snv_175b, officially known as Solaris 11.  But 2 days early.  Ooops!  I must admit to having been tempted to file a "Stopper" bug to cause enough of a smoke screeen to delay the release by two days.  But early is good.  So 11/9/2011 it is.

The Solaris 11 Tech Lead, David Comay, has posted some excuses - er, I mean "reasons" - on his blog as to why we're releasing 2 days early.  See for further information.

Having arrived in New York Tuesday afternoon, I went to the 9/11 memorial to pay my respects. 

May I just say, well done New York!  Well done America! 

It's a truly excellent and moving memorial.  The sound of the water falling and the patterns it makes as it falls into the abyss in the center of the very footprint where the twin towers stood is poignant symbolism.  It's impossible not to be moved.

And the fact that all around the memorial is still a construction site, with all the sounds of rebuilding what was destroyed, is very apt indeed.

Evil will not triumph.  Good will overcome.

It puts our humble efforts in stark perspective.

I hope you enjoy Solaris 11.  It's our most radical Solaris release since SunOS 2.0.  Virtualization built in.  Cloud built in.  Architected for maintainability.  Scalability beyond your imagination (and mine!). 

I'll be presenting an updated version of my Solaris 11 Customer Maintenance Lifecycle presentation at the DOAG (Deutsche Oracle Anwender Gruppe) Conference in Nuremberg, Germany, next week.  I hope to meet some of you there.

I'll then post the presentation here on my blog.

Let the fun begin!  Enjoy!

Best Wishes,


Wednesday Oct 05, 2011

Solaris 10 8/11 (Update 10) Patchset now available

Hi Folks,

The Solaris 10 8/11 (Update 10) patchset is now available from My Oracle Support.  Here's direct links to the common README and the SPARC and x86 downloads.  You need to be logged into MOS and have a valid support contract associated with your account in order to download the patchsets.

BTW: Please see my previous blog posting for details on other useful direct links to Solaris patch downloads and metadata.

As you may know by now, these patchsets will bring all pre-existing packages up to the same software level as the corresponding Solaris Update.  For example, all ZFS and Zones functionality is entirely contained in pre-existing packages, so applying the patchset will provide all the ZFS and Zones functionality and bug fixes contained in the corresponding Solaris Update.  

When we release the Solaris Update patchset, we try to fix any serious late breaking issues found with the corresponding Solaris Update patchset.  A list of additional patches added and the Caveats they address is contained in the patchset README.

Applying the patchset is not the same as upgrading to the Solaris Update release, as the patchset will not include any new packages introduced in the Solaris Update or any obsolete packages deleted in the Update.   

Please see this blog posting for lists of the new packages introduced in each Solaris Update to see if any of them are relevant to you.  If they are, then upgrade to a release which provides them.  If they're not, then applying the patchset may be a reasonable alternative to update your Solaris system. 

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  They are listed at the end of the SPARC and x86 patch lists.

Health Warning: Do not manually apply packages from a later Solaris release to an earlier Solaris release (e.g. by pulling individual packages from an ISO image) as this will result in an inconsistent system state which may lead to system corruption unless careful post-processing is done at the time such packages are applied to ensure that any patches applied to either the pre-existing packages on the system or pre-applied to the new packages been added are reapplied to the system to ensure both the pre-existing and new packages are at the same patch level.  Failure to do this will compromise the patch utilities ability to resolve patch dependencies leading to undefined results.  Even if you take the above steps, Support are likely to frown upon such shenanigans.  So don't do it.  If you need new packages, upgrade to a release which provides them.  Note, Live Upgrade packages are the only exception to this rule and the procedure for them is specified in the Live Upgrade documentation.  

Best Wishes,


Wednesday Sep 14, 2011

Useful Oracle Sun patch download options, including metadata & READMEs

(Updated May 14, 2013)

Here are some Oracle Sun patch download options which my colleague Don O'Malley and I believe you may find useful:

You can download an Oracle Sun patch README simply by using an URI of the following form:

Just replace the PatchID in the URI above with the PatchID you are interested in.

If you are logged on to MOS, and have a valid support contract associated with your account, you can download patches using an URI of the following form for an individual patch:

XML metadata for a patch is available using a URI of the form:

This XML metadata contains useful information like:

  • The MD5 and SHA-1 checksums, see <digest type=...>.  Getting MD5 and SHA-1 checksums directly from MOS or this XML metadata file is the most accurate way to get checksum information. 
  • The latest PatchID in this lineage which obsoletes (supersedes) this patch revision , see <patch_replacements> - in this example 127127-11
  • What bug fixes (CRs) are delivered in the Patch - note if <fixes_bugs truncated="yes">, then the list of CRs fixed in truncated, so see the patch README for the full list of CRs
  • What access entitlement is needed to download this patch - in this example "OS" (Operating System) which means you need a support contract which covers Solaris in order to download it.  Other common access entitlements are "FMW" (Firmware) and "SW" ([other] Software), which means you need a support contract which covers Hardware or other Software respectively.  If multiple access entitlements are shown, then a support contract which covers any of them is sufficient to download the patch.
  • The Oracle BugDB Bug number reference to this patch which can be used as an alternative way to access it (see example below) - in this example 9615556
  • The Oracle BugDB Bug number reference to the README of this patch which can be used as an alternative way to access it (see example below) - in this example 12450076

Note, there are two nearly identical <patch> entries in the XML Metadata file in this example, one for 32-bit and one for 64-bit.  This is common and occurs for the vast majority of Oracle Sun patches.  Java patches are the main exception to this multiple <patch> entries rule for Oracle Sun patches, as they produce a separate 64-bit patch which will have a separate metadata file.  Where multiple <patch> entries exist in a metadata file, they always refer to one and the same patch, so either metadata entry can be parsed.  So while the "aru" <request id> references in the URIs differ for each in addition to <platform>, it's the identical patch which is downloaded in each case.

It's also possible to access a nice landing page using the Oracle BugDB Bug number reference to a patch (taken from the XML Metadata file above) to construct a URI of the form:

The "View Digest" button on the landing page shows the MD4 and SHA-1 Checksums for the patch.  The landing page also facilitates viewing of the README and download of the patch.

The "HTML version" of the patch README can be accessed two ways: (using the PatchID) or (using the Oracle BugDB Bug number reference to the README taken from the XML Metadata file above)

Both of the above URIs resolve to the same patch README.  The "HTML version" of the README contains a download link for the patch at the top of the page.  It also provides links to two key resources for Oracle Sun patching information:

It's also possible to directly access the MOS Flash-based download page using a URI of the form:,(page=PatchDetailPage&id=gnrgyece(patchId=120068-02&patchType=Patch&patchName=120068-02))

Since patchsets are named a little differently, here's a table showing the relevant URIs for key patchsets:
Patchset Name
Landing Page
XML Metadata
Checksums (subset of XML Metadata)
Recommended OS Patchset for Solaris 10 SPARC
Landing Page README Download XML Metadata

Click "View Digest" on Landing Page or extract from XML Metadata

Recommended OS Patchset for Solaris 10 x86
Landing Page README Download XML Metadata

Click "View Digest" on Landing Page or extract from XML Metadata

Critical Patch Update (CPU) Patchset for Solaris 10 SPARC, Apr 2013
Landing Page README Download XML Metadata Checksums
Critical Patch Update (CPU) Patchset for Solaris 10 x86, Apr 2013
Landing Page README Download XML Metadata Checksums
Solaris 10 1/13 (Update 11) SPARC Patchset
Landing Page README

See Landing Page

XML Metadata Checksums
Solaris 10 1/13 (Update 10) x86 Patchset
Landing Page README See Landing Page XML Metadata Checksums
Here are some other useful links:
Sun Alerts - Knowledge article with summary of, and links to, all published Sun Alerts alerting customers to known Security (through the link to the "Critical Patch Update (CPU) and Security Alerts" page), Availability and Data Corruption issues
patchdiag.xref - metadata file listing latest available revision of all Oracle Sun 6-2 digit patches
withdrawn_patch_report - list of all Oracle Sun patches withdrawn from release in the last 12 months
weekly_patch_report - list of all Oracle Sun patches released in the last week

You can be proactively notified daily of Sun Alert issues (and other knowledge articles) by configuring the "Hot Topics" option in My Oracle Support:

   1. Go to url
   2. Sign in
   3. Select the tab "More..." --> Settings
   4. Select "Hot Topics E-Mail" on the left
   5. Update the Hot Topics Settings
         1. Toggle the E-Mail to 'On'
         2. Ensure set "Send Every 1 Days"
         3. Select desired format (text or HTML)
         4. Set the item limit to some number larger than 5 (suggest 25)
         5. Set Service Request to "None"
         6. leave "Product Bugs Marked as Favorites" deselected
   6. Add the needed Sun Alert Filter(s) ** Note: To receive all Sun Alerts, use the following filter **
   7. Select  "Add..." (new window will pop up)
         1. Add the Product "Solaris SPARC Operating System"
         2. Add the Platform "GENERIC (All Platforms)"
         3. Check the "Knowledge Articles" box
         4. Check the "Alerts" box
         5. Select "OK" (selection window closes)
   8. Select "Save"
         1. You should be able to see your Hot Topics filter you just set up.
   9. Log out of MOS

Finally, for details on how to script access to resources such as the URIs listed above, check out:

MOS - Using 'wget' to Automate Sun Patch Downloads

I'd like to thank my colleague, Don O'Malley, for researching much of the above. 

I hope you find this helpful.

Best Wishes,


Monday Aug 29, 2011

Using smpatch to apply Solaris Cluster patches and other enhancements

It is now possible again to use the in-built Solaris 10 patch automation utility, 'smpatch' / Update Manager, to download patches for products such as Oracle Solaris Cluster and Oracle Solaris Studio, as well as Oracle Solaris Operating System patches. 

It is now also possible again to use 'smpatch' / Update Manager on 3rd party hardware. 

To utilize these capabilities, the system must be registered or re-registered as outlined in

These steps effectively switch 'smpatch' / Update Manager from using hardware serial number based access entitlement to User based access entitlement, similar to the access entitlement mechanism used when downloading patches via 'wget' or manually via My Oracle Support (MOS).

The following patches are required to provide this functionality:

121118-19  SunOS 5.10: Update Connection System Client 1.0.19
123893-25  SunOS 5.10: Cacao Patch
123005-09  SunOS 5.10: Basic Registration Update
124171-08  SunOS 5.10: SCN Base cacao module patch
123630-04  SunOS 5.10: HTTP proxy settings patch
121119-19  SunOS 5.10_x86: Update Connection System Client 1.0.19
123896-25  SunOS 5.10_x86: Cacao Patch
123006-09  SunOS 5.10_x86: Basic Registration Update
124187-08  SunOS 5.10_x86: SCN Base cacao module patch
123631-04  SunOS 5.10_x86: HTTP proxy settings patch

'smpatch' / Update Manager patch 12111[89]-19 introduces other significant changes due to the migration to Oracle back-end infrastructure.  The download server and security certs have changed.  As My Oracle Support supports ".zip" file download only, this patch mandatorily migrates 'smpatch' / Update Manager from using ".jar" downloads to using ".zip" downloads.

Caveat: There is currently an issue affecting LPS (Local Proxy Server) functionality following the migration to the Oracle back-end infrastructure.  This issue is currently being worked on.

Thursday Aug 11, 2011

Applying the latest Solaris patches using Ops Center Enterprise Manager

A couple of customers have claimed to me that it's not possible to apply all the latest available Solaris patches using Ops Center Enterprise Manager.  I've checked with my colleagues in Ops Center, and it most certainly is possible.  Here's one way to do it: 

There are multiple ways to perform this task ...

Here is one using the "Report" feature:

1) Select "Host Compliance Report"

2) Use the default setting "Security and Bug fixes" and select proposed target system or group of targets:

3) The Report will show all downrev packages (e.g 824 pkgs) and will allow you to submit a job, that's all that's needed.

Looking at the Job log we can see:

# tail /var/scn/update-agent/logs/resolve.log

add 40025552 (145497-01)

add 40025545 (144998-03)

add 40025534 (145501-01)

add 40025472 (118712-24)

add 40025471 (121734-13)

add 40025380 (118777-17)

add 40024414 (125060-07)

add 40022356 (119788-10)

add 40015326 (121081-08)

Total number of sorted operations : 197

So in total we would install 197 patches.

Best Wishes,


Saturday Jul 02, 2011

A Solaris Recommended Patchset to bind them all

I've long been of the opinion that there should be a single generic set of Solaris recommended patches which customers are consistently recommended to install in proactive maintenance windows for issue prevention. It's something I've been working towards for quite a while.

A collaborative effort between the Software Patch Services, Enterprise Installation Standards (EIS), Sun Risk Analysis System (SRAS) - now renamed Oracle Risk Analysis Services (ORAS) - and the Explominer team in the Oracle Solaris Technical Center (TSC), has achieved this goal with the creation of the Recommended Patchset for Solaris.  

Up until now, while the Solaris OS Recommended Patch Cluster was the core basis for Solaris patch recommendations, various teams tended to recommend their own favorite patches on top of this core set.  This wasn't just by whim.  Each team was looking at patching from a slightly different angle - for example various angles of proactive patching (issue prevention) versus reactive patching (issue correction).

The Recommended Patchset for Solaris is the result of the combined wisdom of the various teams.  It is designed for proactive patching (issue prevention).  The contents are generic and should be suitable for most customer configurations.  You should still read the README file and follow its instructions to ensure all of the patches included are appropriate to your specific environment.  You should test the patchset on a test system which closely mimics your production systems prior to deployment. 

You may still legitimately be asked by support to install additional patches to fix issues specific to your environment in reactive maintenance situations (issue correction).  But this should only be after due diligence to ensure that such patches are likely to fix the specific issue encountered.

The Recommended Patchset for Solaris is the new name for the Solaris OS Recommended Patch Cluster.  It's available from MOS (including 'wget'), EIS, Ops Center, etc.  We've changed the name to use the Oracle standard terminology "patchset".  I never liked the name Solaris Patch Cluster as there was a risk of it being confused with the Solaris Cluster product to which it bears no relation.  In due course other patch "clusters" and patch "bundles" are likely to transition to the name "patchset". 

The install script and code word needed to invoke it (which is contained in the README file) have been renamed to reflect the name change from "cluster" to "patchset". 

Customers who have installed the Solaris OS Recommended Patch Cluster may notice the additional patches included in the Recommended Patchset for Solaris the first time they install it.  After that, it'll be business as usual.  Many of these additional patches are already pre-applied into Solaris Update releases, so customers on later update releases should see little difference.

As before, the Recommended Patchset for Solaris will continue to be updated whenever a patch matching its inclusion criteria is released.  This can happen several times a month.  Just take the latest which matches your proactive maintenance window schedule. 

And as before, once a quarter, the Recommended Patchset for Solaris will be archived and renamed as the Critical Patch Update in line with standard Oracle practice.  (See previous blog postings.)

To create the Recommended Patchset for Solaris, we took the Solaris OS Recommended Patch Cluster and analyzed the additional Solaris patches which the Explominer team recommend be added on top of it for the monthly EIS patch baselines. Where those additional patches added real value - i.e. were of significant benefit to many customers - we added them to the recommended patch set.  Where they didn't add real value, we discarded them.  We then made sure that a system on which the resultant Recommended Patchset for Solaris was installed passed with a clean bill of health from the ORAS risk analysis audits.

So now, the Solaris OS patches in the EIS patch baselines will be the Recommended Patchset for Solaris with input from the Explominer and other teams included, and will be tested with ORAS.  These are the patch baselines available in Ops Center.  We have set up a panel of patch experts from the teams mentioned above to adjudicate on future potential additions to the Recommended Patchset for Solaris.

Previously, the criteria for including a patch in the Solaris OS Recommended Patch Cluster was quite strict: a patch had to address a Security, Data Corruption, or System Availability issue; be a patch utilities patch, or be required by the above.  In future, other patches which add real value for many customers may be included - for example, a patch for a commonly used driver which delivers significant performance improvements.  The goal remains the same - to include the most critical generic patches which we recommend customers install in proactive maintenance windows for issue prevention.

Additional patches outside of the patchset may still be required:

  • For other Oracle products - the Recommended Patchset for Solaris only includes Solaris Operating System patches.  Other products such as Oracle Solaris Cluster, Oracle Solaris Studio, Oracle Database, etc., may have their own patch recommendations.  The monthly EIS update includes patch sets for Oracle Solaris Cluster, SAMFS, QFS, and SunVTS in addition to the Recommended Patchset for Solaris.
  • For specific platforms - for example a Solaris driver patch if a particular network card is installed or where firmware updates are required
  • For specific configurations - for example if the system is connected to 3rd party storage solutions such as EMC Powerpath or Veritas
  • For specific issues in your configuration - for example, break/fix situations where an additional patch fixes the issue encountered

You can download the patchsets or view their Readmes directly, using the following links:

To downloads the patchsets (you must be logged into MOS):

To download the patchset Readme files (no need to be logged into MOS):

The above works for both flash and non-flash (html) MOS users.   Just substitute "9" for "10" to get the Solaris 9 Recommended patchsets and Readmes.

You can also download the patchsets using 'wget' for scripted access as normal.  (See previous blog postings.)  For example, the download filename for Recommended Patchset for Solaris 10 SPARC is still

If, like me, you like to know how to do things from first principles, here's the way to construct the search on My Oracle Support:

For Flash compatible systems (full function MOS version):

  1. Login to My Oracle Support (MOS),
  2. Click on the "Patches&Updates" tab
  3. Click on "Product or Family (Advanced Search)
  4. Type "Solaris Operating System" into the product search box
  5. Select the Releases you are interested in - e.g. Solaris 10 Operating System and Solaris 9 Operating System
  6. Select the Platforms you are interested in - e.g. Oracle Solaris on SPARC (64-bit) and Oracle Solaris on x86-64 (64-bit)
  7. Click on the "+" sign next at the end of the "Platforms" line to add additional search criteria
  8. Click of "Select Filter" and select "Type" from the drop-down menu
  9. Select "Patchset"
  10. Click "Search" 

For non-Flash users (html MOS version):

  1. Login to the html version of My Oracle Support,
  2. Click on the "Patches & Updates" tab
  3. Click on the Advanced Search tab in the search box
  4. Type "Solaris Operating System" in the product search box 
  5. Select the Releases you are interested in - e.g. Solaris 10 Operating System and Solaris 9 Operating System
  6. Select the Platforms you are interested in - e.g. Oracle Solaris on SPARC (64-bit) and Oracle Solaris on x86-64 (64-bit)
  7. For Type, select "Patchset"
  8. Click Search

MOS remembers your previous selections and they'll be shown top of each drop down menu on subsequent invocations.  You can also save searches for future re-use.

I want to thank Don O'Malley, Ed Clark, Howard Mills and the EIS team, Juergen Schleich and the Explominer team, Dr. Rex Martin and the ORAS team, and Rob Hulme and Walter Fisch from the Oracle Technical Support Center (TSC) for all their work in making a single consistent Recommended Patchset for Solaris a reality.

As always, I'm interested to hear your feedback.

Best Wishes,


Tuesday Oct 12, 2010

Oct 2010 Solaris OS CPU now available

The October 2010 Solaris OS CPU (Critical Patch Updates) containing all available Security, Data Corruption, and System Availability fixes are now available from My Oracle Support (MOS) and SunSolve.

See and in particular Document 1446032.1 on My Oracle Support (MOS),, which includes CVE mappings for Oracle Sun products. 

To access the Solaris OS CPUs on MOS, login, select the "Patches & Updates" tab and in the "Patch Search" box, click on "Product or Family (Advanced Search)".  Select "Solaris Operating System" from the product drop down menu, select the Release(s) you are interested in, e.g. "Solaris 10 Operating System", select "Type" and "Patchset" from the drop down menus on the next line, and click "Search".  This will show all the available patch clusters and bundles for your search criteria.  The October 2010 CPUs have titles of the form "CPU OS Cluster 2010/10".

The Solaris OS CPUs are archived copies of the Solaris OS Recommended Patch Clusters.  See for further details.

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Tuesday Jul 13, 2010

Solaris Critical Patch Updates (CPUs)

It's Oracle standard practice to release quarterly Critical Patch Updates (CPUs) containing security fixes.  These scheduled releases enable customers to plan maintenance windows.

Solaris now conforms to this practice and Solaris OS CPUs are now available.

The Solaris OS CPU is an archived snapshot of the Solaris OS Recommended Patch Cluster.

Please note that the Solaris OS bug fixing processes have not changed.  Security and other bugs continue to be fixed as soon as possible, patches containing such fixes for the Solaris OS will continue to be released as quickly as possible, and they will continue to be included in the Recommended Solaris OS Patch Clusters as soon as they become available. 

The Solaris OS CPU simply provides another, archived, patch collation option for customers.

See and in particular Document 1446032.1 on My Oracle Support (MOS),, which includes CVE mappings for Oracle Sun products. 


  1. The CPUs were created on July 6th and released on July 13th.
  2. Solaris 8 is in Vintage support so no patch clusters are updated for Solaris 8.  Instead, the above document lists Solaris 8 patches released in the last quarter which address Security issues.  A Solaris 8 Vintage support contract is needed to access some of them.
Update: CVE to patch mappings are now available for the Solaris CPU from July.  Please see

Thursday May 20, 2010

Merging the Solaris Recommended and Sun Alert Patch Clusters

The Solaris "Recommended" and Sun Alert Patch Clusters have been merged (June 4th 2010). 

The merged clusters are called the "Recommended OS Cluster Solaris <release> <architecture>", for example "Recommended OS Cluster Solaris 10 SPARC". 

The old "Recommended" and Sun Alert Patch Clusters only ever contained Solaris OS patches (with rare exceptions), so we've added "OS" to the new merged cluster name to make this a little clearer.

The merged Recommended OS Clusters have the same access entitlement as the old clusters - namely, you need a support contract which covers Solaris to access them.

The old "Recommended" patch cluster contains the latest revision of Solaris OS patches which fix Sun Alert issues (i.e. Security, Data Corruption, or System Availability issues).  That is, the top-of-tree patches which fix Sun Alert issues.

The Sun Alert patch cluster contains the minimum revision of Solaris OS patches which fix Sun Alert issues.  Thus, the Sun Alert patch cluster provides the minimum amount of change required to get all available Solaris OS fixes for Security, Data Corruption, and System Availability issues.

The contents for the two clusters are very similar, which causes unnecessary confusion as to which one to use.  When the Sun Alert Cluster was released several years ago, it should have replaced the older "Recommended" Cluster, and this merging of the Clusters is to correct that omission.

The inclusion criteria for the Sun Alert cluster is more logically correct, as in the Recommended Cluster there's no more value in adding the latest revision of a patch whose earlier revision provided a fix to a Sun Alert issue than in adding any other random patch.  Many folks assume "latest is greatest", and Oracle Sun wouldn't release a patch unless it is important, but this is slightly simplistic.  Change implies risk, and as many patches address issues which are only seen in very specific configurations, and while Oracle Sun patches are thoroughly tested prior to release, there is little advantage in taking more change than is necessary in minor maintenance windows or reactive patching situations.  Therefore, providing a minimal patch cluster which provides all available fixes for Solaris OS Sun Alert issues for use in minor maintenance windows makes sense.

The old "Recommended" Clusters were often updated several time a week, simply because a later revision of a patch whose earlier revision fixed a Sun Alert issue was released, even though the later revision didn't fix any additional Sun Alert issues.  Since the "Recommended" flag on SunSolve and in the patchdiag.xref metadata file matches the contents of the old "Recommended" Cluster, we were releasing many more patches which were flagged as "Recommended" than customers really needed to apply.

After the merge, new patches added to the Recommended OS Cluster and hence the "Recommended" flag on SunSolve and in the patchdiag.xref metadata file will be the specific revision of patches which address Sun Alert issues.  Only when an obsoleting patch provides a new fix to a Sun Alert issue will it be included and the obsolete patch removed.  The merged Recommended OS Clusters will update on the same cadence as the old Sun Alert clusters, which is typically about once a week for Solaris 10 (5.5 times a month, on average).  We will continue to update the merged Recommended OS Cluster whenever a patch matching the inclusion criteria is released.

To avoid the potential confusion which may be caused if we were to remove the "Recommended" flag from any patches, we will take the "Recommended" Cluster at the beginning of June 2010 as the basis for the merged cluster and then apply the Sun Alert Cluster inclusion criteria going forward.

The merged Recommended OS Cluster was initially released on June 4th, 2010.  The download link (target) file name of the merged cluster will be the same as the old "Recommended" Cluster, e.g., to minimize the changes users need to make to automated download scripts.

Customers who have traditionally downloaded the Sun Alert cluster will need to update download scripts to use the merged cluster file download names as the old Sun Alert cluster are no longer available.

In major maintenance windows, the Best Practice recommendation is to upgrade to the latest available Solaris Update release or at least to apply the equivalent Solaris Update Patch Bundle available from the patch cluster download page.  In both cases, the latest Recommended OS Cluster should also be applied as it will contain any additional Solaris OS Security, Data Corruption, and System Availability fixes released since the Solaris Update contents were finalized.  Solaris Updates are intensely tested, and hence this strategy provides a well tested, stable, and feature rich baseline for production systems.  In between major maintenance windows, the Best Practice recommendation is to try to keep as up to date as possible with the contents of the merged Recommended OS Cluster during minor maintenance windows.

Let's look at an example, to make the rationale for the change clearer: 

In the old model, if a security vulnerability in /usr/bin/ls is fixed in patch 123456-03, then both the old Recommended and Sun Alert clusters will initially include it.  If code interdependencies caused by subsequent code putbacks - e.g. the major Trusted Solaris Extensions feature - result in the contents of the "/usr/bin/ls" patch 123456-07 being accumulated into a feature Kernel patch associated with a Solaris 10 Update, e.g. 234567-14, then the old "Recommended" Cluster would include 234567-14 instead of 123456-03, even if 234567-14 contained no additional fixes for Sun Alert issues (i.e. Security, Data Corruption, or System Availability issues) compared to 123456-03.  The "Recommended" flag on SunSolve, in patchdiag.xref, and elsewhere would be updated every time a patch revision obsoletes the original patch, even though these later patch revisions contain no additional fixes to Sun Alert issues.  This can lead to customers who try to stay up to date with "Recommended" patches patching more content and potentially more often than is really necessary.  In contrast, 123456-03 would remain in the Sun Alert cluster for as long as no additional fixes for Sun Alert issues are contained in obsoleting patches.

In the new merged Recommended OS patch cluster model, while the starting point will be the old "Recommended" Cluster as of the start of June 2010 (to avoid dropping the "Recommended" from any patches, which might cause confusion), further changes to the cluster will follow the old Sun Alert cluster inclusion criteria - that is, the merged Recommended OS patch cluster contents and corresponding Recommended flag in SunSolve and patchdiag.xref will only be updated if a new patch delivers a new fix for a Sun Alert issue.   This means that only patches which we really recommend will be included in the Recommended OS patch cluster and flagged as Recommended in SunSolve and patchdiag.xref.  Since the rate of change will be less, it'll be easier for customers to see what's really recommended and allow more informed decisions regarding when to apply such patches.

Please note that this change has nothing whatsoever to do with the integration into Oracle.  This is an enhancement I've been looking to do for some time to avoid the confusion caused by having two very similar patch clusters and a corresponding "Recommended" flag which was updated much more frequently than was necessary.

My team has been working with known consumers of the "Recommended" patch flag such as TLP, Ops Center, 'smpatch', Update Manager, SRAS, EIS, and 'pca' to ensure that the transition goes smoothly.  

For example, TLP and 'pca' consume the patchdiag.xref file which up to now typically only contained entries for top-of-tree (latest) patch revisions.  From June 4th 2010, patchdiag.xref will contain whatever revision of a patch is flagged as "Recommended" as well as the top-of-tree patch revision.  Hence, a single base PatchID, e.g. 123456, may have two entries in the file, e.g. 123456-03 marked "R" for Recommended and "O" for Obsolete and 123456-08 which is the latest revision of that patch but which won't carry the "R" flag as it contains no additional Sun Alert fixes over rev-03.  

From my discussion with Martin Paul, author of 'pca', my understanding is that initially, he plans to propagate the "R" flag forward to the latest patch revision in his 'pca' metadata as currently 'pca' only handles the latest revision of patches, but he'll look at some stage in the future to leverage the more precise "Recommended" flag data we'll be providing with this change.

Friday Jan 29, 2010

Important new features in latest PatchFinder release

Firstly, please allow me to get something off my chest:


It's been a long wait and we're finally there!

I, for one, am tickled pink.

There's likely be a lot of changes for all of us in the coming months, some good, some maybe controversial to some folk, but I passionately believe that Oracle will bring much needed commercial sense which will ensure that Solaris and Sun-Oracle hardware continues to innovate like hell to provide the solutions you, our customers, need.  So strap yourselves in, the fun is about to begin!

But much more than the red Oracle logo has changed on PatchFinder today.

I want to let you know about two key new features which I believe significantly improve our customers' patch searching experience:

Search for Patches which deliver New Security Fixes 

The PatchFinder "Security Filter" now differentiates between patches which introduce a new security fix (shown by the "NS" symbol in search returns) and patches which simply deliver any security fix, either new or pre-existing (shown by the "S" symbol in search returns). 

Up until now only the latter was available, which made it difficult for customers to differentiate between patch revisions which deliver new security fixes and patch revisions which simply re-deliver old security fixes.

The "New Security Fix" search option under "Security Filter" should typically be used in combination with the "Show Obsolete" option so that you can see all patch revisions delivering new security fixes.  Otherwise you'll just see the subset of patches which are contain both new security fixes and are not obsoleted.

Solaris OS Patches which deliver (or redeliver) security fixes will continue to be added to the "Recommended" Patch Clusters as before, along with OS patches which deliver (or redeliver) Data Corruption or System Availability fixes, the latest patch utility patches, and any other patches required by the above.

Solaris OS Patches which deliver new security fixes will continue to be be added to the Sun Alert Patch Clusters as before, along with OS patches which deliver new Data Corruption or System Availability fixes, the latest patch utility patches, and any other patches required by the above.

But with this New Security Fix option in PatchFinder, you can now find all (6-2 digit PatchID) patches for all products which deliver new security fixes, not just Solaris OS patches.

BTW: This "New Security Fix" feature has actually been in PatchFinder since the last release in December, but this is the first opportunity I've had to blog about it.

Search for patches by the objects they deliver

You can now search for patches by the objects they deliver. 

For example, type "/usr/bin/vi" into the "File Included" search box, filter the search using the other search options if desired ( e.g. select "Solaris 10" under "OS Release" ), and PatchFinder will return the patches which deliver "/usr/bin/vi".  

This is useful if you are having problems with a particular utility or object and want to find if any patches are available for it.  Then reading the CR synopses listed in the README for the appropriate patches returned may help you figure out if the patch is likely to address the problem you are experiencing.

Try searching for "zoneadmd", or "genunix", for example.

Remember, if you enter something like "vi" or "ls" in the "File Included" search box, you'll get all objects which contain those strings in their pathnames, so a well qualified search such as "/usr/bin/vi" or "/usr/bin/ls" may be more useful.

Watch out for symlinks, e.g. on Solaris 10:

$ whence patchadd/usr/sbin/patchadd
$ ls -l /usr/sbin/patchadd
lrwxrwxrwx   1 root     root          16 May 15  2009 /usr/sbin/patchadd -> ../lib/patch/pdo\*
So on Solaris 10, search for "/usr/lib/patch" rather than "/usr/sbin/patchadd" to find patch utility patches.  FYI, 'pdo' is the preprocessor to 'patchadd' on Solaris 10 and both are contained in /usr/lib/patch.  Alternatively, just search for "patchadd".

I hope you find these new PatchFinder features useful.   A lot of work went in behind the scenes, especially on ensuring the accuracy of the "New Security Fix" flag.  I'd like to thank my colleagues, Brian, Julien, Slim, Mark, Don, and the rest of the team for making these enhancements a reality.  Nice work guys!

Friday Sep 18, 2009

Patching Pre-flight Checks (ppc) tool now available

The new Patching Pre-flight Checks ('ppc') tool is now available to all customers who have a support contract.

The idea for this tool comes directly from customer feedback.  

The customer wanted to reduce the cost of patching Solaris systems by enabling more junior Sys Admins to successfully patch Solaris 10 zones systems.   Their concern was that potential zones patching issues in versions of Solaris prior to Solaris 10 8/07 (Update 4) meant that they needed to assign senior System Administrators to patch such systems to identify and resolve potential issues.  

Furthermore, the customer was concerned that such issues had the potential to derail planned maintenance windows - for example, if during the patching session an unexpected issue was encountered and the patching session couldn't be completed as planned.

To address these concerns, my colleague, Ronan O'Connor, has written the Patching Pre-flight Checks tool, 'ppc'.  It can be run prior to a planned patching session to check that the target system is in a clean state ready for patching.

It's important to understand the scope of the tool.  It checks a target system (and a patch set, if supplied) for a variety of inconsistencies which could cause problems.

It looks for left over lock files from previously aborted patching or packaging operations, inconsistencies in the contents database, IDRs installed on the target system, zones "mountability", space issues, etc.  Some of these issues can occur on early versions of Solaris 10, particularly in a Zones environment.  Many of the underlying causes of such issues are fixed in the latest versions of the patch utility patches (119254 SPARC / 119255 x86), which is why we always recommend you apply the latest patch utility patches before applying other patches.

If you have a directory of patches to be applied, 'ppc' checks the integrity of those patches, and cross-checks whether any of the patches patch pkgs which have been locked down by any IDRs on the system and warns if there is a conflict.

The 'ppc' Release Notes provide information to help interpret the messages produced.

The idea is that 'ppc' can be run by a junior Sys Admin prior to a planned patching session, and any potential issues uncovered can then be analyzed by a more experienced Sys Admin.  This helps avoid nasty surprises during patches sessions and also helps to reduce the level of expertise required to patch Solaris systems, leading to cost savings for customers.

It is outside the scope of the 'ppc' tool to do root cause analysis of why the inconsistency arose or what actions may be needed, if any, to correct the situation.

If 'ppc' returns without noting any problems, you can be pretty confident that the patching session will succeed.  If 'ppc' notes potential issues, they can be investigated prior to the planned maintenance window.

The next version of 'ppc' will include a Zones consistency check to check that all zones are at a consistent patch level.   It will also contain a more sophisticated space checking algorithm.  There's no planned release date yet for Version "2.0" yet as we're awaiting feedback on Version 1.0.x first.

Some of the ideas in 'ppc' may find their way back into 'patchadd', although it's probably appropriate to keep 'ppc' as a separate tool.

You can download the Patching Pre-flight Checks tool, 'ppc', here.  It's an attachment to the knowledge article.  The Patching Pre-flight Checks tool, 'ppc', and the Customer Patch Forum are only available to customers with a support contract, so you'll need to login to SunSolve to access the knowledge article.

We're very interested in your feedback as to the usefulness of this tool and how you'd like to see 'ppc' develop going forward.

Many thanks to Ronan O'Connor for all his work on the tool!

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Friday Aug 14, 2009

Improvements to Solaris 10 Recommended and Sun Alert Patch Clusters released

My colleague, Ed Clark, has made significant improvements to the Solaris 10 Recommended and Sun Alert patch clusters.  These improvements have just been released and are in the current clusters available to contract customers from the Patch Cluster & Patch Bundle Downloads on SunSolve.

Ed's improvements include:

  • Filtering out "false negatives" from the patch utility return codes, so that if the cluster install script returns "1", you know you've got a real problem which needs investigating.   As you may know, the Solaris patch utility, 'patchadd', can return errors for some acceptable situations - for example, if the patch is already applied to the system, or a later revision of the patch or a patch which obsoletes it is already applied to the system, or none of the packages in the patch are on the target system (e.g. because a reduced Install Metacluster was used to install it or the system has been security hardened by package removal), etc.   Such conditions are acceptable "errors" which do not usually require further investigation by the user.  By filtering these conditions out, if the 'installcluster' script returns "1", you know it isn't because of one of these acceptable "errors", and therefore you need to look at the logfiles to find out what's gone wrong.  For further information, please see the cluster README and Analyzing a patchadd or patchrm Failure in the Solaris OS.
  • The new 'installcluster' script will exit as soon as it encounters an unexpected failure - i.e. not one of the acceptable "errors" mentioned above.  This prevents potentially compounding issues by attempting to apply further patches.
  • The new 'installcluster' script includes context intelligence for patching operations.   It informs the user when zones need to be halted, and it provides phased installation to handle patches which absolutely require an immediate reboot before further patches can be applied.  Such interim reboots are only needed when patching a live boot environment on a system below Kernel patch 118833-36 (SPARC) / 118855-36 (x86) and well as the earlier interim reboot required on x86 related to '' patches and Kernel patch 118844-14.  On systems below these patch levels, the 'installcluster' will stop at the appropriate point when patching the live boot environment, and inform the user to reboot and re-invoke the 'installcluster' script.  (In the old cluster install script, it simply tried to carry on blindly past such interim reboots, spewing out error messages, although code in the relevant patches prevented any harm from being done).  These interim reboots, when required, are dealt with relatively early in the cluster install sequence so that once completed, the Sys Admin can leave the rest of the installation to finish unattended and move onto other systems.
  • The new 'installcluster' script provides better integration with Solaris Live Upgrade as the user can now specify the Live Upgrade alternate boot environment to patch by name.
  • The new 'installcluster' script performs space checking prior to installing each patch, and will halt if it believes there is insufficient space to complete the installation successfully.  For example, this helps avoid non-global zones getting out of sync regarding patch levels with respect to the global zone.  This is an important enhancement as running out of space during patching can potentially leave the system in an inconsistent state and is to be avoided.  Even removing a patch requires space, so immediate removal of a patch which has failed to apply correctly due to space issues should be avoided until sufficient space is freed up and potential issues caused by its partial installation investigated - for example, was the undo.Z file successfully created to enable backout ? (Tip: It may be better to retry the patch installation once space has been freed up rather than patch removal in such circumstances.  Contact Sun Support for instructions if you encounter such issues.).   The space checking enhancements in the 'installcluster' script are designed to prevent such problems occurring.
  • The messages and log files produced by the 'installcluster' script are clear and well structured.  For example, a "failed" log is created if a patch fails to apply.  See the Cluster README for further information.
  • The 'patch_order' places patches in an optimal order for installation to avoid known issues - for example, the patch utilities patches are installed as early in the sequence as possible to avoid hitting patch installation bugs which are fixed in the patch utility patches, and the Kernel patch procedural script override patch, 125555 (SPARC) / 125556 (x86), is ordered prior to 137137-09 (SPARC) / 137138-09 (x86) to resolve some known issues.  When patching an alternate boot environment (which is recommended), a small sub-set of pre-requisite patches, primarily the patch utility patches, need to be applied to the live boot environment to ensure correct patching operation.  The 'installcluster' script will check for these pre-requisite patches are halt installation if they are not present, advising the user of the 'installcluster' script option to use to install these pre-requisite patches.   Further patches may need to be installed on the live boot environment to support Live Upgrade.  See the cluster README for further information.
  • The patches have been moved to a 'patches' sub-directory, to de-clutter the top level directory of the unzipped cluster.
  • Please see the cluster README file for further information.  Customers should read the cluster README file and look at the Special Install Instructions in the patches within the cluster prior to installation.

I really want to thank Ed Clark for the enormous amount of thought and effort he has put into improving the cluster installation experience.   The work he's done on the Solaris 10 Recommended and Sun Alert patch cluster is a continuation of his previous work on the Solaris Update Patch Bundles and the Solaris 10 Live Upgrade Zones Starter Patch Bundle.  Nice work, Ed!

While the 'installcluster' script is copyrighted, I am happy for customers to use it, and the 'patch_order' file, as a starting point for their own customized patch bundles, so long as it is for their own use and is not to be given to a 3rd party or used for commercial gain (e.g. by a 3rd party maintainer or 3rd party commercial automation tool).

We have also made significant improvements to the back end processes to ensure higher and more consistent cluster quality. 

Originally, the clusters were created by the Patch Operations and Distribution (POD) team after patch release.  The POD Cluster QA process left a lot to be desired, resulting in inconsistent cluster quality.   To plug this gap, my Patch System Test team have been testing the clusters for several years, but the old process only allowed us to test them in parallel with their release, which meant that we found issues at the same time that early downloaders of the cluster encountered them.  Although we ensured such issues were fixed as quickly as possible, it still obviously compromised our customers' experience.

In the new process, the clusters are routed to Patch System Test (PST) prior to release.  PST run a transformation script on them to optimize the patch installation order, etc.  The clusters will only be released once they have passed PST testing.  This should ensure higher and more consistent quality for customers.  Work is continuing to move the entire patch cluster generation process to PST, although these future backend enhancements in this regard should be invisible to customers.

Thursday Jun 18, 2009

Solaris 10 5/09 (Update 7) patch bundle now available!

The Solaris 10 5/09 (Update 7) patch bundle is now available for download from the SunSolve Patch Cluster & Patch Bundle Download Page.  Click on the "Solaris Update Patch Bundles" link.

As with previous patch bundles, it contains the patches which are included in the corresponding Solaris Update, in this case Solaris 10 5/09 (Update 7).

This is useful for Sys Admins who wish to bring all their systems up to the same patch level as the Solaris Update without wanting to upgrade to the release - for example, due to change control policy restrictions in their organizations.

See previous blog entries for previous Solaris Update patch bundles for further information.

Wednesday Jun 17, 2009

Zones Parallel Patching feature now available!

The Zones Parallel Patching feature is now available in the latest Solaris 10 patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

This is available for use on all Solaris 10 systems. 

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

Prior to this feature, each non-global zone was patched sequentially, leading to unnecessarily long patching times for zones systems.  (Sequential patching remains the default behavior unless the config file is edited to enable Zones Parallel Patching.)

With this feature invoked, the global zone continues to be patched first, but then the non-global zones can be patched in parallel, leading to significant performance gains in patching operations on Zones systems.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

Here's the relevant note from the patch README file:

NOTE 10: 119255-66 is the first revision of the patch utilities to deliver "zones parallel patching".
          This new functionality allows multiple non-global zones to be patched in parallel by patchadd.
          Prior to revision 66, patchadd would patch all applicable non-global zones sequentially,
          that is one after another. With zones parallel patching, a sysadmin can now set the number
          of zones to patch in parallel in a new configuration file for patchadd called /etc/patch/pdo.conf.

         The two factors that affect the number of non-global zones that can be patched in parallel are
         1. Number of on-line CPUs
         2. The value of num_proc in /etc/patch/pdo.conf

          If the value of num_proc is less than or equal to 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in
          parallel to num_proc. If the value of num_proc is greater than 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in parallel
          to 1.5 times the number of on line CPUs. Note that patchadd will patch all applicable non-global
          zones on a system, the above description outlines only how patchaadd determines the
          maximum number of job slots to be used during parallel patching of non-global zones.

          An example of this in operation would be where:
          and number of on line CPU's is 4

          In this case the maximum setting for num_proc would be 6, that is the maximum number
          of zones that could be patched in parallel is 6.  If there are more than this number of non-global zones on the
          system, the first 6 will be patched in parallel, then the remaining non-global zones will be patched
          as processes finish patching the first 6 non-global zones.   Only one patch process will be used for each
          non-global zone, so if there are less than 6 non-global zones on the system, then only the number of processes
          equal to the number of non-global zones will be initiated.

          Please see comments in /etc/patch/pdo.conf for more details on setting num_proc.

I would like to thank Ed Clark and Enda O'Connor from my own team for all their work in developing and testing Zones Parallel Patching.

I would also like to thank Jon Bowman, Arindam Sarkar, and the rest of the RPE (Sustaining) Install team for all their work in getting this feature integrated into the patch utilities and delivered to production.

I would also like to thank our selected key customers who kindly Beta tested the feature for us.

I believe this feature is an important milestone in improving our customers' patching experience in a Zones environment as it addresses a long standing customer complaint on Zones patching performance.


Wednesday Feb 11, 2009

Now possible to upgrade directly from Solaris 8 SPARC to latest Solaris 10 release

Thanks to my colleague Enda O'Connor, who has made p7zip available for Solaris 8 SPARC, it's now possible to upgrade directly from Solaris 8 SPARC to the latest Solaris 10 Update releases such as Solaris 10 5/08 and Solaris 10 10/08. 

See document 1019995.1 on My Oracle Support (MOS),

Previously, due to the lack of p7zip on Solaris 8, customers needed to perform an interim upgrade to Solaris 9 or an earlier Solaris 10 release before upgrading to the latest Solaris 10 release.

Monday Jan 05, 2009

Stricter Solaris patch entitlement implementation roll-out commencing this week

I've updated this blog entry to avoid causing unnecessary confusion with the current patch entitlement policy now that Oracle has acquired Sun.  

The Solaris patch entitlement policy is available on

BTW: It's important to remember that hardware warranties only provide access to Firmware and hardware driver patches.   Hardware warranties do not cover software support or access to other Solaris patches.

Wednesday Dec 17, 2008

Definitive interpretation of the "rebootimmediate" and "reconfigimmediate" patch flags

The following is now available as Infodoc 249046:

What follows is an open letter to customers in response to customer confusion over how to handle the "rebootimmediate" and "reconfigimmediate" flags specified in some patches.

Despite the READMEs of patch clusters which contain such patches clearly stating that during a patching session, a reboot is only required in exceptional and documented circumstances, it has come to my attention that some customers are initiating reboots after applying every single patch in a patch set which specifies such flags.  Not surprisingly, such customers are concerned at the length of time this takes.

Open Letter with definitive interpretation of the "rebootimmediate" and "reconfigimmediate" patch flags

To whom it may concern,

Summary: When patching a live boot environment, it is usually OK to apply any number of patches before performing a single reboot at the end, even if multiple patches specify "rebootimmediate" or "reconfigimmediate".  On the rare occasion when it is found that this is not possible, specifically for 118833-36 (SPARC) and 118855-36 (x86) and 118844-14+ (x86), code will typically be inserted into the relevant patches to prevent the application of further patches which could cause problems.  Use of Live Upgrade to patch an inactive boot environment is recommended as it avoids the need for interim reboots for even these atypical patches.  Details below.

The "reboot" metadata flags which may be contained in the patch 'pkginfo' file(s) have the following meaning:

rebootafter - a reboot is required to activate some of the content delivered in the patch, but the system remains in a consistent state until the reboot is performed.

reconfigafter - a reconfiguration reboot is required to activate some of the content in the patch, but the system remains in a consistent state until the reconfiguration reboot is performed.

rebootimmediate - the system is in a potentially inconsistent state until the system is rebooted.  The objects applied in the patch are potentially inconsistent with processes running in memory.  Normal production must not be resumed until a reboot takes place to bring the system back into a fully consistent state.  However, since the footprint of the patch utilities is relatively small, it is normally OK to continue to apply further patches before initiating the reboot.   In cases where this is not OK, the patch in question will typically contain additional code to prevent further patches from being applied until the reboot takes place\*.  Since the system is in a potentially inconsistent state, it's advisable to avoid running any additional processes until the reboot takes place.  If patch automation tools are being used to apply "rebootimmediate" or "reconfigimmediate" patches, it's up to the automation tools' QA to ensure that their additional code footprint does not hit the potential inconsistent system state when applying such patches.

reconfigimmediate - exactly the same as rebootimmediate, except a reconfiguration reboot is required.

\*This is the case with Kernel patch 118833-36 (SPARC) / 118855-36 (x86), whose patch scripts replace 'patchadd' with a no-op telling the user to reboot the system.  The only other known reboot required before further patching can be done is specific to x86, and only if the system is running at a Kernel patch level below 118844-14.  A later revision of 118844, e.g. 118844-20, needs to be applied and the system rebooted to ensure the Kernel running in memory is compatible with library changes supplied in the libc patch 121208-02.  The prepatch script in 121208-02 and -03, and 118855-xx which obsoletes it, contains code to ensure 118844-14 or later is installed and active on the system.  (BTW, 118844-14 wasn't released. 118844-20 is recommended to fulfill the libc compatibility requirement.)

UPDATE, Jan 20, 2009: Murphy's Law strikes again!.  There's currently an issue, CR 6704883, with the "Sun Fibre Channel Device Drivers" patches 125184-05, -06, -07, and -08 (SPARC) and 125185-05, -06, -07, and -08 (x86) as described in Sun Alert 238630.  The fix for this issue is in rev-09 of the patches which is currently available as a T-Patch and will be released shortly.  Rev-09 of the patches uses modloading in its prepatch script to avoid the issue.  In the meantime, a workaround is to apply the affected patches last, immediately prior to rebooting the system.  The patches in the Solaris 10 10/08 patch bundle were specifically ordered to avoid this issue.  Where such issues are found, SunAlerts are published and the issue fixed.

Remember, patches can be downloaded and installed individually.  Therefore, each patch which requires a reboot must specify the reboot requirements.  But if patches are installed collectively in the same patching session, for example, as part of a patch cluster, then the install instructions contained in the cluster README file take precedence - e.g. that reboots are only required \*during\* patching sessions for the specific cases mentioned above.

Since the above patches were created, a significant enhancement has been made to the Solaris patch utilities called Deferred Activation Patching.  This enhancement is not retrospective, so the above historical problematic patches remain.

Deferred Activation Patching

The problem with the above atypical patches is that the new code they deliver may be invoked by the original patchadd code and the utilities it calls \*during\* patch installation.  A patch may patch many packages.  The packages are applied in alphabetic order.  In a Zones environment, the patch is applied to the global zone first, then to each non-global zone.

In the case of 118833-36 (SPARC) / 118855-36 (x86), the new versions of the and libraries delivered in the patch could be invoked by patchadd and are potentially incompatible with the processes running in memory.

The solution devised in the patch scripts contained in 118833-36 (SPARC) / 118855-36 (x86) is to overlay mount the old objects on top of the newly laid down objects using the loopback filesystem (lofs).  This ensures that the system remains in a consistent state \*during\* the patch process as the old library versions which are compatible with what's running in memory will be called.

To avoid the application of further patches, which patch the same objects as 118833-36 (SPARC) / 118855-36 (x86), from patching the overlay mounted objects instead of the patched objects, 118833-36 (SPARC) / 118855-36 (x86) replace 'patchadd' with a no-op telling the customer to reboot the system before applying any further patches.

During reboot, the loopback filesystem mounts are torn down exposing the patched objects.  Further patching can now continue as the system is in a fully consistent state.

This loopback filesystem mount solution is the basis of Deferred Activation Patching.  After patch 118833-36 (SPARC) / 118855-36 (x86) was released, the solution was perfected and moved to the patch utilities.  The few patches which require application using Deferred Activation Patching specify the SUNW_PATCH_SAFE_MODE=true flag in their pkginfo files.  The solution was enhanced so that any subsequent patch applied prior to a reboot of the system, which patches the same objects as a patch explicitly specifying Deferred Activation Patching, will itself be automatically applied in Deferred Activation Patching mode.   This is known as implicit Deferred Activation Patching and enables other patches to be applied on top of a patch applied using Deferred Activation Patching without the need for an intervening reboot.  When a patch specifying Deferred Activation Patching mode is applied to a system, the user will see lots of loopback filesystem mounts on the system until such time as the reboot takes place.  Upon reboot, the loopback filesystem mounts are torn down, exposing the newly patched objects.

Kernel patch 12001[12]-14 which is included in Solaris 10 8/07 (Update 4), Kernel patch 12712[78]-11 which is included in Solaris 10 5/08 (Update 5), and Kernel patch 13713[78]-09 which is included in Solaris 10 10/08 (Update 6), are currently the only patches which specify application in Deferred Activation Patching mode.  Future Kernel patch included in future Solaris 10 Update releases are the likely candidates requiring application using Deferred Activation Patching.

With the introduction of Deferred Activation Patching, it is highly unlikely that future patches will require an interim reboot before further patches can be applied.

The problems with the system getting into an inconsistent state \*during\* patching (which Deferred Activation Patching resolves) could only occur when patching a live boot environment as it's due to the interaction between newly patched objects which are incompatible with processes running in memory being invoked prior to the system being rebooted.

To avoid this and other issues, Sun strongly recommends the use of Live Upgrade to patch (or upgrade) an inactive boot environment, which dramatically reduces the risk and downtime associated with patching.  For example, even though Deferred Activation Patching resolves the inconsistency issue, patching a live boot environment takes time and the system is out of production.

Using Live Upgrade, the inactive boot environment is patched, potentially while the system is still in production.  Issues such as those described above with Kernel patch 118833-36 (SPARC) / 118855-36 (x86), and 118844-20 (x86) simply don't apply when patching an inactive boot environment as there is no interaction between the objects being patched and the processes running in memory, as all the calls patchadd makes will be to the objects on the live partition, not the patched objects on the inactive partition.  A single reboot is required to boot into the new boot environment.

Another advantage of Live Upgrade is that if a problem arises with the new boot environment for whatever reason, the user can simply reboot back into the old boot environment to enable production to resume and the issues with the now inactive boot environment can be resolved later.

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Thursday Dec 04, 2008

Patching enhancements and other stuff

New title, same role, same me

I was promoted to Director, Software Patch Services in September.  The last couple of months have been quite hectic, as I've suddenly got a whole new bunch of buddies in Marketing and elsewhere who want some of my time.  That's a good thing, and I believe it will help me to drive and co-ordinate improvements for you, our customers, patching experience. 

Resources are limited and, as always, I'm interested in getting your thoughts as to what areas I should concentrate on next.  

Some of the stuff we're currently working on is outlined below as well as other information which I hope you will find useful.

Solaris 10 10/08 Patch Bundle

The Solaris 10 10/08 Patch Bundle, which delivers the equivalent set of patches to the Solaris 10 10/08 (Update 6) release image, is now available from SunSolve.  See my blog entry below on the Solaris 10 5/08 (Update 5) Patch Bundle for further information on why we produce it, what it contains, why you might wish to use it, how to download it, etc.

Recommended and Sun Alert patch cluster contents updated

I discussed the purpose of, and difference between, the Solaris Recommended and Sun Alert patch clusters in a previous blog posting. To recap:

The "Recommended" Cluster contains the latest revision of any Solaris OS patch which addresses a Sun Alert issue.  That is, a fix for a Security, Data Corruption, or System Availability issue.  The cluster also contains the latest revision of the patch utility patches to ensure correct patch application and any patch required by any other patch in the cluster.

The Sun Alert Cluster is newer, and contains the minimum revision of any Solaris OS patch which addresses a Sun Alert issue. The cluster also contains the latest revision of the patch utility patches to ensure correct patch application and any patch required by any other patch in the cluster.  Therefore, the Sun Alert Cluster provides the minimum amount of change to fix all Solaris OS Sun Alert issues. 

Both clusters are updated whenever a new patch meeting their inclusion criteria is released.  The Sun Alert Cluster changes less frequently than the "Recommended" Cluster as it contains only what is really needed to address Sun Alert issues and apply the patches.

One of my team members has been reconciling the cluster contents against the Sun Alert reports and the cluster contents have been updated as a result.  Some issues where found, largely to do with patches for things like GNOME which are also part of the Solaris OS.  A process has been put in place to ensure the cluster contents match the patches specified in the Sun Alert reports.   

Keeping as up to date as possible with the SunAlert or Recommended Cluster contents is advisable.   Remember also to keep firmware up to date.

BTW: The monthly EIS (Enterprise Installation Standards) patch baseline is based upon the Recommended Cluster contents but also includes ca. 150 additional patches to address irritants which are not Sun Alert fixes and includes patches for SunCluster, SunVTS, etc.  The monthly EIS patch baselines are available through xVM Ops Center and Sun Proactive Services.

I am planning to merge the Recommended and Sun Alert patch clusters into a single cluster using the Sun Alert cluster criteria as having two very similar clusters tends to confuse customers unnecessarily.  

I also intend to merge the two cluster pages on SunSolve as one is essentially a better formated subset of the other. 

ZFS and Zones features fully contained in patches

As I've mentioned previously, there's effectively a single customer visible code branch for each Solaris named release.  That means that there's one set of patches for all of Solaris 10, a separate set for Solaris 9, and a separate set for Solaris 8.  Within a named release, e.g. Solaris 10, the same set of patches will apply to any of the Solaris 10 releases, from the original Solaris 10 3/05 release right up to the current Solaris 10 10/08 (Update 6) release.  This simplifies System Administration and enables Sun to provide very long term support at reasonable cost for each Solaris named release. 

A consequence of effectively having a single code branch for each Solaris named release is that any change to pre-existing packages will be delivered in patch format.

New features are typically only added to the current Solaris named release, which is currently Solaris 10.  (They are also available via OpenSolaris.)

This means that if new features don't add any new packages, then the entire feature functionality is fully available in patches.  Customers can utilize the new features by simply applying the appropriate patches to their existing Solaris 10 system.  This is the case with all current Zones and ZFS\* functionality, including neat features like ZFS Root, ZFS Boot, and Zones "Update on Attach".

Other features which deliver new packages are only available from the Solaris Update release in which they were first included.  So, for example, if a new package was first delivered in Solaris 10 8/07 (Update 4), then a customer wishing to use that feature would need to install or upgrade to the Solaris 10 8/07 (Update 4) or subsequent update release image.   Such features are not available in patches.

\*OK, we cheated with ZFS.  ZFS does deliver new packages, but they are streamed into existence from a patch.  This type of patch is called a "genesis" patch, but they are hard to perfect, so we don't intend to release any more "genesis" patches.

Improving Zones Patching Performance

Zones Parallel Patching

My team has been working with those awfully nice folks in the Sustaining organization to deliver a Zones Parallel Patching enhancement to the patch utilities to dramatically improve Zones patching performance.  We have a fully stable prototype which has been given to selected Beta customers to trial. 

For a simple T2000 with 5 sparse non-global zones, the performance improvement is >3x.  On systems with optimized I/O (as Zones patching is primarily I/O bound), we expect the performance improvement to be even better.  A configuration file will allow users to select how many Zones to patch in parallel.  This will typically equate to the number of processors or threads available on the target system.

The general release of this feature is planned for April 2009.

Zones "Update on Attach" 

The Kernel patch associated with Solaris 10 10/08 (Update 6), 137137-09 (SPARC) / 137138-09 (x86) contains some cool new features, such as ZFS Root, ZFS Boot, and Zones "Update on Attach".  Beware, installing this patch requires significant free disk space to install!  See Sun Alert

Zones "Update on Attach" is a very cool feature indeed.

For example, if the patch level of non-global Zones is out-of-sync with respect to the global Zone, e.g. because the non-global Zones ran out of disk space during patch application, Zones "Update on Attach" provides a very neat way to bring the Zones back into sync.  Simply detach the affected non-global Zones, apply Kernel patch 137137-09 (SPARC) / 137138-09 (x86) to the global zones, and reattach the affected non-global Zones using 'zoneadm -z <zone-name> attach -u'.  The non-global Zones will be automagically updated to the same patch level as the global Zone.  Neat!

There are other interesting possibilities.  For example, detach all non-global Zones, apply an arbitrary set of patches to the global Zone (including 13713[78]-09), and reattach the non-global Zones using 'zoneadm -z <zone-name> attach -u'.  Viola!, the non-global Zones will be automagically updated with all of the patches applied to the global Zone.  Way neat!  And more importantly, way faster than even the Zones Parallel Patching solution we're working on.  And even better, it's available now!  This could be a key solution for customers having difficulty completing patching updates on Zones systems during tight maintenance windows.

We are working to explore potential caveats.  For example, when a patch is applied using 'patchadd' to a non-global zone, an "Undo.Z" file containing the data necessary to back out the patch is created specifically for each non-global zone to which the patch is applied.   Using Zones "Update on Attach" to patch non-global Zones will cause the "Undo.Z" file from the global Zone to be propagated to the non-global Zones.  This could theoretically cause issues if the patch is subsequently backed out (e.g. data from global Zone config files could potentially be merged into non-global Zone config files during patch backout which could potentially cause issues), although we've never actually encountered such an issue.  BTW: The same caveat applies to creating non-global Zones after the global Zone has been patched.  Again, we have yet to see this causing an actual issue, so it appears to be more of a theoretically caveat than a practical issue.

Improvements to 'smpatch' and Update Manager

The way the PatchPro analysis engine for 'smpatch' and Update Manager used to work was fine in theory, but in practice was what I call "a process with too many moving parts".   Too many steps had to happen correctly for the overall result to be correct.  In Six Sigma terms, there was too much error opportunity.  Occasionally, it would end up recommending a SPARC patch for an x86 system or a Solaris 8 patch for a Solaris 10 system.  Not surprisingly, its reputation suffered.

I'm pleased to say that a major overhaul to dramatically simplify the back end processing of 'smpatch' and Update Manager has just been rolled out by their engineering team.  The way 'smpatch' and Update Manager work is that Realization Detector(s) are associated with each patch.  These Realization Detectors determine whether it's appropriate to recommend a patch for application on a target system.  In the vast majority of cases, the Realization Detectors are simply comparing the packages contained in the patch to the packages installed on the system to see if the patch is applicable.  The enhancement is to replace these myriad Realization Detectors, which could potentially contain coding bugs, with a single Generic Realization Detector to map patch packages to packages on the target system.  It looks at the package name, package version, and package architecture fields (in pkginfo) for each package in the patch, and compares them to the same values for the packages installed on the target system.  If they match, the patch is recommended, else not.  Guess what, this is exactly how 'patchadd' decides whether a patch is applicable or not when installing a patch.  It's also how 'pca' works too in determining which patches to apply.

A few specialist Realization Detectors remain for a small number of patches which require special handling.

The changes to 'smpatch' and Update Manager should dramatically improve the reliability of these tools and the accuracy of their patching recommendations.

One remaining distinction between 'smpatch' / Update Manager and 'pca' is that 'pca' "knows" about all current Sun patches via the patchdiag.xref file, whereas 'smpatch' / Update Manager "knows" about all patches containing a 'patchinfo' file, including older patch revisions.  All Solaris OS and Java Enterprise System (middleware) patches contain a 'patchinfo' file.  These account for 49% of patches.  For patching the Solaris OS, the tools should produce similar results.  A decision was made not to "auto-include" all other patches for 'smpatch' and Update Manager, as it was felt that the explicit step of the patch creator including a non-blank PATCH_CORRECTS realization detector specification line in the 'patchinfo' file to signal that the patch was suitable for patch automation was potentially useful.  (Don't worry about what value the PATCH_CORRECTS field has.  This is overriden by the Generic Realization Detector in the vast majority of cases.  It has no meaning from a customer perspective.)

This enhancement is not an attempt to undermine 'pca'.  It's simply to improve 'smpatch' and Update Manager.  I will continue to work closely with Martin Paul to give him heads-ups on any initiative which may impact 'pca' and resolve any issues with patchdiag.xref.

One thing I want to do when I can free up some resources, is a comparative study of the patching recommendations of the various available patch automation tools, 'smpatch' / Update Manager, 'pca', UCE (a.k.a Sun Connection Satellite),  xVM Ops Center\*, and TLP (Traffic Light Patching) which is used by Sun Proactive Services to provide tailored patching solutions for customers in conjunction with SRAS (Sun Risk Analysis Service) and the EIS (Enterprise Installation Standards) methodology, with a view to ensuring that the patching recommendations of the various tools are coherent and consistent, with the higher value tools providing more sophisticated analysis.  It's part of my efforts to co-ordinate patching improvements to improve our customers' patching experience.

\*xVM OC also utilitizes the monthly EIS patch "baselines".

Same Patch Entitlement policy, new Patch Entitlement implementation

Solaris changed its business model a few years ago from selling Solaris and providing patches for free to a model of giving away the software releases for free and charging for patches. 

The policy is that patches delivering new security fixes will remain free to all customers, irrespective of whether or not they have a support contract, but most other patches require that customers have a valid support contract to access them.  (See my earlier blog entry on the subject.)

All fixes will all be available for free in the next Solaris Update release (and OpenSolaris), so customers not willing to pay for a support contract can still get the fixes by installing or upgrading to the next Solaris Update release.  They'll just need to wait for it to ship.  Alternatively, they can use OpenSolaris.

This policy is not changing.

What is changing is the implementation of patch entitlement to ensure it matches the policy.  Currently, circa 60% of Solaris patches are free, including most of the key patches.  Under the new entitlement implementation, 18% of Solaris patches will remain free, including the specific revision of all Solaris patches which include new security fixes.  The rest will require a valid support contract to access. 

Any of the following support contracts will provide access to all Solaris patches and patch clusters: a Solaris subscription, a Software Support Contract, a Sun System Service Plan for Solaris, a Sun Spectrum Storage Plan, or a Sun Spectrum Enterprise Service Plan.  Since the names of the support contracts change from time-to-time, this list may change.

The new implementation will roll out in Phases, starting this month.  The roll-out should be transparent to customers with valid support contracts.

Patch signing certificate renewal

The signing certificate used to sign Sun patches expires shortly.  A new signing certificate will be rolled out in January and instructions provided on how to adopt it.

Customers who download the unsigned patch versions will not need to take any action.

"Accumulation-only" patches

The "SplitGate" source code management model we first introduced in Solaris 10 8/07 (Update 4) has dramatically improved Solaris 10 patch quality.  A side-effect of the "SplitGate" model is that base PatchIDs (the first 6 digits) change at the end of each Update release.  See my earlier Solaris 10 Kernel PatchID Sequence posting.

In the "SplitGate" model, when building an Update release, we effectively have two parallel source code gates, one called the Sustaining Gate containing just the bug fixes we need to release to customers in patches asynchronous to the Update release, and the other called the Update Gate containing a superset of the the Sustaining Gate and as well as new features and less critical bug fixes which will be released as part of the Update release. 

The two gates remain separate (split) for the duration of the Update release build process.  Once the Update release has reached release quality, the Update Gate is promoted to become the new Sustaining Gate and the process repeats.  Since the Update Gate is always a strict superset of the Sustaining Gate, no regressions should result from the promotion of the Update Gate to become the new Sustaining Gate.  Each patch in the old Sustaining Gate is obsoleted by a corresponding patch from the Update Gate which has accumulated its contents.  When the Update is released, these new PatchIDs are released to SunSolve.  This is why you see the base PatchIDs changing after each Update release. 

If the Update Gate patch doesn't contain any additional code changes over the corresponding Sustaining Gate patch, then there's no need for customers to install the new Update Gate patch.  Such patches are called "accumulation-only" patches and can be identified as they have a different base PatchID (the first 6 digits) but don't contain any additional CR numbers over the Sustaining patch which they obsolete.

The reason Sun releases these "accumulation-only" patches is because some customers insist that all of the PatchIDs pre-applied into a Solaris Update release image be also available from SunSolve.

Thursday Mar 06, 2008

Sun Alert Notifications

You can sign up to receive a weekly notification advising of new and updated SunAlerts .

Sun Alerts inform customers of the most critical issues affecting Sun's hardware and software.

They cover Security, Data Corruption, and System Availability issues.

Customers with a valid support contract will be able to access all Sun Alerts and patches which fix Sun Alert issues, including the Sun Alert patch clusters available on SunSolve which contain all Solaris OS patches which address Sun Alert issues.

Customer without a valid support contract will be able to access Sun Alerts and Patches only for Security related issues when they log onto SunSolve.


This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer


« October 2015