Monday Mar 10, 2014

ORAchk Health Checks for the Oracle Stack (including Solaris)

Cross posting from my Solaris 11 Lifecycle blog, https://blogs.oracle.com/Solaris11Life/entry/orachk_health_checks_for_the  as this is applicable to Solaris 10 too:

My colleagues, Susan Miller and Erwann Chénedé, have been working with the nice people behind the ORAchk tool (formerly RACcheck) to add Solaris health checks to the tool.

ORAchk 2.2.4, containing the initial 8 Solaris health checks, is now available:

ORAchk includes EXAchks functionality and replaces the popular RACcheck tool, extending the coverage based on prioritization of top issues reported by users, to proactively scan for known problems within:

  • E-Business Suite Financials Accounts Payables
  • Oracle Database
  • Sun Systems

ORAchk features:

  • Proactively scans for the most impactful known problems across your entire system as well as various layers of your stack
  • Simplifies and streamlines how to investigate and analyze which known issues present a risk to you
  • Lightweight tool runs within your environment; no data will be sent to Oracle
  • High level reports show your system health risks with the ability to drill down into specific problems and understand their resolutions
  • Can be configured to send email notifications when it detects problems
  • Collection Manager, a companion Application Express web app, provides a single dashboard view of collections across your entire enterprise

ORAchk will expand in the future with more high impact checks in existing and additional product areas. If you have particular checks or product areas you would like to see covered, please post suggestions in the ORAchk community thread accessed from the support tab on the below document.

For more details about ORAchk see Document 1268927.1

Thursday Sep 12, 2013

Solaris 10 Patches Now On Monthly Release Cadence

(Updated Nov 25, 2013)

We've recently moved to a monthly release cadence for Solaris 10 OS patches.

New Solaris 10 OS patches are now available from MOS by the Tuesday closest to 17th of each month. 

The updated Solaris 10 OS Recommended Patchset will be available by the next day, Wednesday, assuming there are new patches released which meet its inclusion criteria - that is, patches which address security or other critical issues.

This enables customers to predict patch release dates and schedule maintenance windows.

This is similar to the monthly release cadence for Solaris Repository Updates (SRUs) for Solaris 11.

Please note that the Solaris 10 OS Recommended Patchset may not be updated every month.  This is because in some months there may be no new patches meeting the inclusion criteria.  That is, patches which address security, availability, data corruption, or other critical issues.

Monday Jun 17, 2013

Nice Compare & Contrast of Solaris 10 and Solaris 11 Zones

My colleague, Jeff Victor, has a nice blog posting comparing and contrasting Solaris 10 and Solaris 11 Zones, which I think you may find interesting.

Thursday Jun 06, 2013

Next Solaris 10 Kernel PatchIDs, 150400 (SPARC) & 150401 (x86)

As I've noted in an update to my previous blog posting, Murphy's Law strikes again!

No sooner had I written that Solaris 10 Kernel PatchIDs 148888-xx (SPARC) and 148889-xx (x86) were here to stay for the foreseeable future, than the integration of the SR-IOV feature into rev-04 of these patches made it prudent to rejuvenate them. 

So from July 2013, the Solaris 10 Kernel PatchIDs will change to be 150400-xx (SPARC) and 150401-xx (x86).

See here for the full Solaris 10 Kernel PatchID sequence.

Monday Apr 22, 2013

Ooops, incorrect Recommended patchset uploaded April 21/22

Update: April 22, 2013, 17:10 PST: The issue is now fixed and the correct Solaris 10 SPARC Recommended patchset is now available from MOS.  I apologize again for any inconvenience caused.

---- 

Ooops!

Due to human error, the incorrect Recommended patchset for Solaris 10 SPARC was uploaded to MOS on April 21, at ca. 18:54 PST.  The April 20 2012 patchset was uploaded instead of the April 20 2013 patchset. 

The date is in the patchset README, so if you've downloaded the Solaris 10 SPARC Recommended patchset in the last 24 hours, please check that the date is not 2012.  If it is, please download the corrected version from MOS.

I apologize most sincerely for any inconvenience caused.

Best Wishes,

Gerry.

Friday Mar 22, 2013

Solaris 10 1/13 patchset released and latest Solaris 10 Kernel PatchIDs

Posting updated June 6, 2013, with new Solaris 10 Kernel PatchIDs 150400-xx (SPARC) and 150401-xx (x86):

As usual, we've released a patchset of all the patches contained in Solaris 10 1/13 (Update 11):

We've also included an important post-S10U11 patch - 150125-01 (SPARC) / 149637-02 (x86) - in this patchset, which fixes ZFS Bug 15809921.  See Doc 1535270.1.

This patchset can be applied to any existing Solaris 10 system to bring all pre-existing packages up to the same software level as Solaris 10 1/13.

It is not the same as upgrading to Solaris 10 1/13 (available here), as upgrading will additionally install any new packages delivered in the Update. 

I've also updated my Solaris 10 Kernel PatchID sequence posting with the latest Solaris 10 Kernel PatchIDs, namely: 

  • The Solaris 10 1/13 Kernel patch, 147147-26 (SPARC) / 147148-26 (x86)
  • Post Solaris 10 1/13 Kernel patches have the PatchIDs 148888-xx (SPARC) / 148889-xx (x86)

Please note that there are no more planned updates to Solaris 10, so these latest Kernel PatchIDs - 148888-xx (SPARC) / 148889-xx (x86) - will continue to be used for the foreseeable future.

Murphy's Law strikes again!

No sooner had I written that Solaris 10 Kernel PatchIDs 148888-xx (SPARC) and 148889-xx (x86) were here to stay for the foreseeable future, than the integration of the SR-IOV feature into rev-04 of these patches made it prudent to rejuvenate them. 

So from July 2013, the Solaris 10 Kernel PatchIDs will change to be 150400-xx (SPARC) and 150401-xx (x86).

Dare I tempt fate again by saying these Solaris 10 PatchIDs are likely to remain the same for the foreseeable future ?

I've also updated my Useful Patch Related Downloads posting with links to the Solaris 10 1/13, Jan 2013 CPU, and latest Recommended patchsets.

Friday Oct 19, 2012

October 2012 Security "Critical Patch Update" (CPU) information and downloads released

The October 2012 security "Critical Patch Update" information and downloads are now available from My Oracle Support (MOS).

See http://www.oracle.com/technetwork/topics/security/alerts-086861.html and in particular Document 1475188.1 on My Oracle Support (MOS), http://support.oracle.com, which includes security CVE mappings for Oracle Sun products.

For Solaris 11, Doc 1475188.1 points to the relevant SRUs containing the fixes for each issue.  SRU12.4 was released on the CPU date and contains the current cumulative security fixes for the Solaris 11 OS.

For Solaris 10, we take a copy of the Recommended Solaris OS patchset containing the relevant security fixes and rename it as the October CPU patchset on MOS.  See link provided from Doc 1475188.1

Doc 1475188.1 also contains references for Firmware, etc., and links to other useful security documentation, including information on Userland/FOSS vulnerabilities and fixes in https://blogs.oracle.com/sunsecurity/

Wednesday Oct 12, 2011

Live Upgrade document updated and simplified

I forgot to let you know, but a couple of months ago, my colleagues, Don O'Malley and Ed Clark updated the Oracle Solaris Live Upgrade (LU) document describing the pre-requisites for Live Upgrade.

The original document was pretty convoluted and required several cups of strong coffee to parse.  The updated version is a little easier to understand, even without caffeine.

Thanks also to Beth Barrett, Rick Ramsey, and Jon Bowman who helped make this happen.

Wednesday Oct 05, 2011

Solaris 10 8/11 (Update 10) Patchset now available

Hi Folks,

The Solaris 10 8/11 (Update 10) patchset is now available from My Oracle Support.  Here's direct links to the common README and the SPARC and x86 downloads.  You need to be logged into MOS and have a valid support contract associated with your account in order to download the patchsets.

BTW: Please see my previous blog posting for details on other useful direct links to Solaris patch downloads and metadata.

As you may know by now, these patchsets will bring all pre-existing packages up to the same software level as the corresponding Solaris Update.  For example, all ZFS and Zones functionality is entirely contained in pre-existing packages, so applying the patchset will provide all the ZFS and Zones functionality and bug fixes contained in the corresponding Solaris Update.  

When we release the Solaris Update patchset, we try to fix any serious late breaking issues found with the corresponding Solaris Update patchset.  A list of additional patches added and the Caveats they address is contained in the patchset README.

Applying the patchset is not the same as upgrading to the Solaris Update release, as the patchset will not include any new packages introduced in the Solaris Update or any obsolete packages deleted in the Update.   

Please see this blog posting for lists of the new packages introduced in each Solaris Update to see if any of them are relevant to you.  If they are, then upgrade to a release which provides them.  If they're not, then applying the patchset may be a reasonable alternative to update your Solaris system. 

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  They are listed at the end of the SPARC and x86 patch lists.

Health Warning: Do not manually apply packages from a later Solaris release to an earlier Solaris release (e.g. by pulling individual packages from an ISO image) as this will result in an inconsistent system state which may lead to system corruption unless careful post-processing is done at the time such packages are applied to ensure that any patches applied to either the pre-existing packages on the system or pre-applied to the new packages been added are reapplied to the system to ensure both the pre-existing and new packages are at the same patch level.  Failure to do this will compromise the patch utilities ability to resolve patch dependencies leading to undefined results.  Even if you take the above steps, Support are likely to frown upon such shenanigans.  So don't do it.  If you need new packages, upgrade to a release which provides them.  Note, Live Upgrade packages are the only exception to this rule and the procedure for them is specified in the Live Upgrade documentation.  

Best Wishes,

Gerry.

Monday Sep 05, 2011

Applying arbitrary patches using Ops Center

I asked my colleague, Juergen Fleischer, to let me know how to apply arbitrary patches - i.e. any specific patches you want to apply - using Ops Center.

Here's his response on how to do it:

Create a Custom Profile by selecting the desired patches. The Profile can get used for Compliance Reports or standard jobs to install these patches.

Here are the steps:

1) Select Plan Management -> Update Profiles

2) Select Action "New Profile" and name it appropiate e.g. "Solaris Cluster 3.1 APR-2011"

3) Select all required patches e.g. 120500-27 for SPARC and/or 120501-27 for x86 and save the Profile:

4) Use the Profile for plain Jobs, Reports, etc.

That's all

Best Wishes,

Gerry.

Saturday Jul 02, 2011

A Solaris Recommended Patchset to bind them all

I've long been of the opinion that there should be a single generic set of Solaris recommended patches which customers are consistently recommended to install in proactive maintenance windows for issue prevention. It's something I've been working towards for quite a while.

A collaborative effort between the Software Patch Services, Enterprise Installation Standards (EIS), Sun Risk Analysis System (SRAS) - now renamed Oracle Risk Analysis Services (ORAS) - and the Explominer team in the Oracle Solaris Technical Center (TSC), has achieved this goal with the creation of the Recommended Patchset for Solaris.  

Up until now, while the Solaris OS Recommended Patch Cluster was the core basis for Solaris patch recommendations, various teams tended to recommend their own favorite patches on top of this core set.  This wasn't just by whim.  Each team was looking at patching from a slightly different angle - for example various angles of proactive patching (issue prevention) versus reactive patching (issue correction).

The Recommended Patchset for Solaris is the result of the combined wisdom of the various teams.  It is designed for proactive patching (issue prevention).  The contents are generic and should be suitable for most customer configurations.  You should still read the README file and follow its instructions to ensure all of the patches included are appropriate to your specific environment.  You should test the patchset on a test system which closely mimics your production systems prior to deployment. 

You may still legitimately be asked by support to install additional patches to fix issues specific to your environment in reactive maintenance situations (issue correction).  But this should only be after due diligence to ensure that such patches are likely to fix the specific issue encountered.

The Recommended Patchset for Solaris is the new name for the Solaris OS Recommended Patch Cluster.  It's available from MOS (including 'wget'), EIS, Ops Center, etc.  We've changed the name to use the Oracle standard terminology "patchset".  I never liked the name Solaris Patch Cluster as there was a risk of it being confused with the Solaris Cluster product to which it bears no relation.  In due course other patch "clusters" and patch "bundles" are likely to transition to the name "patchset". 

The install script and code word needed to invoke it (which is contained in the README file) have been renamed to reflect the name change from "cluster" to "patchset". 

Customers who have installed the Solaris OS Recommended Patch Cluster may notice the additional patches included in the Recommended Patchset for Solaris the first time they install it.  After that, it'll be business as usual.  Many of these additional patches are already pre-applied into Solaris Update releases, so customers on later update releases should see little difference.

As before, the Recommended Patchset for Solaris will continue to be updated whenever a patch matching its inclusion criteria is released.  This can happen several times a month.  Just take the latest which matches your proactive maintenance window schedule. 

And as before, once a quarter, the Recommended Patchset for Solaris will be archived and renamed as the Critical Patch Update in line with standard Oracle practice.  (See previous blog postings.)

To create the Recommended Patchset for Solaris, we took the Solaris OS Recommended Patch Cluster and analyzed the additional Solaris patches which the Explominer team recommend be added on top of it for the monthly EIS patch baselines. Where those additional patches added real value - i.e. were of significant benefit to many customers - we added them to the recommended patch set.  Where they didn't add real value, we discarded them.  We then made sure that a system on which the resultant Recommended Patchset for Solaris was installed passed with a clean bill of health from the ORAS risk analysis audits.

So now, the Solaris OS patches in the EIS patch baselines will be the Recommended Patchset for Solaris with input from the Explominer and other teams included, and will be tested with ORAS.  These are the patch baselines available in Ops Center.  We have set up a panel of patch experts from the teams mentioned above to adjudicate on future potential additions to the Recommended Patchset for Solaris.

Previously, the criteria for including a patch in the Solaris OS Recommended Patch Cluster was quite strict: a patch had to address a Security, Data Corruption, or System Availability issue; be a patch utilities patch, or be required by the above.  In future, other patches which add real value for many customers may be included - for example, a patch for a commonly used driver which delivers significant performance improvements.  The goal remains the same - to include the most critical generic patches which we recommend customers install in proactive maintenance windows for issue prevention.

Additional patches outside of the patchset may still be required:

  • For other Oracle products - the Recommended Patchset for Solaris only includes Solaris Operating System patches.  Other products such as Oracle Solaris Cluster, Oracle Solaris Studio, Oracle Database, etc., may have their own patch recommendations.  The monthly EIS update includes patch sets for Oracle Solaris Cluster, SAMFS, QFS, and SunVTS in addition to the Recommended Patchset for Solaris.
  • For specific platforms - for example a Solaris driver patch if a particular network card is installed or where firmware updates are required
  • For specific configurations - for example if the system is connected to 3rd party storage solutions such as EMC Powerpath or Veritas
  • For specific issues in your configuration - for example, break/fix situations where an additional patch fixes the issue encountered

You can download the patchsets or view their Readmes directly, using the following links:

To downloads the patchsets (you must be logged into MOS):

https://updates.oracle.com/patch_cluster/10_Recommended.zip
https://updates.oracle.com/patch_cluster/10_x86_Recommended.zip

To download the patchset Readme files (no need to be logged into MOS):

https://updates.oracle.com/patch_cluster/10_Recommended.README
https://updates.oracle.com/patch_cluster/10_x86_Recommended.README

The above works for both flash and non-flash (html) MOS users.   Just substitute "9" for "10" to get the Solaris 9 Recommended patchsets and Readmes.

You can also download the patchsets using 'wget' for scripted access as normal.  (See previous blog postings.)  For example, the download filename for Recommended Patchset for Solaris 10 SPARC is still 10_Recommended.zip.

If, like me, you like to know how to do things from first principles, here's the way to construct the search on My Oracle Support:

For Flash compatible systems (full function MOS version):

  1. Login to My Oracle Support (MOS), https://support.oracle.com
  2. Click on the "Patches&Updates" tab
  3. Click on "Product or Family (Advanced Search)
  4. Type "Solaris Operating System" into the product search box
  5. Select the Releases you are interested in - e.g. Solaris 10 Operating System and Solaris 9 Operating System
  6. Select the Platforms you are interested in - e.g. Oracle Solaris on SPARC (64-bit) and Oracle Solaris on x86-64 (64-bit)
  7. Click on the "+" sign next at the end of the "Platforms" line to add additional search criteria
  8. Click of "Select Filter" and select "Type" from the drop-down menu
  9. Select "Patchset"
  10. Click "Search" 

For non-Flash users (html MOS version):

  1. Login to the html version of My Oracle Support, https://supporthtml.oracle.com
  2. Click on the "Patches & Updates" tab
  3. Click on the Advanced Search tab in the search box
  4. Type "Solaris Operating System" in the product search box 
  5. Select the Releases you are interested in - e.g. Solaris 10 Operating System and Solaris 9 Operating System
  6. Select the Platforms you are interested in - e.g. Oracle Solaris on SPARC (64-bit) and Oracle Solaris on x86-64 (64-bit)
  7. For Type, select "Patchset"
  8. Click Search

MOS remembers your previous selections and they'll be shown top of each drop down menu on subsequent invocations.  You can also save searches for future re-use.

I want to thank Don O'Malley, Ed Clark, Howard Mills and the EIS team, Juergen Schleich and the Explominer team, Dr. Rex Martin and the ORAS team, and Rob Hulme and Walter Fisch from the Oracle Technical Support Center (TSC) for all their work in making a single consistent Recommended Patchset for Solaris a reality.

As always, I'm interested to hear your feedback.

Best Wishes,

Gerry.

Thursday Jan 13, 2011

List of new and up'rev'd packages in each Solaris 10 Update

Here are lists in .pdf (SPARC / x86) and OpenOffice (SPARC / x86) format of new and up'rev'd packages in each Solaris 10 Update release.

As you may know from my previous blog postings, Oracle Sun recommends customers to install or upgrade to the latest Solaris 10 Update in major maintenance windows. Based on a request from customers whose change control policies prevent them from upgrading, we've been producing Solaris Update Patch Bundles which bring pre-existing packages up to the same software level as the corresponding Solaris Update.  The difference is that the Patch Bundles don't provide new or up'rev'd packages introduced in the corresponding Solaris Update.

For customers considering use of the Solaris Update Patch Bundles, that raises the obvious question as to which packages are introduced or up'rev'd in each Solaris Update release.  The lists above answer that question.

Aside: As discussed in previous blog postings, all core Solaris OS packages are updated via patches.  The up'rev'd packages above refer to some 3rd party and community based apps included in Solaris (e.g. Mozilla Firefox, Thunderbird, etc.) which are updated via package updates (i.e. where one package version is removed and replaced with a later version).  This is to tie in better with the release strategy for such apps.

Many thanks to my colleague, Roisin Doran, for all her work in putting this together.

I'll ask Roisin to work with the Technical Writers to include updated versions of these lists in future Solaris 10 Update release documentation.

Thursday Jan 06, 2011

Solaris 10 Recommended Patching Strategy

Here's a document and a corresponding presentation I've written describing the Oracle Solaris 10 Recommended Patching Strategy. They contain a number of links to resources which I hope you will find useful.

As always, I look forward to your feedback.

BTW: If you have any queries about patching, why not post them on the Oracle Solaris Install, Booting, and Patching Community Forum.

Best Wishes,

Gerry.

Friday Oct 15, 2010

Solaris Updates and Patch Bundles in the picture

A colleague of mine, Ken Brucker, has drawn this picture showing the composition of Solaris Updates which you may find useful to visualize the process.  

See the relevant slides from the presentation on http://blogs.sun.com/patch/entry/updated_customer_patching_presentation_and for more detailed information.

Saturday Sep 18, 2010

Solaris 10 9/10 (Update 9) Patch Bundle now available

The Solaris 10 9/10 (Update 9) Patch Bundles are now available from SunSolve and My Oracle Support (MOS).

These patch bundles provides the set of patch pre-applied into the corresponding Solaris 10 9/10 (Update 9) release image.  These patches provide all the Solaris 10 bug fixes which were available when the contents of the Solaris 10 9/10 release was finalized.

See http://blogs.sun.com/patch/entry/solaris_10_10_08_patch for further information on Solaris Update Patch Bundles.

See http://blogs.sun.com/patch/entry/oracle_sun_patches_now_available for information on how to access patch bundles on MOS.

Many thanks to the Patch System Test, Patch Operations and Distribution, and SunSolve teams for expediting the release of these patch bundles.

Thursday Sep 09, 2010

Solaris 10 9/10 (Update 9) released

Solaris 10 9/10 (Update 9) has been released.  See here for information and here for the download (remember to accept the license agreement at the top).  There's also a podcast and a dedicated Solaris blog.

A number of technical articles have been released, including George Wilson's video overview of ZFS enhancements in Solaris 10 9/10.

As with all Solaris Updates, Solaris 10 9/10 contains all available bug fixes which were available at the time that its contents were finalized, pre-applied into the Solaris Update image. 

It also contains a significant number of feature enhancements as described in the above links.

The corresponding Solaris Update Patch Bundle is currently in test and I expect that it should be released in a similar timeframe to previous Updates.  See http://blogs.sun.com/patch/entry/solaris_10_10_08_patch  for information on Solaris Update Patch Bundles.

All standard patches in Update 9 have already been released to SunSolve and My Oracle Support (MOS).  I've updated the Solaris 10 Kernel PatchID Sequence entry below with the Kernel PatchIDs for Solaris 10 9/10 (Update 9).

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  See the SPARC and x86 patch lists. 

<pet peeve>

Please note it is incorrect to refer to Kernel Patch 142909-17 (SPARC) / 142910-17 (x86) as the "Update 9 Kernel patch".  It is the latest Kernel Patch included in Update 9, but this Kernel patch can equally be applied to all previous Solaris 10 releases.   Solaris Updates are built from patches (and a few new packages), patches are not built from Solaris Updates.

</pet peeve>

Thursday May 20, 2010

Merging the Solaris Recommended and Sun Alert Patch Clusters

The Solaris "Recommended" and Sun Alert Patch Clusters have been merged (June 4th 2010). 

The merged clusters are called the "Recommended OS Cluster Solaris <release> <architecture>", for example "Recommended OS Cluster Solaris 10 SPARC". 

The old "Recommended" and Sun Alert Patch Clusters only ever contained Solaris OS patches (with rare exceptions), so we've added "OS" to the new merged cluster name to make this a little clearer.

The merged Recommended OS Clusters have the same access entitlement as the old clusters - namely, you need a support contract which covers Solaris to access them.

The old "Recommended" patch cluster contains the latest revision of Solaris OS patches which fix Sun Alert issues (i.e. Security, Data Corruption, or System Availability issues).  That is, the top-of-tree patches which fix Sun Alert issues.

The Sun Alert patch cluster contains the minimum revision of Solaris OS patches which fix Sun Alert issues.  Thus, the Sun Alert patch cluster provides the minimum amount of change required to get all available Solaris OS fixes for Security, Data Corruption, and System Availability issues.

The contents for the two clusters are very similar, which causes unnecessary confusion as to which one to use.  When the Sun Alert Cluster was released several years ago, it should have replaced the older "Recommended" Cluster, and this merging of the Clusters is to correct that omission.

The inclusion criteria for the Sun Alert cluster is more logically correct, as in the Recommended Cluster there's no more value in adding the latest revision of a patch whose earlier revision provided a fix to a Sun Alert issue than in adding any other random patch.  Many folks assume "latest is greatest", and Oracle Sun wouldn't release a patch unless it is important, but this is slightly simplistic.  Change implies risk, and as many patches address issues which are only seen in very specific configurations, and while Oracle Sun patches are thoroughly tested prior to release, there is little advantage in taking more change than is necessary in minor maintenance windows or reactive patching situations.  Therefore, providing a minimal patch cluster which provides all available fixes for Solaris OS Sun Alert issues for use in minor maintenance windows makes sense.

The old "Recommended" Clusters were often updated several time a week, simply because a later revision of a patch whose earlier revision fixed a Sun Alert issue was released, even though the later revision didn't fix any additional Sun Alert issues.  Since the "Recommended" flag on SunSolve and in the patchdiag.xref metadata file matches the contents of the old "Recommended" Cluster, we were releasing many more patches which were flagged as "Recommended" than customers really needed to apply.

After the merge, new patches added to the Recommended OS Cluster and hence the "Recommended" flag on SunSolve and in the patchdiag.xref metadata file will be the specific revision of patches which address Sun Alert issues.  Only when an obsoleting patch provides a new fix to a Sun Alert issue will it be included and the obsolete patch removed.  The merged Recommended OS Clusters will update on the same cadence as the old Sun Alert clusters, which is typically about once a week for Solaris 10 (5.5 times a month, on average).  We will continue to update the merged Recommended OS Cluster whenever a patch matching the inclusion criteria is released.

To avoid the potential confusion which may be caused if we were to remove the "Recommended" flag from any patches, we will take the "Recommended" Cluster at the beginning of June 2010 as the basis for the merged cluster and then apply the Sun Alert Cluster inclusion criteria going forward.

The merged Recommended OS Cluster was initially released on June 4th, 2010.  The download link (target) file name of the merged cluster will be the same as the old "Recommended" Cluster, e.g. 10_Recommended.zip, to minimize the changes users need to make to automated download scripts.

Customers who have traditionally downloaded the Sun Alert cluster will need to update download scripts to use the merged cluster file download names as the old Sun Alert cluster are no longer available.

In major maintenance windows, the Best Practice recommendation is to upgrade to the latest available Solaris Update release or at least to apply the equivalent Solaris Update Patch Bundle available from the patch cluster download page.  In both cases, the latest Recommended OS Cluster should also be applied as it will contain any additional Solaris OS Security, Data Corruption, and System Availability fixes released since the Solaris Update contents were finalized.  Solaris Updates are intensely tested, and hence this strategy provides a well tested, stable, and feature rich baseline for production systems.  In between major maintenance windows, the Best Practice recommendation is to try to keep as up to date as possible with the contents of the merged Recommended OS Cluster during minor maintenance windows.

Let's look at an example, to make the rationale for the change clearer: 

In the old model, if a security vulnerability in /usr/bin/ls is fixed in patch 123456-03, then both the old Recommended and Sun Alert clusters will initially include it.  If code interdependencies caused by subsequent code putbacks - e.g. the major Trusted Solaris Extensions feature - result in the contents of the "/usr/bin/ls" patch 123456-07 being accumulated into a feature Kernel patch associated with a Solaris 10 Update, e.g. 234567-14, then the old "Recommended" Cluster would include 234567-14 instead of 123456-03, even if 234567-14 contained no additional fixes for Sun Alert issues (i.e. Security, Data Corruption, or System Availability issues) compared to 123456-03.  The "Recommended" flag on SunSolve, in patchdiag.xref, and elsewhere would be updated every time a patch revision obsoletes the original patch, even though these later patch revisions contain no additional fixes to Sun Alert issues.  This can lead to customers who try to stay up to date with "Recommended" patches patching more content and potentially more often than is really necessary.  In contrast, 123456-03 would remain in the Sun Alert cluster for as long as no additional fixes for Sun Alert issues are contained in obsoleting patches.

In the new merged Recommended OS patch cluster model, while the starting point will be the old "Recommended" Cluster as of the start of June 2010 (to avoid dropping the "Recommended" from any patches, which might cause confusion), further changes to the cluster will follow the old Sun Alert cluster inclusion criteria - that is, the merged Recommended OS patch cluster contents and corresponding Recommended flag in SunSolve and patchdiag.xref will only be updated if a new patch delivers a new fix for a Sun Alert issue.   This means that only patches which we really recommend will be included in the Recommended OS patch cluster and flagged as Recommended in SunSolve and patchdiag.xref.  Since the rate of change will be less, it'll be easier for customers to see what's really recommended and allow more informed decisions regarding when to apply such patches.

Please note that this change has nothing whatsoever to do with the integration into Oracle.  This is an enhancement I've been looking to do for some time to avoid the confusion caused by having two very similar patch clusters and a corresponding "Recommended" flag which was updated much more frequently than was necessary.

My team has been working with known consumers of the "Recommended" patch flag such as TLP, Ops Center, 'smpatch', Update Manager, SRAS, EIS, and 'pca' to ensure that the transition goes smoothly.  

For example, TLP and 'pca' consume the patchdiag.xref file which up to now typically only contained entries for top-of-tree (latest) patch revisions.  From June 4th 2010, patchdiag.xref will contain whatever revision of a patch is flagged as "Recommended" as well as the top-of-tree patch revision.  Hence, a single base PatchID, e.g. 123456, may have two entries in the file, e.g. 123456-03 marked "R" for Recommended and "O" for Obsolete and 123456-08 which is the latest revision of that patch but which won't carry the "R" flag as it contains no additional Sun Alert fixes over rev-03.  

From my discussion with Martin Paul, author of 'pca', my understanding is that initially, he plans to propagate the "R" flag forward to the latest patch revision in his 'pca' metadata as currently 'pca' only handles the latest revision of patches, but he'll look at some stage in the future to leverage the more precise "Recommended" flag data we'll be providing with this change.

Friday Mar 26, 2010

Interesting article on Sun's collaboration with Intel to improve performance

This isn't specifically to do with patching, but here's an interesting article from Intel detailing some of the work Intel and Sun have been doing to maximize Solaris performance on "Nehalem" Xeon 5500 series systems, enabling Solaris to leverage the advanced features of the chip set.

Please note that this does not in any way imply anything negative about SPARC.  Solaris is committed to supporting both x86/x64 and SPARC architectures to give you choice.

Tuesday Nov 24, 2009

Do not apply packages from one Update onto a system installed with a different Update

I'd just like to make the following clear to customers:

Customers who install packages from one Update release (e.g. S10U6) on a system installed with another release (e.g. S10U3) risk corrupting their system.

You must not install packages from one Update release onto a system installed with any other Update release.

The reason for this is that all available patches are pre-applied into all packages for each Update release.   Patches and packages have a many-to-many relationship.  That is, one patch can patch many packages.   One package can be patched by many patches.

If you install a package from an Update release onto a system installed with a different Update release, you completely compromise future patch dependency checking as you've introduced patch metadata from a later Update release.   This is likely to lead to system corruption as further patches are applied.

'patchadd' checks that dependencies are satisfied when installing a patch.  If 'patchadd' finds any installed package patched with a patch which satisfies the dependency, it assumes the patch is applied to all packages.  This is done for performance reasons.  Hence, if a package from a later release is installed on the system, it's pre-applied patches may fool subsequent 'patchadd' invocations into thinking that a hard code dependency has been satisfied for all packages on the system when this is not the case.   The patch application will be allowed to continue, potentially corrupting the system.

The converse is also true.   If a package from an earlier Update is applied to a system, the patch delta from that Update to the Update installed on the system plus any additional patches installed on the system would need to be applied to the package to avoid a mismatch in software levels between the packages on this system, which could lead to incorrect patch dependency resolution and hence to system corruption.  Since this is difficult to get right, adding packages from a different Update release onto an installed system is, in general, unsupported. 

There are two exceptions to the above rules, for Live Upgrade and encryption packages.

  • Live Upgrade: If you are upgrading from Solaris 8 or Solaris 9 to Solaris 10, you need to apply the Live Upgrade packages from the Solaris 10 system to your Solaris 8 or Solaris 9 system.  See document 1004881.1 on My Oracle Support (MOS), http://support.oracle.com, for details.
  • SUNWcry\* encryption packages: These weren't included in the Solaris distribution prior to Solaris 10 8/07 (Update 4).  For pre-Update4 systems, the recommended way to get these packages is to upgrade to a later Update release.  For the reasons outlined above, while not recommended, a possible alternative is to download the packages from the Sun Download Center (SDLC), install them, and then re-apply any patches that patch the SUNWcry\* packages which you have already applied to the system (\*\*). Also, for the reasons outlined above, it is not recommended to apply the packages from the media of a later Update release, as you would need to ensure that all the other packages installed on the system are patched to the same or higher patch level  (e.g. by installing the appropriate Solaris Update Patch Bundle available from My Oracle Support, http://support.oracle.com) to avoid completely compromising future patch dependency checking on the system.

\*\* A way to check which patches need to be re-applied in this scenario is as follows:

cd /var/sadm/patch
egrep -i "PKG=SUNWcryr|PKG=SUNWcry" \*/log|cut -f1 -d /|sort -u

as in:

# cd /var/sadm/patch
# egrep -i "PKG=SUNWcryr|PKG=SUNWcry" \*/log|cut -f1 -d /|sort -u
127127-11
137137-09
139555-08
141444-09
#

The above example is on a system installed with Soalris 10 11/06 (Update 3).  The user would need to re-apply the patches listed above and be extremely careful to ensure they are applied in the correct dependency order as 'patchadd' will not be able to ensure the correct dependency order as it's dependency checking remains compromised until the added packages are brought up to the same patch level.

Please note that if a package from the same Update is applied to a system, then any additional patches already installed on the system that patch the added package must be re-applied to bring that package up to the same software level as the rest of the system.  This is called "incremental patching".   This is supported, but care must be taken.  The easiest way to do this is to reapply all patches installed on the system (as listed by 'patchadd -p').  This will bring the added package(s) up to the same software level as the rest of the system.  Again, you need to be extremely careful to ensure they are applied in the correct dependency order as 'patchadd' will not be able to ensure the correct dependency order as it's dependency checking remains compromised until the added packages are brought up to the same patch level as the rest of the system.

Best Wishes,

Gerry Haskins,
Director, Software Patch Services

Tuesday Nov 17, 2009

New FREE online Patch Training Courses

I'm delighted to announce the availability of 10 new free online patch training modules.

This is the result of a lot of work from those nice people in Sun Learning Services, the Install Revenue Product Engineering (RPE, a.k.a. Sustaining) team, and my own folk.

The modules concentrate on using Live Upgrade for patching, as well as providing background on Deferred Activation Patching, Kernel patches, and other useful information.

You can access the modules as follows:

I think even experienced Sys Admins will find the modules useful in clarifying patching best practices and providing context and background information on the evolution of patching technology and best practices in Solaris 10.

If you don't like the online course format, or if you want a reference document to refer back to after taking the course, please see the attached .pdf.

Enjoy!

Best Wishes,

Gerry Haskins,
Director, Software Patch Services

Wednesday Nov 04, 2009

Solaris 10 10/09 Patch Bundle now available

I'm delighted to announce that the Solaris 10 10/09 (Update 8) Patch Bundle is now available for download by customers with a Solaris support contract.

Each Solaris Update Patch Bundle contains the equivalent set of patches which are pre-applied into the corresponding Solaris Update release image.

It is provided to enable customers who cannot upgrade for whatever reason to be able to patch systems up to the same patch level as the Update release.

Each Solaris Update is intensely tested as a unit by myriad QA teams across Sun.  Therefore, Solaris Updates and their corresponding Solaris Update Patch Bundles provide good quality "baselines" on which customers can standardize their deployments.

Standardizing deployments on such "baselines" also provides customers with a "safety in numbers" effect, as any pervasive issues are likely to be found and fixed quickly, so each customer benefits from the experience of others.

The Solaris Update Patch Bundle brings all existing packages up to the same software level as the Update release.   Any features which are entirely contained in pre-existing packages, such as Zones and ZFS functionality, are entirely available in patches and hence applying the Solaris Update Patch Bundle brings them up to the same functional level as the Update release.

However, installing the Patch Bundle is not completely equivalent to upgrading to the corresponding Solaris Update as the Patch Bundles do not include any new packages introduced in the Solaris Update release image.  Therefore, any new features which are dependent upon new packages will not be available by applying the Solaris Update Patch Bundle.

Here's a summary of the new packages in Solaris 10 10/09 (Update 8) which are not available in the Solaris 10 10/09 Patch Bundle:

SUNWhxge: SUN 10Gb hxge Ethernet Network Adapter Driver
SUNWio-tools: Administrative tools to modify the pci/pcie fabric
SUNWmrsas: LSI MegaRAID SAS2.0 HBA driver
SUNWpixman: Pixman Library
SUNWntp4r: NTPv4 (root)
SUNWntp4u: NTPv4 (usr)
SUNWntp4S: NTPv4 (source)
SUNWmptsas: LSI MPT SAS2.0 HBA driver

Please remember to apply the latest Sun Alert Cluster on top of the Solaris Update Patch Bundle in order to get all Solaris OS security, data corruption, and system availability fixes released since the final build of the Update release.

Please see previous blog entries for further details on Solaris Update Patch Bundles.

Top Tip: If you are installing in a zones environment, make sure you have the latest patch utility patches installed and Zones Parallel Patching configured before you apply a Solaris Update Patch Bundle as Zones Parallel Patching will improve non-global zone patching performance by ~300%.   See this blog entry for details.

BTW: There is no need to take any action to enable "Turbo-Charging SVR4 Package Installation" as the necessary patches are installed early on when installing the Solaris 10 10/09 Patch Bundle and will be automatically enabled for subsequent patch application when the bundle is applied to the live boot environment.  While "Turbo-Charging" has little affect when installing most patches, it does significantly speed up the application of a small number of older patches with non-optimized deletes file processing install scripts and so does speed up the Solaris 10 10/09 Patch Bundle installation somewhat.

Best Wishes,

Gerry.

Thursday Oct 22, 2009

Major PatchFinder enhancements available now!

I'm delighted to announce the release of the 2nd phase of our PatchFinder tool enhancements, which include:

  • The ability to see the "Entitlement Classes" of patches and get information on the support contracts necessary to access and use them.  
  • A "Patch Basket", into which you can add selected patches from multiple search results.
  • When you click on the "Go To Patch Basket" link, the patch dependencies for all the patches you have in your Patch Basket will be dynamically resolved, including filtering out redundant dependencies.   This saves you having to manually transfer patch dependency trees!   If you already have some of these installed, you can de-select them.
  • You can then click the "Download Selected" button to download a 'wget' script and instructions which you can use to download all of the selected patches from SunSolve.   Once you make sure you install the latest version of the patch utilities patch first, you can then use "patchadd -M" to install all the patches in the correct order on your target system.

Sample Searches

Let's assume you applied the Solaris 10 SPARC Recommended Patch Cluster on August 15th 2009.  So what Solaris 10 SPARC Recommended Cluster patches have been released since then ?   To find out, for "OS Release" select "Solaris 10", for "Architecture" select SPARC", select "Recommended Only", and select August 15th 2009 from the calendar beside the "Released After" box.   (Select view 50, 100, or 200 to see the entire list in one page.)   You can then decide if you want to download some of all of these patches to add to your system.  Coupled with the dynamic dependency resolution and 'wget' download capability, this effectively enables you to create customized patch clusters for yourself with just the patches you need, rather than having to download the entire Recommended Cluster each time.

Or you could bookmark a search to show you all the patches released in the last day: Simply enter the number "1" into the "Released After" box and select any other selection criteria you are interested in and click "Search".  Depending on timezone differences with respect to California and your local time of day, you may need to enter the number "2" in the "Released After" box.

You can also use PatchFinder to see what Solaris 8 Vintage patches Sun has released since Solaris 8 entered End-Of-Service-Life (EOSL) Phase 2 on April 1, 2009.   Simply select "Solaris 8" for "OS Release", select "OS Patches Only" and click "Search".  Since the patches are listed in date order, most of the patches with a release date after April 1, 2009, including patches delivering security fixes, will have the "Solaris8VintageSoftwareUpdate" Entitlement Class associated with them if you mouse-over the red padlock symbol shown for them (assuming you don't have a Solaris 8 Vintage Patch Service Plan associated with your Sun Online Account).   You will see a couple of non-Vintage patches released after April 1, 2009.  This is a transition phase and these patches address issues escalated by customers prior to April 1, 2009.

Some other sample searches to satisfy your curiosity:

Ever wondered how many patches Sun has ever released ?   To find out, simply select "Show Obsolete" and then click "Search".

How many current "active" patches does Sun have ?   De-select "Show Obsolete" and then click "Search".

How many patches can be installed on Solaris 10, including application product patches ?   For "OS Release" select "Solaris 10" (and optionally "Show Obsolete" ) and then click "Search".

How many current "active" Solaris 10 OS patches there are for SPARC ?  For "OS Release" select "Solaris 10", for "Architecture" select "SPARC" and then select "OS Patches Only" and then click "Search".

Patch Access Entitlement Classes

You need a support contract or have hardware under warranty in order to access and use patches.

When you look at the list of patches returned from a search, a green open padlock symbol shows the patches you have access to thanks to the support contracts which you currently have associated with your Sun Online Account (SOA).  A red closed padlock shows the patches which you are not currently entitled to access or use with the support contracts you currently have associated with your Sun Online Aaccout.

You can mouse-over these symbols for any patch and it will show you the "Entitlement Classes" associated with the patch. 

Read the "What is it?" help link and the SunSolve "How Entitlement Works" wiki to find out about the support contracts which you need to buy in order to access and use these patches.

Feedback

I hope you'll find the new PatchFinder enhancements useful.

We are really interested in your feedback as to what further enhancements you would like to see, so feel free to post your comments here or else use the feedback link on the PatchFinder page.

Many thanks to Brian Kidney and Julien Colomb for all their work on this - nice work guys!

Wednesday Oct 07, 2009

Latest Solaris 10 Kernel PatchIDs

I've updated the Solaris 10 Kernel PatchID sequence table in http://blogs.sun.com/patch/entry/solaris_10_kernel_patchid_progression with the latest Kernel PatchIDs for Solaris 10 10/09 (Update 8) and Solaris 10 Update 9.

Friday Sep 18, 2009

Patching Pre-flight Checks (ppc) tool now available

The new Patching Pre-flight Checks ('ppc') tool is now available to all customers who have a support contract.

The idea for this tool comes directly from customer feedback.  

The customer wanted to reduce the cost of patching Solaris systems by enabling more junior Sys Admins to successfully patch Solaris 10 zones systems.   Their concern was that potential zones patching issues in versions of Solaris prior to Solaris 10 8/07 (Update 4) meant that they needed to assign senior System Administrators to patch such systems to identify and resolve potential issues.  

Furthermore, the customer was concerned that such issues had the potential to derail planned maintenance windows - for example, if during the patching session an unexpected issue was encountered and the patching session couldn't be completed as planned.

To address these concerns, my colleague, Ronan O'Connor, has written the Patching Pre-flight Checks tool, 'ppc'.  It can be run prior to a planned patching session to check that the target system is in a clean state ready for patching.

It's important to understand the scope of the tool.  It checks a target system (and a patch set, if supplied) for a variety of inconsistencies which could cause problems.

It looks for left over lock files from previously aborted patching or packaging operations, inconsistencies in the contents database, IDRs installed on the target system, zones "mountability", space issues, etc.  Some of these issues can occur on early versions of Solaris 10, particularly in a Zones environment.  Many of the underlying causes of such issues are fixed in the latest versions of the patch utility patches (119254 SPARC / 119255 x86), which is why we always recommend you apply the latest patch utility patches before applying other patches.

If you have a directory of patches to be applied, 'ppc' checks the integrity of those patches, and cross-checks whether any of the patches patch pkgs which have been locked down by any IDRs on the system and warns if there is a conflict.

The 'ppc' Release Notes provide information to help interpret the messages produced.

The idea is that 'ppc' can be run by a junior Sys Admin prior to a planned patching session, and any potential issues uncovered can then be analyzed by a more experienced Sys Admin.  This helps avoid nasty surprises during patches sessions and also helps to reduce the level of expertise required to patch Solaris systems, leading to cost savings for customers.

It is outside the scope of the 'ppc' tool to do root cause analysis of why the inconsistency arose or what actions may be needed, if any, to correct the situation.

If 'ppc' returns without noting any problems, you can be pretty confident that the patching session will succeed.  If 'ppc' notes potential issues, they can be investigated prior to the planned maintenance window.

The next version of 'ppc' will include a Zones consistency check to check that all zones are at a consistent patch level.   It will also contain a more sophisticated space checking algorithm.  There's no planned release date yet for Version "2.0" yet as we're awaiting feedback on Version 1.0.x first.

Some of the ideas in 'ppc' may find their way back into 'patchadd', although it's probably appropriate to keep 'ppc' as a separate tool.

You can download the Patching Pre-flight Checks tool, 'ppc', here.  It's an attachment to the knowledge article.  The Patching Pre-flight Checks tool, 'ppc', and the Customer Patch Forum are only available to customers with a support contract, so you'll need to login to SunSolve to access the knowledge article.

We're very interested in your feedback as to the usefulness of this tool and how you'd like to see 'ppc' develop going forward.

Many thanks to Ronan O'Connor for all his work on the tool!

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Wednesday Sep 09, 2009

Patches for "Turbo-Charging SVR4 Package Install" are now available

I am delighted to announce that Casper Dik's "Turbo-Charging SVR4 Package Install" feature enhancement is now available by downloading and installing the following patches:

119254-70 (SPARC) / 119255-70 (x86): Patch utilites patch
121428-13 (SPARC) / 121429-13 (x86): Live Upgrade Zones Support Patch
121430-40 (SPARC) / 121431-41 (x86): Live Upgrade Patch
124630-28 (SPARC) / 124631-29 (x86): System Administration Applications, Network, and Core Libraries Patch

It is important to apply 119254-70 (SPARC) / 119255-70 (x86) and 121428-13 (SPARC) / 121428-13 (x86) if the system is running non-global zones.  Otherwise booting newly installed zones will fail until the pkgserv daemon exits, about 5 minutes after zoneadm install finished.  Zones which were already installed can be booted as expected.

"Turbo Packaging" is designed to dramatically improve the performance of install, upgrade, Live Upgrade, and zone creation on Solaris 10.  It delivers only a small improvement for patching performance.   (See my Zones Parallel Patching blog entry for information on dramatic patching performance improvements.)

For background reading on the "Turbo Packaging" feature, please see
http://www.opensolaris.org/jive/thread.jspa?messageID=358081 and http://arc.opensolaris.org/caselog/PSARC/2009/173/

The "Turbo-charging SVR4 Package Install" feature will also be included in the forthcoming Solaris 10 Update 8 release and will be documented in the Release Notes for that release.

Great work, Casper - well done!

Friday Aug 14, 2009

Improvements to Solaris 10 Recommended and Sun Alert Patch Clusters released

My colleague, Ed Clark, has made significant improvements to the Solaris 10 Recommended and Sun Alert patch clusters.  These improvements have just been released and are in the current clusters available to contract customers from the Patch Cluster & Patch Bundle Downloads on SunSolve.

Ed's improvements include:

  • Filtering out "false negatives" from the patch utility return codes, so that if the cluster install script returns "1", you know you've got a real problem which needs investigating.   As you may know, the Solaris patch utility, 'patchadd', can return errors for some acceptable situations - for example, if the patch is already applied to the system, or a later revision of the patch or a patch which obsoletes it is already applied to the system, or none of the packages in the patch are on the target system (e.g. because a reduced Install Metacluster was used to install it or the system has been security hardened by package removal), etc.   Such conditions are acceptable "errors" which do not usually require further investigation by the user.  By filtering these conditions out, if the 'installcluster' script returns "1", you know it isn't because of one of these acceptable "errors", and therefore you need to look at the logfiles to find out what's gone wrong.  For further information, please see the cluster README and Analyzing a patchadd or patchrm Failure in the Solaris OS.
  • The new 'installcluster' script will exit as soon as it encounters an unexpected failure - i.e. not one of the acceptable "errors" mentioned above.  This prevents potentially compounding issues by attempting to apply further patches.
  • The new 'installcluster' script includes context intelligence for patching operations.   It informs the user when zones need to be halted, and it provides phased installation to handle patches which absolutely require an immediate reboot before further patches can be applied.  Such interim reboots are only needed when patching a live boot environment on a system below Kernel patch 118833-36 (SPARC) / 118855-36 (x86) and well as the earlier interim reboot required on x86 related to 'libc.so' patches and Kernel patch 118844-14.  On systems below these patch levels, the 'installcluster' will stop at the appropriate point when patching the live boot environment, and inform the user to reboot and re-invoke the 'installcluster' script.  (In the old cluster install script, it simply tried to carry on blindly past such interim reboots, spewing out error messages, although code in the relevant patches prevented any harm from being done).  These interim reboots, when required, are dealt with relatively early in the cluster install sequence so that once completed, the Sys Admin can leave the rest of the installation to finish unattended and move onto other systems.
  • The new 'installcluster' script provides better integration with Solaris Live Upgrade as the user can now specify the Live Upgrade alternate boot environment to patch by name.
  • The new 'installcluster' script performs space checking prior to installing each patch, and will halt if it believes there is insufficient space to complete the installation successfully.  For example, this helps avoid non-global zones getting out of sync regarding patch levels with respect to the global zone.  This is an important enhancement as running out of space during patching can potentially leave the system in an inconsistent state and is to be avoided.  Even removing a patch requires space, so immediate removal of a patch which has failed to apply correctly due to space issues should be avoided until sufficient space is freed up and potential issues caused by its partial installation investigated - for example, was the undo.Z file successfully created to enable backout ? (Tip: It may be better to retry the patch installation once space has been freed up rather than patch removal in such circumstances.  Contact Sun Support for instructions if you encounter such issues.).   The space checking enhancements in the 'installcluster' script are designed to prevent such problems occurring.
  • The messages and log files produced by the 'installcluster' script are clear and well structured.  For example, a "failed" log is created if a patch fails to apply.  See the Cluster README for further information.
  • The 'patch_order' places patches in an optimal order for installation to avoid known issues - for example, the patch utilities patches are installed as early in the sequence as possible to avoid hitting patch installation bugs which are fixed in the patch utility patches, and the Kernel patch procedural script override patch, 125555 (SPARC) / 125556 (x86), is ordered prior to 137137-09 (SPARC) / 137138-09 (x86) to resolve some known issues.  When patching an alternate boot environment (which is recommended), a small sub-set of pre-requisite patches, primarily the patch utility patches, need to be applied to the live boot environment to ensure correct patching operation.  The 'installcluster' script will check for these pre-requisite patches are halt installation if they are not present, advising the user of the 'installcluster' script option to use to install these pre-requisite patches.   Further patches may need to be installed on the live boot environment to support Live Upgrade.  See the cluster README for further information.
  • The patches have been moved to a 'patches' sub-directory, to de-clutter the top level directory of the unzipped cluster.
  • Please see the cluster README file for further information.  Customers should read the cluster README file and look at the Special Install Instructions in the patches within the cluster prior to installation.

I really want to thank Ed Clark for the enormous amount of thought and effort he has put into improving the cluster installation experience.   The work he's done on the Solaris 10 Recommended and Sun Alert patch cluster is a continuation of his previous work on the Solaris Update Patch Bundles and the Solaris 10 Live Upgrade Zones Starter Patch Bundle.  Nice work, Ed!

While the 'installcluster' script is copyrighted, I am happy for customers to use it, and the 'patch_order' file, as a starting point for their own customized patch bundles, so long as it is for their own use and is not to be given to a 3rd party or used for commercial gain (e.g. by a 3rd party maintainer or 3rd party commercial automation tool).

We have also made significant improvements to the back end processes to ensure higher and more consistent cluster quality. 

Originally, the clusters were created by the Patch Operations and Distribution (POD) team after patch release.  The POD Cluster QA process left a lot to be desired, resulting in inconsistent cluster quality.   To plug this gap, my Patch System Test team have been testing the clusters for several years, but the old process only allowed us to test them in parallel with their release, which meant that we found issues at the same time that early downloaders of the cluster encountered them.  Although we ensured such issues were fixed as quickly as possible, it still obviously compromised our customers' experience.

In the new process, the clusters are routed to Patch System Test (PST) prior to release.  PST run a transformation script on them to optimize the patch installation order, etc.  The clusters will only be released once they have passed PST testing.  This should ensure higher and more consistent quality for customers.  Work is continuing to move the entire patch cluster generation process to PST, although these future backend enhancements in this regard should be invisible to customers.

Thursday Jun 25, 2009

Training for Solaris 10 Patching

Sun Learning Services are in the process of creating a number of patch related training lessons.

They've launched a blog, which contains the initial introductory videos.

Future lessons will be much more detailed, concentrating for example on Live Upgrade.   These lessons will be available on the Sun Open Learning Center (SOLC) website: https://learning.sun.com/solc/smartstart.

Heads up on Kernel patch installation issues with jumpstart or ZFS Root

I'd like to give you a heads-up on a couple of Kernel patch installation issues:

1. There was a bug (since fixed) in the Deferred Activation Patching functionality in a ZFS Root environment on x86 only.  See Sun Alert 263928.  An error message to the effect that a Class Action Script has failed to complete and failure to set up environment for Deferred Activation Patching may be seen.   The relevant CR is 6850329: "KU 139556-08 fails to apply on x86 systems that have ZFS root filesystems and corrupts the OS".    SPARC systems are similarly affected.  The following error message is returned:
mv: cannot rename /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to /lib/libc.so.1: Device busy
ERROR: Move of /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to dstActual failed
usage: puttext [-r rmarg] [-l lmarg] string
pkgadd: ERROR: class action script did not complete successfully

Installation of <SUNWcslr> failed.

This issue is fixed in patch in the Patch Utilities patch 119255-70 or later revision.

BTW: The principal reason ZFS Root support was implemented in Live Upgrade is so that patch application like this to the live boot environment would not be necessary.   With ZFS Root, creating a clone Boot Environment is so easy that there's no good reason not to.   This avoids the need to use technologies such as Deferred Activation Patching which attempt to make it safer to apply arbitrary change to a live boot environment, which is an inherently risky process.

2. There are reproducible issues using jumpstart finish scripts and other scenarios to install Kernel patch 137137-09 followed by Kernel patch 139555-08.   Here's the gist of the issue which I've pulled from an engineering email thread on the subject:

Issue 1: I have a customer whose system is not booting after applying the patch cluster with Live Upgrade (LU).

Solution 1: If using 'luupgrade -t', then you must ensure that latest version of LU patch is installed first, currently 121430-36 is currently the latest revision on SPARC, 121431-37 on x86. Once these patches are installed, LU will automatically handle the build of the boot archive when 'luactivate' is called, thus avoiding the problem.

Issue 2: There are other ways to get oneself into situations where a boot archive is out of sync: e.g. If using jumpstart finish scripts to apply patches that include 137137-09.  Basically any operation that involves patching to an ABE outside of 'luupgrade' will involve a manual build of boot-archive.

Solution 2: One must manually rebuild the boot-archive on the /a partition after applying the patches.  Otherwise once the system boots, the boot-archive will be out of sync.

Here's some more detail on the jumpstart finish script version of this: 

We've seen the same panic a few times when the latest patch cluster is applied via a finish script to a boot environment prior to  s10u6 via a jumpstart installation. It appears that the boot archive is out of sync with the kernel on the system. The boot archive was created from the 137137-09 patch and not updated after the 139555-08 kernel was applied, therefore the mismatch between the kernel and the boot archive.

In these instances updating the boot archive allows the system to boot successfully. Boot failsafe (ok boot -F failsafe) will detect an out of sync boot archive.  Execute the automated update then reboot.  This will now boot from the later kernel (139555-08) which successfully installed from the finish script.

I reproduced the problem in a jumpstart installation environment applying the latest 10_Recommended patch cluster from a finish script. The initial installation was S10U5 which is deployed from a miniroot that has no knowledge of a boot archive (my theory anyway).  This is similar to a live upgrade environment if the boot environment doing the patching is also boot archive unaware (meaning the kernel is pre 137137-09).

In the jumpstart scenario the immediate problem was solved by updating the boot archive by booting failsafe as previously described.  The Solution was to update the boot archive from the finish script after the patch cluster installation completed.  BTW, all patches in the patch cluster installed successfully per the /var/sadm/system/logs.finish.log.

In a standard jumpstart the boot device (install target) is mounted to /a, therefore adding the following entry to the finish script solved the problem:

/a/boot/solaris/bin/create_ramdisk -R /a

Depending on the finish script configuration, and variables the following would also work:

$ROOTDIR/boot/solaris/bin/create_ramdisk -R $ROOTDIR
Issue 3: This above issues are sometimes mis-diagnosed as CR 6850202: "bootadm fails to build bootarchive in certain configurations leading to unbootable system".

But CR 6850202 will only be encountered in very specific circumstances, all of which must occur in order to hit this specific bug, namely:

1. Install u6 SUNWCreq - there's  no mkisofs so we build ufs boot archive

2. Limit /tmp to 512M - thus forcing the ufs build to happen in /var/run

3. Have a separate /var - bootadm.c only lofs nosub mounts "/" when creating the alt root for DAP patching build of boot archive

4. Install 139555-08

You must have all 4 of above in order to hit this, i.e. step 4 must be installing a DAP patch such as a Kernel patch associated with a Solaris 10 Update such as 139555-08. 

Solution 3: Removing the 512MB limit (or whatever limit has been imposed) to /tmp in /etc/vfstab and/or adding SUNWmkcd (and probably SUNWmkcdS) so that mkisofs is available on the system is sufficient to avoid the code path that fails this way.

Booting failsafe and recreating the boot archive will successfully recreate the boot archive.

Here's further input from one of my senior engineers, Enda O'Connor:

If using Live Upgrade (LU), and LU on the live partition is up to date in terms of latest revision of the LU patch, 121430 (SPARC) and 121431 (x86), the boot-archive will be built automatically once users runs shutdown ( after luactivate to activate the new BE ).  This is done from a kill script in rcd.0.

If using a jumpstart finish script, or jumpstart profile to patch a pre-U6 image with latest kernel patches, then you need to run create_ramdisk from the finish script after all patching/packaging operations have been finished.  Alternatively, you can patch your pre-U6 miniroot to the U6 SPARC NewBoot level (137137-09), at which point the modified miniroot will handle the build of the boot_archive after the finish script has run.

If patching U6 and upwards from jumpstart, the boot_archive will get built automatically after finish script has run, so there's no issue in this scenario.

If using any home grown technology to patch or install/modify software on an Alternate Boot Environment ( ABE ), such as ufsrestore/cpio/tar for example, you must always run create_ramdisk manually before booting to said ABE.

Best Wishes,

Gerry.

Friday Jun 19, 2009

Zones Parallel Patching versus Update On Attach: When to use which one ?

The Zones Parallel Patching enhancement for the Solaris 10 patch utilities was released this week giving customers a choice of how to improve zones patching performance.

In the Zones "Update On Attach" section of a previous blog posting, I mentioned that the Zones "Update On Attach" feature could also be used to improve Zones patching perfomance.

Zones Parallel Patching is a true patching solution utilizing the 'patchadd' utility.  

Whereas Zones "Update On Attach" uses zones functionality similar to that used during zones creation to provide a pseudo-patching solution that does not utilize 'patchadd'. 

So which one to choose ?

Let's look at the two options in more detail:

Zones Parallel Patching

Zones Parallel Patching is an enhancement to the standard Solaris 10 patch utilities and is delivered in the patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

It works for all Solaris 10 systems. 

It also works well in conjunction with higher level patch automation tools such as xVM Ops Center. 

It can dramatically improve zones patching performance by patching non-global zones in parallel.  The global zone is still patched first.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

See my previous Zones Parallel Patching blog entry for further information.

Since it's a pure enhancement to 'patchadd', it's normal 'patchadd' functionality.  You can subsequently remove patches using 'patchrm', etc.  Nothing has changed except that it's now much faster to patch non global Zones with Zones Parallel Patching invoked.

Zones "Update On Attach"

The primary purpose of Zones "Update on Attach" is Zones migration from one server to another.  

For example, a database instance in a non-global zone hosted on a server has grown to the extent that the Sys Admin wants to transfer it to a better spec'd server which can better handle the workload.   The Sys Admin can detach it from the old server (e.g. a Sun4u) and reattach it to the new server (e.g. a Sun4v) using Zones "Update On Attach".   This will bring the OS Software level on the non-global zone up to the same level as the new server's global zone.

Zones "Update On Attach" can certainly be used for patching but there are limitations you need to be aware of as outlined below.

For example, detach the non-global zones from a system, apply a bunch of patches to the global zone, reattach the non-global zones using "Update On Attach" and viola, the non-global zones will be brought up to the same software level as the global zone (for OS type packages), effectively patching the non-global zones without using 'patchadd' at all.   This is typically even faster than using Zones Parallel Patching.  But there are limitations to this approach which users must be aware of (see below).

My senior engineer, Enda O'Connor, has just published an interesting article on The Zones Update on Attach Feature and Patching in the Solaris 10 OS

Zones "Update On Attach" limitations as a patching aid

Zones "Update On Attach" only works for packages which are SUNW_PKG_ALLZONES=true - i.e. typically OS level packages, and not application packages.

So when to use Zones Parallel Patching in 'patchadd' and when to use Zones "Update On Attach" ?

Here's what my senior engineer, Enda O'Connor, says:

"The Zones Update on Attach Feature and Patching in the Solaris 10 OS document may help customers understand how the technology works, applying a cluster via patching and via zones Update On Attach is not quite the same really.

It really depends on the patches being applied, i.e. applying a firefox patch via Update On Attach would not work if you wanted it to apply to the global zone and all non-global zones as well.

One has to understand how Update On Attach works and then apply that to the list of patches to see if it gets them to a desirable state.

There is no black or white answer here.

I'd recommend Zones Parallel Patching using 'patchadd' as it has a known outcome all the time, whereas Update On Attach makes it's own internal determination based on a number of things, that can vary from system to system ( e.g. inherited directories ).

But if time to patch is critical then if the customer does proper testing to validate things, and are happy with the results, then by all means use Update On Attach.

But using Update On Attach without:

1. Understanding how it determines what packages to update

2. Not inspecting the patches being applied.

...will most likely lead to grief at some point."

And my other senior engineer, Ed Clark, says:

"In terms of giving guidance on which technology to use, there are a number of considerations -- two of these considerations are:

1. Using Update On Attach to update sparse zones can require significantly more disk storage space than would be needed by applying patches with 'patchadd' (3-4 times as much space would not be uncommon i think), due to Update On Attach copying fully populated global zone 'undo' files into the non-global zones, as opposed to having patchadd build sparsely populated 'undo' files in the non-global zones.

2. If a customer is really concerned about the ability to back out patches reliably, then 'patchadd' is a lower risk option than Update On Attach -- 'patchrm' of a patch from a non-global zone that has a copy of the global zones 'undo' pkg data (as is the case after Update On Attach) may potentially have unexpected side effects." [although we have yet to see any actual cases of negative results from this.]

Conclusion

In general, we recommend using the Zones Parallel Patching enhancement in the patch utilities rather than the Zones "Update On Attach" feature as Zones Parallel Patching is standard patching functionality, only faster, whereas Zones "Update On Attach" is really designed for migrating zones from one server as another and was not primarily designed to speed up patching.  

Because Zones "Update On Attach" uses Zones functionality similar to the zone creation functionality, rather than 'patchadd' functionality, limitations exist on what will be patched (typically the OS but not applications) and there's the potential for anomalies around things like the "undo" files which would be used by 'patchrm' if patches applied using Zones "Update On Attach" were subsequently removed from the non-global zones using 'patchrm' (although we have yet to see any actual cases of serious issues resulting from this).

So in patching situations where time is absolutely critical, Zones "Update On Attach" may provide a good option, as long as it's well tested in the customer environment prior to deployment on production systems.

Remember too, Live Upgrade is also your friend in such situations, enabling you to patch an inactive boot environment while the system is still in production.   So a combination of Live Upgrade and Zones Parallel Patching would be ideal.

I hope you find this helpful!

Best Wishes,

Gerry.

Thursday Jun 18, 2009

Solaris 10 5/09 (Update 7) patch bundle now available!

The Solaris 10 5/09 (Update 7) patch bundle is now available for download from the SunSolve Patch Cluster & Patch Bundle Download Page.  Click on the "Solaris Update Patch Bundles" link.

As with previous patch bundles, it contains the patches which are included in the corresponding Solaris Update, in this case Solaris 10 5/09 (Update 7).

This is useful for Sys Admins who wish to bring all their systems up to the same patch level as the Solaris Update without wanting to upgrade to the release - for example, due to change control policy restrictions in their organizations.

See previous blog entries for previous Solaris Update patch bundles for further information.

About

This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today