Friday Mar 22, 2013

Solaris 10 1/13 patchset released and latest Solaris 10 Kernel PatchIDs

Posting updated June 6, 2013, with new Solaris 10 Kernel PatchIDs 150400-xx (SPARC) and 150401-xx (x86):

As usual, we've released a patchset of all the patches contained in Solaris 10 1/13 (Update 11):

We've also included an important post-S10U11 patch - 150125-01 (SPARC) / 149637-02 (x86) - in this patchset, which fixes ZFS Bug 15809921.  See Doc 1535270.1.

This patchset can be applied to any existing Solaris 10 system to bring all pre-existing packages up to the same software level as Solaris 10 1/13.

It is not the same as upgrading to Solaris 10 1/13 (available here), as upgrading will additionally install any new packages delivered in the Update. 

I've also updated my Solaris 10 Kernel PatchID sequence posting with the latest Solaris 10 Kernel PatchIDs, namely: 

  • The Solaris 10 1/13 Kernel patch, 147147-26 (SPARC) / 147148-26 (x86)
  • Post Solaris 10 1/13 Kernel patches have the PatchIDs 148888-xx (SPARC) / 148889-xx (x86)

Please note that there are no more planned updates to Solaris 10, so these latest Kernel PatchIDs - 148888-xx (SPARC) / 148889-xx (x86) - will continue to be used for the foreseeable future.

Murphy's Law strikes again!

No sooner had I written that Solaris 10 Kernel PatchIDs 148888-xx (SPARC) and 148889-xx (x86) were here to stay for the foreseeable future, than the integration of the SR-IOV feature into rev-04 of these patches made it prudent to rejuvenate them. 

So from July 2013, the Solaris 10 Kernel PatchIDs will change to be 150400-xx (SPARC) and 150401-xx (x86).

Dare I tempt fate again by saying these Solaris 10 PatchIDs are likely to remain the same for the foreseeable future ?

I've also updated my Useful Patch Related Downloads posting with links to the Solaris 10 1/13, Jan 2013 CPU, and latest Recommended patchsets.

Friday Oct 19, 2012

October 2012 Security "Critical Patch Update" (CPU) information and downloads released

The October 2012 security "Critical Patch Update" information and downloads are now available from My Oracle Support (MOS).

See http://www.oracle.com/technetwork/topics/security/alerts-086861.html and in particular Document 1475188.1 on My Oracle Support (MOS), http://support.oracle.com, which includes security CVE mappings for Oracle Sun products.

For Solaris 11, Doc 1475188.1 points to the relevant SRUs containing the fixes for each issue.  SRU12.4 was released on the CPU date and contains the current cumulative security fixes for the Solaris 11 OS.

For Solaris 10, we take a copy of the Recommended Solaris OS patchset containing the relevant security fixes and rename it as the October CPU patchset on MOS.  See link provided from Doc 1475188.1

Doc 1475188.1 also contains references for Firmware, etc., and links to other useful security documentation, including information on Userland/FOSS vulnerabilities and fixes in https://blogs.oracle.com/sunsecurity/

Wednesday Oct 12, 2011

Live Upgrade document updated and simplified

I forgot to let you know, but a couple of months ago, my colleagues, Don O'Malley and Ed Clark updated the Oracle Solaris Live Upgrade (LU) document describing the pre-requisites for Live Upgrade.

The original document was pretty convoluted and required several cups of strong coffee to parse.  The updated version is a little easier to understand, even without caffeine.

Thanks also to Beth Barrett, Rick Ramsey, and Jon Bowman who helped make this happen.

Wednesday Oct 05, 2011

Solaris 10 8/11 (Update 10) Patchset now available

Hi Folks,

The Solaris 10 8/11 (Update 10) patchset is now available from My Oracle Support.  Here's direct links to the common README and the SPARC and x86 downloads.  You need to be logged into MOS and have a valid support contract associated with your account in order to download the patchsets.

BTW: Please see my previous blog posting for details on other useful direct links to Solaris patch downloads and metadata.

As you may know by now, these patchsets will bring all pre-existing packages up to the same software level as the corresponding Solaris Update.  For example, all ZFS and Zones functionality is entirely contained in pre-existing packages, so applying the patchset will provide all the ZFS and Zones functionality and bug fixes contained in the corresponding Solaris Update.  

When we release the Solaris Update patchset, we try to fix any serious late breaking issues found with the corresponding Solaris Update patchset.  A list of additional patches added and the Caveats they address is contained in the patchset README.

Applying the patchset is not the same as upgrading to the Solaris Update release, as the patchset will not include any new packages introduced in the Solaris Update or any obsolete packages deleted in the Update.   

Please see this blog posting for lists of the new packages introduced in each Solaris Update to see if any of them are relevant to you.  If they are, then upgrade to a release which provides them.  If they're not, then applying the patchset may be a reasonable alternative to update your Solaris system. 

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  They are listed at the end of the SPARC and x86 patch lists.

Health Warning: Do not manually apply packages from a later Solaris release to an earlier Solaris release (e.g. by pulling individual packages from an ISO image) as this will result in an inconsistent system state which may lead to system corruption unless careful post-processing is done at the time such packages are applied to ensure that any patches applied to either the pre-existing packages on the system or pre-applied to the new packages been added are reapplied to the system to ensure both the pre-existing and new packages are at the same patch level.  Failure to do this will compromise the patch utilities ability to resolve patch dependencies leading to undefined results.  Even if you take the above steps, Support are likely to frown upon such shenanigans.  So don't do it.  If you need new packages, upgrade to a release which provides them.  Note, Live Upgrade packages are the only exception to this rule and the procedure for them is specified in the Live Upgrade documentation.  

Best Wishes,

Gerry.

Monday Aug 29, 2011

Using smpatch to apply Solaris Cluster patches and other enhancements

It is now possible again to use the in-built Solaris 10 patch automation utility, 'smpatch' / Update Manager, to download patches for products such as Oracle Solaris Cluster and Oracle Solaris Studio, as well as Oracle Solaris Operating System patches. 

It is now also possible again to use 'smpatch' / Update Manager on 3rd party hardware. 

To utilize these capabilities, the system must be registered or re-registered as outlined in https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1347266.1

These steps effectively switch 'smpatch' / Update Manager from using hardware serial number based access entitlement to User based access entitlement, similar to the access entitlement mechanism used when downloading patches via 'wget' or manually via My Oracle Support (MOS).

The following patches are required to provide this functionality:

SPARC
121118-19  SunOS 5.10: Update Connection System Client 1.0.19
123893-25  SunOS 5.10: Cacao Patch
123005-09  SunOS 5.10: Basic Registration Update
124171-08  SunOS 5.10: SCN Base cacao module patch
123630-04  SunOS 5.10: HTTP proxy settings patch
x86
121119-19  SunOS 5.10_x86: Update Connection System Client 1.0.19
123896-25  SunOS 5.10_x86: Cacao Patch
123006-09  SunOS 5.10_x86: Basic Registration Update
124187-08  SunOS 5.10_x86: SCN Base cacao module patch
123631-04  SunOS 5.10_x86: HTTP proxy settings patch

'smpatch' / Update Manager patch 12111[89]-19 introduces other significant changes due to the migration to Oracle back-end infrastructure.  The download server and security certs have changed.  As My Oracle Support supports ".zip" file download only, this patch mandatorily migrates 'smpatch' / Update Manager from using ".jar" downloads to using ".zip" downloads.

Caveat: There is currently an issue affecting LPS (Local Proxy Server) functionality following the migration to the Oracle back-end infrastructure.  This issue is currently being worked on.

Friday Jan 28, 2011

Resolving 'smpatch' / Update Manager issues

A number of customers have reported issues with 'smpatch' / Update Manager resulting from the recent migration to the My Oracle Support (MOS) infrastructure.

My colleagues BethB, PeterM, and EthanR have published Document 1288579.1 which explains what to do if you are unable to register systems & download patches via sconadm, smpatch, and Update Manager . This document is also accessible via the Oracle Sun OS Community page too.

I apologize most sincerely for any inconvenience caused.

Thursday Jan 13, 2011

List of new and up'rev'd packages in each Solaris 10 Update

Here are lists in .pdf (SPARC / x86) and OpenOffice (SPARC / x86) format of new and up'rev'd packages in each Solaris 10 Update release.

As you may know from my previous blog postings, Oracle Sun recommends customers to install or upgrade to the latest Solaris 10 Update in major maintenance windows. Based on a request from customers whose change control policies prevent them from upgrading, we've been producing Solaris Update Patch Bundles which bring pre-existing packages up to the same software level as the corresponding Solaris Update.  The difference is that the Patch Bundles don't provide new or up'rev'd packages introduced in the corresponding Solaris Update.

For customers considering use of the Solaris Update Patch Bundles, that raises the obvious question as to which packages are introduced or up'rev'd in each Solaris Update release.  The lists above answer that question.

Aside: As discussed in previous blog postings, all core Solaris OS packages are updated via patches.  The up'rev'd packages above refer to some 3rd party and community based apps included in Solaris (e.g. Mozilla Firefox, Thunderbird, etc.) which are updated via package updates (i.e. where one package version is removed and replaced with a later version).  This is to tie in better with the release strategy for such apps.

Many thanks to my colleague, Roisin Doran, for all her work in putting this together.

I'll ask Roisin to work with the Technical Writers to include updated versions of these lists in future Solaris 10 Update release documentation.

Friday Oct 15, 2010

Solaris Updates and Patch Bundles in the picture

A colleague of mine, Ken Brucker, has drawn this picture showing the composition of Solaris Updates which you may find useful to visualize the process.  

See the relevant slides from the presentation on http://blogs.sun.com/patch/entry/updated_customer_patching_presentation_and for more detailed information.

Saturday Sep 18, 2010

Solaris 10 9/10 (Update 9) Patch Bundle now available

The Solaris 10 9/10 (Update 9) Patch Bundles are now available from SunSolve and My Oracle Support (MOS).

These patch bundles provides the set of patch pre-applied into the corresponding Solaris 10 9/10 (Update 9) release image.  These patches provide all the Solaris 10 bug fixes which were available when the contents of the Solaris 10 9/10 release was finalized.

See http://blogs.sun.com/patch/entry/solaris_10_10_08_patch for further information on Solaris Update Patch Bundles.

See http://blogs.sun.com/patch/entry/oracle_sun_patches_now_available for information on how to access patch bundles on MOS.

Many thanks to the Patch System Test, Patch Operations and Distribution, and SunSolve teams for expediting the release of these patch bundles.

Thursday Sep 09, 2010

Solaris 10 9/10 (Update 9) released

Solaris 10 9/10 (Update 9) has been released.  See here for information and here for the download (remember to accept the license agreement at the top).  There's also a podcast and a dedicated Solaris blog.

A number of technical articles have been released, including George Wilson's video overview of ZFS enhancements in Solaris 10 9/10.

As with all Solaris Updates, Solaris 10 9/10 contains all available bug fixes which were available at the time that its contents were finalized, pre-applied into the Solaris Update image. 

It also contains a significant number of feature enhancements as described in the above links.

The corresponding Solaris Update Patch Bundle is currently in test and I expect that it should be released in a similar timeframe to previous Updates.  See http://blogs.sun.com/patch/entry/solaris_10_10_08_patch  for information on Solaris Update Patch Bundles.

All standard patches in Update 9 have already been released to SunSolve and My Oracle Support (MOS).  I've updated the Solaris 10 Kernel PatchID Sequence entry below with the Kernel PatchIDs for Solaris 10 9/10 (Update 9).

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  See the SPARC and x86 patch lists. 

<pet peeve>

Please note it is incorrect to refer to Kernel Patch 142909-17 (SPARC) / 142910-17 (x86) as the "Update 9 Kernel patch".  It is the latest Kernel Patch included in Update 9, but this Kernel patch can equally be applied to all previous Solaris 10 releases.   Solaris Updates are built from patches (and a few new packages), patches are not built from Solaris Updates.

</pet peeve>

Tuesday Nov 24, 2009

Do not apply packages from one Update onto a system installed with a different Update

I'd just like to make the following clear to customers:

Customers who install packages from one Update release (e.g. S10U6) on a system installed with another release (e.g. S10U3) risk corrupting their system.

You must not install packages from one Update release onto a system installed with any other Update release.

The reason for this is that all available patches are pre-applied into all packages for each Update release.   Patches and packages have a many-to-many relationship.  That is, one patch can patch many packages.   One package can be patched by many patches.

If you install a package from an Update release onto a system installed with a different Update release, you completely compromise future patch dependency checking as you've introduced patch metadata from a later Update release.   This is likely to lead to system corruption as further patches are applied.

'patchadd' checks that dependencies are satisfied when installing a patch.  If 'patchadd' finds any installed package patched with a patch which satisfies the dependency, it assumes the patch is applied to all packages.  This is done for performance reasons.  Hence, if a package from a later release is installed on the system, it's pre-applied patches may fool subsequent 'patchadd' invocations into thinking that a hard code dependency has been satisfied for all packages on the system when this is not the case.   The patch application will be allowed to continue, potentially corrupting the system.

The converse is also true.   If a package from an earlier Update is applied to a system, the patch delta from that Update to the Update installed on the system plus any additional patches installed on the system would need to be applied to the package to avoid a mismatch in software levels between the packages on this system, which could lead to incorrect patch dependency resolution and hence to system corruption.  Since this is difficult to get right, adding packages from a different Update release onto an installed system is, in general, unsupported. 

There are two exceptions to the above rules, for Live Upgrade and encryption packages.

  • Live Upgrade: If you are upgrading from Solaris 8 or Solaris 9 to Solaris 10, you need to apply the Live Upgrade packages from the Solaris 10 system to your Solaris 8 or Solaris 9 system.  See document 1004881.1 on My Oracle Support (MOS), http://support.oracle.com, for details.
  • SUNWcry\* encryption packages: These weren't included in the Solaris distribution prior to Solaris 10 8/07 (Update 4).  For pre-Update4 systems, the recommended way to get these packages is to upgrade to a later Update release.  For the reasons outlined above, while not recommended, a possible alternative is to download the packages from the Sun Download Center (SDLC), install them, and then re-apply any patches that patch the SUNWcry\* packages which you have already applied to the system (\*\*). Also, for the reasons outlined above, it is not recommended to apply the packages from the media of a later Update release, as you would need to ensure that all the other packages installed on the system are patched to the same or higher patch level  (e.g. by installing the appropriate Solaris Update Patch Bundle available from My Oracle Support, http://support.oracle.com) to avoid completely compromising future patch dependency checking on the system.

\*\* A way to check which patches need to be re-applied in this scenario is as follows:

cd /var/sadm/patch
egrep -i "PKG=SUNWcryr|PKG=SUNWcry" \*/log|cut -f1 -d /|sort -u

as in:

# cd /var/sadm/patch
# egrep -i "PKG=SUNWcryr|PKG=SUNWcry" \*/log|cut -f1 -d /|sort -u
127127-11
137137-09
139555-08
141444-09
#

The above example is on a system installed with Soalris 10 11/06 (Update 3).  The user would need to re-apply the patches listed above and be extremely careful to ensure they are applied in the correct dependency order as 'patchadd' will not be able to ensure the correct dependency order as it's dependency checking remains compromised until the added packages are brought up to the same patch level.

Please note that if a package from the same Update is applied to a system, then any additional patches already installed on the system that patch the added package must be re-applied to bring that package up to the same software level as the rest of the system.  This is called "incremental patching".   This is supported, but care must be taken.  The easiest way to do this is to reapply all patches installed on the system (as listed by 'patchadd -p').  This will bring the added package(s) up to the same software level as the rest of the system.  Again, you need to be extremely careful to ensure they are applied in the correct dependency order as 'patchadd' will not be able to ensure the correct dependency order as it's dependency checking remains compromised until the added packages are brought up to the same patch level as the rest of the system.

Best Wishes,

Gerry Haskins,
Director, Software Patch Services

Wednesday Nov 04, 2009

Solaris 10 10/09 Patch Bundle now available

I'm delighted to announce that the Solaris 10 10/09 (Update 8) Patch Bundle is now available for download by customers with a Solaris support contract.

Each Solaris Update Patch Bundle contains the equivalent set of patches which are pre-applied into the corresponding Solaris Update release image.

It is provided to enable customers who cannot upgrade for whatever reason to be able to patch systems up to the same patch level as the Update release.

Each Solaris Update is intensely tested as a unit by myriad QA teams across Sun.  Therefore, Solaris Updates and their corresponding Solaris Update Patch Bundles provide good quality "baselines" on which customers can standardize their deployments.

Standardizing deployments on such "baselines" also provides customers with a "safety in numbers" effect, as any pervasive issues are likely to be found and fixed quickly, so each customer benefits from the experience of others.

The Solaris Update Patch Bundle brings all existing packages up to the same software level as the Update release.   Any features which are entirely contained in pre-existing packages, such as Zones and ZFS functionality, are entirely available in patches and hence applying the Solaris Update Patch Bundle brings them up to the same functional level as the Update release.

However, installing the Patch Bundle is not completely equivalent to upgrading to the corresponding Solaris Update as the Patch Bundles do not include any new packages introduced in the Solaris Update release image.  Therefore, any new features which are dependent upon new packages will not be available by applying the Solaris Update Patch Bundle.

Here's a summary of the new packages in Solaris 10 10/09 (Update 8) which are not available in the Solaris 10 10/09 Patch Bundle:

SUNWhxge: SUN 10Gb hxge Ethernet Network Adapter Driver
SUNWio-tools: Administrative tools to modify the pci/pcie fabric
SUNWmrsas: LSI MegaRAID SAS2.0 HBA driver
SUNWpixman: Pixman Library
SUNWntp4r: NTPv4 (root)
SUNWntp4u: NTPv4 (usr)
SUNWntp4S: NTPv4 (source)
SUNWmptsas: LSI MPT SAS2.0 HBA driver

Please remember to apply the latest Sun Alert Cluster on top of the Solaris Update Patch Bundle in order to get all Solaris OS security, data corruption, and system availability fixes released since the final build of the Update release.

Please see previous blog entries for further details on Solaris Update Patch Bundles.

Top Tip: If you are installing in a zones environment, make sure you have the latest patch utility patches installed and Zones Parallel Patching configured before you apply a Solaris Update Patch Bundle as Zones Parallel Patching will improve non-global zone patching performance by ~300%.   See this blog entry for details.

BTW: There is no need to take any action to enable "Turbo-Charging SVR4 Package Installation" as the necessary patches are installed early on when installing the Solaris 10 10/09 Patch Bundle and will be automatically enabled for subsequent patch application when the bundle is applied to the live boot environment.  While "Turbo-Charging" has little affect when installing most patches, it does significantly speed up the application of a small number of older patches with non-optimized deletes file processing install scripts and so does speed up the Solaris 10 10/09 Patch Bundle installation somewhat.

Best Wishes,

Gerry.

Wednesday Sep 09, 2009

Patches for "Turbo-Charging SVR4 Package Install" are now available

I am delighted to announce that Casper Dik's "Turbo-Charging SVR4 Package Install" feature enhancement is now available by downloading and installing the following patches:

119254-70 (SPARC) / 119255-70 (x86): Patch utilites patch
121428-13 (SPARC) / 121429-13 (x86): Live Upgrade Zones Support Patch
121430-40 (SPARC) / 121431-41 (x86): Live Upgrade Patch
124630-28 (SPARC) / 124631-29 (x86): System Administration Applications, Network, and Core Libraries Patch

It is important to apply 119254-70 (SPARC) / 119255-70 (x86) and 121428-13 (SPARC) / 121428-13 (x86) if the system is running non-global zones.  Otherwise booting newly installed zones will fail until the pkgserv daemon exits, about 5 minutes after zoneadm install finished.  Zones which were already installed can be booted as expected.

"Turbo Packaging" is designed to dramatically improve the performance of install, upgrade, Live Upgrade, and zone creation on Solaris 10.  It delivers only a small improvement for patching performance.   (See my Zones Parallel Patching blog entry for information on dramatic patching performance improvements.)

For background reading on the "Turbo Packaging" feature, please see
http://www.opensolaris.org/jive/thread.jspa?messageID=358081 and http://arc.opensolaris.org/caselog/PSARC/2009/173/

The "Turbo-charging SVR4 Package Install" feature will also be included in the forthcoming Solaris 10 Update 8 release and will be documented in the Release Notes for that release.

Great work, Casper - well done!

Friday Jun 19, 2009

Zones Parallel Patching versus Update On Attach: When to use which one ?

The Zones Parallel Patching enhancement for the Solaris 10 patch utilities was released this week giving customers a choice of how to improve zones patching performance.

In the Zones "Update On Attach" section of a previous blog posting, I mentioned that the Zones "Update On Attach" feature could also be used to improve Zones patching perfomance.

Zones Parallel Patching is a true patching solution utilizing the 'patchadd' utility.  

Whereas Zones "Update On Attach" uses zones functionality similar to that used during zones creation to provide a pseudo-patching solution that does not utilize 'patchadd'. 

So which one to choose ?

Let's look at the two options in more detail:

Zones Parallel Patching

Zones Parallel Patching is an enhancement to the standard Solaris 10 patch utilities and is delivered in the patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

It works for all Solaris 10 systems. 

It also works well in conjunction with higher level patch automation tools such as xVM Ops Center. 

It can dramatically improve zones patching performance by patching non-global zones in parallel.  The global zone is still patched first.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

See my previous Zones Parallel Patching blog entry for further information.

Since it's a pure enhancement to 'patchadd', it's normal 'patchadd' functionality.  You can subsequently remove patches using 'patchrm', etc.  Nothing has changed except that it's now much faster to patch non global Zones with Zones Parallel Patching invoked.

Zones "Update On Attach"

The primary purpose of Zones "Update on Attach" is Zones migration from one server to another.  

For example, a database instance in a non-global zone hosted on a server has grown to the extent that the Sys Admin wants to transfer it to a better spec'd server which can better handle the workload.   The Sys Admin can detach it from the old server (e.g. a Sun4u) and reattach it to the new server (e.g. a Sun4v) using Zones "Update On Attach".   This will bring the OS Software level on the non-global zone up to the same level as the new server's global zone.

Zones "Update On Attach" can certainly be used for patching but there are limitations you need to be aware of as outlined below.

For example, detach the non-global zones from a system, apply a bunch of patches to the global zone, reattach the non-global zones using "Update On Attach" and viola, the non-global zones will be brought up to the same software level as the global zone (for OS type packages), effectively patching the non-global zones without using 'patchadd' at all.   This is typically even faster than using Zones Parallel Patching.  But there are limitations to this approach which users must be aware of (see below).

My senior engineer, Enda O'Connor, has just published an interesting article on The Zones Update on Attach Feature and Patching in the Solaris 10 OS

Zones "Update On Attach" limitations as a patching aid

Zones "Update On Attach" only works for packages which are SUNW_PKG_ALLZONES=true - i.e. typically OS level packages, and not application packages.

So when to use Zones Parallel Patching in 'patchadd' and when to use Zones "Update On Attach" ?

Here's what my senior engineer, Enda O'Connor, says:

"The Zones Update on Attach Feature and Patching in the Solaris 10 OS document may help customers understand how the technology works, applying a cluster via patching and via zones Update On Attach is not quite the same really.

It really depends on the patches being applied, i.e. applying a firefox patch via Update On Attach would not work if you wanted it to apply to the global zone and all non-global zones as well.

One has to understand how Update On Attach works and then apply that to the list of patches to see if it gets them to a desirable state.

There is no black or white answer here.

I'd recommend Zones Parallel Patching using 'patchadd' as it has a known outcome all the time, whereas Update On Attach makes it's own internal determination based on a number of things, that can vary from system to system ( e.g. inherited directories ).

But if time to patch is critical then if the customer does proper testing to validate things, and are happy with the results, then by all means use Update On Attach.

But using Update On Attach without:

1. Understanding how it determines what packages to update

2. Not inspecting the patches being applied.

...will most likely lead to grief at some point."

And my other senior engineer, Ed Clark, says:

"In terms of giving guidance on which technology to use, there are a number of considerations -- two of these considerations are:

1. Using Update On Attach to update sparse zones can require significantly more disk storage space than would be needed by applying patches with 'patchadd' (3-4 times as much space would not be uncommon i think), due to Update On Attach copying fully populated global zone 'undo' files into the non-global zones, as opposed to having patchadd build sparsely populated 'undo' files in the non-global zones.

2. If a customer is really concerned about the ability to back out patches reliably, then 'patchadd' is a lower risk option than Update On Attach -- 'patchrm' of a patch from a non-global zone that has a copy of the global zones 'undo' pkg data (as is the case after Update On Attach) may potentially have unexpected side effects." [although we have yet to see any actual cases of negative results from this.]

Conclusion

In general, we recommend using the Zones Parallel Patching enhancement in the patch utilities rather than the Zones "Update On Attach" feature as Zones Parallel Patching is standard patching functionality, only faster, whereas Zones "Update On Attach" is really designed for migrating zones from one server as another and was not primarily designed to speed up patching.  

Because Zones "Update On Attach" uses Zones functionality similar to the zone creation functionality, rather than 'patchadd' functionality, limitations exist on what will be patched (typically the OS but not applications) and there's the potential for anomalies around things like the "undo" files which would be used by 'patchrm' if patches applied using Zones "Update On Attach" were subsequently removed from the non-global zones using 'patchrm' (although we have yet to see any actual cases of serious issues resulting from this).

So in patching situations where time is absolutely critical, Zones "Update On Attach" may provide a good option, as long as it's well tested in the customer environment prior to deployment on production systems.

Remember too, Live Upgrade is also your friend in such situations, enabling you to patch an inactive boot environment while the system is still in production.   So a combination of Live Upgrade and Zones Parallel Patching would be ideal.

I hope you find this helpful!

Best Wishes,

Gerry.

Thursday Jun 18, 2009

Solaris 10 5/09 (Update 7) patch bundle now available!

The Solaris 10 5/09 (Update 7) patch bundle is now available for download from the SunSolve Patch Cluster & Patch Bundle Download Page.  Click on the "Solaris Update Patch Bundles" link.

As with previous patch bundles, it contains the patches which are included in the corresponding Solaris Update, in this case Solaris 10 5/09 (Update 7).

This is useful for Sys Admins who wish to bring all their systems up to the same patch level as the Solaris Update without wanting to upgrade to the release - for example, due to change control policy restrictions in their organizations.

See previous blog entries for previous Solaris Update patch bundles for further information.

Friday May 08, 2009

Ooops! Please ignore empty feature kernel patches accidentally released

A software glitch caused a number of empty patches to be accidentally released last week.   The PatchIDs are:

  • 139555, revisions -01 to -07: Solaris 10: Kernel Patch
  • 139556, revisions -01 to -07: Solaris 10_x86: Kernel Patch
  • 139981-01: Solaris 10_x86: md patch

The above patches have been removed from SunSolve.   All of these patches were empty (i.e. they contained no payload packages), so they are incapable of being installed or causing any problems on a customer system.   This notice is simply to clear up any confusion.

The correct Kernel patch revisions are now available.  These are:

  • 139555-08, which is the Solaris 10 Kernel patch included in Solaris 10 5/09 (Update 7)
  • 139556-08, which is the Solaris 10 Kernel patch included in Solaris 10_x86 5/09 (Update 7)

The correct "md" patch revision will be released shortly.  This is:

  • 139981-03: Solaris 10_x86: md patch

On behalf of Sun, I apologize most sincerely for any confusion or inconvenience caused. 

Thursday Apr 02, 2009

Patching Zones goes Zoom!

My colleague, Jeff Victor, has written an excellent and informative blog posting on how to improve patching performance, "patching zones goes zoom!".   Enjoy!

Thursday Dec 04, 2008

Patching enhancements and other stuff

New title, same role, same me

I was promoted to Director, Software Patch Services in September.  The last couple of months have been quite hectic, as I've suddenly got a whole new bunch of buddies in Marketing and elsewhere who want some of my time.  That's a good thing, and I believe it will help me to drive and co-ordinate improvements for you, our customers, patching experience. 

Resources are limited and, as always, I'm interested in getting your thoughts as to what areas I should concentrate on next.  

Some of the stuff we're currently working on is outlined below as well as other information which I hope you will find useful.

Solaris 10 10/08 Patch Bundle

The Solaris 10 10/08 Patch Bundle, which delivers the equivalent set of patches to the Solaris 10 10/08 (Update 6) release image, is now available from SunSolve.  See my blog entry below on the Solaris 10 5/08 (Update 5) Patch Bundle for further information on why we produce it, what it contains, why you might wish to use it, how to download it, etc.

Recommended and Sun Alert patch cluster contents updated

I discussed the purpose of, and difference between, the Solaris Recommended and Sun Alert patch clusters in a previous blog posting. To recap:

The "Recommended" Cluster contains the latest revision of any Solaris OS patch which addresses a Sun Alert issue.  That is, a fix for a Security, Data Corruption, or System Availability issue.  The cluster also contains the latest revision of the patch utility patches to ensure correct patch application and any patch required by any other patch in the cluster.

The Sun Alert Cluster is newer, and contains the minimum revision of any Solaris OS patch which addresses a Sun Alert issue. The cluster also contains the latest revision of the patch utility patches to ensure correct patch application and any patch required by any other patch in the cluster.  Therefore, the Sun Alert Cluster provides the minimum amount of change to fix all Solaris OS Sun Alert issues. 

Both clusters are updated whenever a new patch meeting their inclusion criteria is released.  The Sun Alert Cluster changes less frequently than the "Recommended" Cluster as it contains only what is really needed to address Sun Alert issues and apply the patches.

One of my team members has been reconciling the cluster contents against the Sun Alert reports and the cluster contents have been updated as a result.  Some issues where found, largely to do with patches for things like GNOME which are also part of the Solaris OS.  A process has been put in place to ensure the cluster contents match the patches specified in the Sun Alert reports.   

Keeping as up to date as possible with the SunAlert or Recommended Cluster contents is advisable.   Remember also to keep firmware up to date.

BTW: The monthly EIS (Enterprise Installation Standards) patch baseline is based upon the Recommended Cluster contents but also includes ca. 150 additional patches to address irritants which are not Sun Alert fixes and includes patches for SunCluster, SunVTS, etc.  The monthly EIS patch baselines are available through xVM Ops Center and Sun Proactive Services.

I am planning to merge the Recommended and Sun Alert patch clusters into a single cluster using the Sun Alert cluster criteria as having two very similar clusters tends to confuse customers unnecessarily.  

I also intend to merge the two cluster pages on SunSolve as one is essentially a better formated subset of the other. 

ZFS and Zones features fully contained in patches

As I've mentioned previously, there's effectively a single customer visible code branch for each Solaris named release.  That means that there's one set of patches for all of Solaris 10, a separate set for Solaris 9, and a separate set for Solaris 8.  Within a named release, e.g. Solaris 10, the same set of patches will apply to any of the Solaris 10 releases, from the original Solaris 10 3/05 release right up to the current Solaris 10 10/08 (Update 6) release.  This simplifies System Administration and enables Sun to provide very long term support at reasonable cost for each Solaris named release. 

A consequence of effectively having a single code branch for each Solaris named release is that any change to pre-existing packages will be delivered in patch format.

New features are typically only added to the current Solaris named release, which is currently Solaris 10.  (They are also available via OpenSolaris.)

This means that if new features don't add any new packages, then the entire feature functionality is fully available in patches.  Customers can utilize the new features by simply applying the appropriate patches to their existing Solaris 10 system.  This is the case with all current Zones and ZFS\* functionality, including neat features like ZFS Root, ZFS Boot, and Zones "Update on Attach".

Other features which deliver new packages are only available from the Solaris Update release in which they were first included.  So, for example, if a new package was first delivered in Solaris 10 8/07 (Update 4), then a customer wishing to use that feature would need to install or upgrade to the Solaris 10 8/07 (Update 4) or subsequent update release image.   Such features are not available in patches.

\*OK, we cheated with ZFS.  ZFS does deliver new packages, but they are streamed into existence from a patch.  This type of patch is called a "genesis" patch, but they are hard to perfect, so we don't intend to release any more "genesis" patches.

Improving Zones Patching Performance

Zones Parallel Patching

My team has been working with those awfully nice folks in the Sustaining organization to deliver a Zones Parallel Patching enhancement to the patch utilities to dramatically improve Zones patching performance.  We have a fully stable prototype which has been given to selected Beta customers to trial. 

For a simple T2000 with 5 sparse non-global zones, the performance improvement is >3x.  On systems with optimized I/O (as Zones patching is primarily I/O bound), we expect the performance improvement to be even better.  A configuration file will allow users to select how many Zones to patch in parallel.  This will typically equate to the number of processors or threads available on the target system.

The general release of this feature is planned for April 2009.

Zones "Update on Attach" 

The Kernel patch associated with Solaris 10 10/08 (Update 6), 137137-09 (SPARC) / 137138-09 (x86) contains some cool new features, such as ZFS Root, ZFS Boot, and Zones "Update on Attach".  Beware, installing this patch requires significant free disk space to install!  See Sun Alert http://sunsolve.sun.com/search/document.do?assetkey=1-66-246207-1

Zones "Update on Attach" is a very cool feature indeed.

For example, if the patch level of non-global Zones is out-of-sync with respect to the global Zone, e.g. because the non-global Zones ran out of disk space during patch application, Zones "Update on Attach" provides a very neat way to bring the Zones back into sync.  Simply detach the affected non-global Zones, apply Kernel patch 137137-09 (SPARC) / 137138-09 (x86) to the global zones, and reattach the affected non-global Zones using 'zoneadm -z <zone-name> attach -u'.  The non-global Zones will be automagically updated to the same patch level as the global Zone.  Neat!

There are other interesting possibilities.  For example, detach all non-global Zones, apply an arbitrary set of patches to the global Zone (including 13713[78]-09), and reattach the non-global Zones using 'zoneadm -z <zone-name> attach -u'.  Viola!, the non-global Zones will be automagically updated with all of the patches applied to the global Zone.  Way neat!  And more importantly, way faster than even the Zones Parallel Patching solution we're working on.  And even better, it's available now!  This could be a key solution for customers having difficulty completing patching updates on Zones systems during tight maintenance windows.

We are working to explore potential caveats.  For example, when a patch is applied using 'patchadd' to a non-global zone, an "Undo.Z" file containing the data necessary to back out the patch is created specifically for each non-global zone to which the patch is applied.   Using Zones "Update on Attach" to patch non-global Zones will cause the "Undo.Z" file from the global Zone to be propagated to the non-global Zones.  This could theoretically cause issues if the patch is subsequently backed out (e.g. data from global Zone config files could potentially be merged into non-global Zone config files during patch backout which could potentially cause issues), although we've never actually encountered such an issue.  BTW: The same caveat applies to creating non-global Zones after the global Zone has been patched.  Again, we have yet to see this causing an actual issue, so it appears to be more of a theoretically caveat than a practical issue.

Improvements to 'smpatch' and Update Manager

The way the PatchPro analysis engine for 'smpatch' and Update Manager used to work was fine in theory, but in practice was what I call "a process with too many moving parts".   Too many steps had to happen correctly for the overall result to be correct.  In Six Sigma terms, there was too much error opportunity.  Occasionally, it would end up recommending a SPARC patch for an x86 system or a Solaris 8 patch for a Solaris 10 system.  Not surprisingly, its reputation suffered.

I'm pleased to say that a major overhaul to dramatically simplify the back end processing of 'smpatch' and Update Manager has just been rolled out by their engineering team.  The way 'smpatch' and Update Manager work is that Realization Detector(s) are associated with each patch.  These Realization Detectors determine whether it's appropriate to recommend a patch for application on a target system.  In the vast majority of cases, the Realization Detectors are simply comparing the packages contained in the patch to the packages installed on the system to see if the patch is applicable.  The enhancement is to replace these myriad Realization Detectors, which could potentially contain coding bugs, with a single Generic Realization Detector to map patch packages to packages on the target system.  It looks at the package name, package version, and package architecture fields (in pkginfo) for each package in the patch, and compares them to the same values for the packages installed on the target system.  If they match, the patch is recommended, else not.  Guess what, this is exactly how 'patchadd' decides whether a patch is applicable or not when installing a patch.  It's also how 'pca' works too in determining which patches to apply.

A few specialist Realization Detectors remain for a small number of patches which require special handling.

The changes to 'smpatch' and Update Manager should dramatically improve the reliability of these tools and the accuracy of their patching recommendations.

One remaining distinction between 'smpatch' / Update Manager and 'pca' is that 'pca' "knows" about all current Sun patches via the patchdiag.xref file, whereas 'smpatch' / Update Manager "knows" about all patches containing a 'patchinfo' file, including older patch revisions.  All Solaris OS and Java Enterprise System (middleware) patches contain a 'patchinfo' file.  These account for 49% of patches.  For patching the Solaris OS, the tools should produce similar results.  A decision was made not to "auto-include" all other patches for 'smpatch' and Update Manager, as it was felt that the explicit step of the patch creator including a non-blank PATCH_CORRECTS realization detector specification line in the 'patchinfo' file to signal that the patch was suitable for patch automation was potentially useful.  (Don't worry about what value the PATCH_CORRECTS field has.  This is overriden by the Generic Realization Detector in the vast majority of cases.  It has no meaning from a customer perspective.)

This enhancement is not an attempt to undermine 'pca'.  It's simply to improve 'smpatch' and Update Manager.  I will continue to work closely with Martin Paul to give him heads-ups on any initiative which may impact 'pca' and resolve any issues with patchdiag.xref.

One thing I want to do when I can free up some resources, is a comparative study of the patching recommendations of the various available patch automation tools, 'smpatch' / Update Manager, 'pca', UCE (a.k.a Sun Connection Satellite),  xVM Ops Center\*, and TLP (Traffic Light Patching) which is used by Sun Proactive Services to provide tailored patching solutions for customers in conjunction with SRAS (Sun Risk Analysis Service) and the EIS (Enterprise Installation Standards) methodology, with a view to ensuring that the patching recommendations of the various tools are coherent and consistent, with the higher value tools providing more sophisticated analysis.  It's part of my efforts to co-ordinate patching improvements to improve our customers' patching experience.

\*xVM OC also utilitizes the monthly EIS patch "baselines".

Same Patch Entitlement policy, new Patch Entitlement implementation

Solaris changed its business model a few years ago from selling Solaris and providing patches for free to a model of giving away the software releases for free and charging for patches. 

The policy is that patches delivering new security fixes will remain free to all customers, irrespective of whether or not they have a support contract, but most other patches require that customers have a valid support contract to access them.  (See my earlier blog entry on the subject.)

All fixes will all be available for free in the next Solaris Update release (and OpenSolaris), so customers not willing to pay for a support contract can still get the fixes by installing or upgrading to the next Solaris Update release.  They'll just need to wait for it to ship.  Alternatively, they can use OpenSolaris.

This policy is not changing.

What is changing is the implementation of patch entitlement to ensure it matches the policy.  Currently, circa 60% of Solaris patches are free, including most of the key patches.  Under the new entitlement implementation, 18% of Solaris patches will remain free, including the specific revision of all Solaris patches which include new security fixes.  The rest will require a valid support contract to access. 

Any of the following support contracts will provide access to all Solaris patches and patch clusters: a Solaris subscription, a Software Support Contract, a Sun System Service Plan for Solaris, a Sun Spectrum Storage Plan, or a Sun Spectrum Enterprise Service Plan.  Since the names of the support contracts change from time-to-time, this list may change.

The new implementation will roll out in Phases, starting this month.  The roll-out should be transparent to customers with valid support contracts.

Patch signing certificate renewal

The signing certificate used to sign Sun patches expires shortly.  A new signing certificate will be rolled out in January and instructions provided on how to adopt it.

Customers who download the unsigned patch versions will not need to take any action.

"Accumulation-only" patches

The "SplitGate" source code management model we first introduced in Solaris 10 8/07 (Update 4) has dramatically improved Solaris 10 patch quality.  A side-effect of the "SplitGate" model is that base PatchIDs (the first 6 digits) change at the end of each Update release.  See my earlier Solaris 10 Kernel PatchID Sequence posting.

In the "SplitGate" model, when building an Update release, we effectively have two parallel source code gates, one called the Sustaining Gate containing just the bug fixes we need to release to customers in patches asynchronous to the Update release, and the other called the Update Gate containing a superset of the the Sustaining Gate and as well as new features and less critical bug fixes which will be released as part of the Update release. 

The two gates remain separate (split) for the duration of the Update release build process.  Once the Update release has reached release quality, the Update Gate is promoted to become the new Sustaining Gate and the process repeats.  Since the Update Gate is always a strict superset of the Sustaining Gate, no regressions should result from the promotion of the Update Gate to become the new Sustaining Gate.  Each patch in the old Sustaining Gate is obsoleted by a corresponding patch from the Update Gate which has accumulated its contents.  When the Update is released, these new PatchIDs are released to SunSolve.  This is why you see the base PatchIDs changing after each Update release. 

If the Update Gate patch doesn't contain any additional code changes over the corresponding Sustaining Gate patch, then there's no need for customers to install the new Update Gate patch.  Such patches are called "accumulation-only" patches and can be identified as they have a different base PatchID (the first 6 digits) but don't contain any additional CR numbers over the Sustaining patch which they obsolete.

The reason Sun releases these "accumulation-only" patches is because some customers insist that all of the PatchIDs pre-applied into a Solaris Update release image be also available from SunSolve.

Wednesday Sep 10, 2008

Why it is now more important to keep your SPARC and x86 Firmware up to date

My colleague, Damien Farnham, from the Performance Regression Benchmarking team, has written an informative blog posting on why it is increasingly important to keep SPARC firmware up to date (similar to the importance of keeping x86/x64 firmware up to date).  A lot has changed since the humble OBP!

Please see Best Practise In Firmware Patching for the T1000/T2000/T5X20 platforms.

And he's now added an entry for x86 Firmware too.

See also the Firmware section on the Big Admin Patching Hub, especially the FAQ.

One area you may not be aware of is that upgrading Firmware can lead to significant performance improvements in certain circumstances.

Monday Jun 09, 2008

Solaris 10 05/08 (Update 5) Patch Bundle

Last week, the Solaris 10 05/08 (Update 5) Patch Bundle was released on SunSolve.  The patch bundle provides another option to customers when deciding their patching strategy to maintain their Solaris systems.

What is the Solaris 10 05/08 Patch Bundle ?

The Solaris 10 05/08 Patch Bundle contains the equivalent set of patches contained in the Solaris 10 05/08 (Update 5) release image.

Why use the Solaris 10 05/08 Patch Bundle ?

The Solaris 10 05/08 Patch Bundle was created as a result of direct customer feedback after the Solaris 10 08/07 (Update 4) release.  New hardware may require a specific minimum Solaris 10 Update release such as the Solaris 10 05/08 (Update 5) release.  Some customers may wish to bring their other existing Solaris 10 systems up to the same patch level as the new hardware running Solaris 10 05/08.  The recommended way to do this is to upgrade the existing systems to the Solaris 10 05/08 release using either regular Solaris Upgrade or Solaris Live Upgrade.  But some customers may have policies in place which make it difficult to upgrade but OK to patch a system.  The Solaris 10 05/08 Patch Bundle facilitates such customers to bring their existing systems up to the equivalent patch level to the Solaris 10 05/08 (Update 5) Release.  In theory, this should mean that pre-existing functionality on all of the customers' systems should react the same, warts and all. This makes for a more homogeneous environment which may help lower support costs.

The Solaris 10 Update releases are very intensely tested by a wide variety of QA teams within Sun.  Therefore, the functionality contained in the patches within the Solaris 10 05/08 Patch Bundle have been intensely tested as a unit through the testing performed on the Solaris 10 05/08 (Update 5) release image.  Additional testing of the Solaris 10 05/08 Patch Bundle has also been performed by the Patch System Test team.  Therefore, the Solaris 10 05/08 Patch Bundle provides a well tested "baseline" option on which to standardize systems.

So while the patch bundle may deliver more change than some other patching strategies, that change has been well tested as a unit and hence may actually reduce the risk of introducing regressions when compared to "dim sum" patching (i.e. choosing an arbitrary combination of patches).  Note that intensive processes are also in place to ensure "dim sum" patching works, and it's rare to encounter a problem caused by "dim sum" patching.

How does the Patch Bundle differ from the Solaris 10 05/08 (Update 5) release image ?

The Solaris 10 05/08 (Update 5) release is a complete Solaris release image.  It contains new packages to support new features in the Solaris 10 05/08 (Update 5) release as well as all Solaris patches which were available when the Update was built.  The patches are pre-applied into the Solaris 10 05/08 release image.  This means that one doesn't have to spend time adding the patches using 'patchadd'.  On the flipside, since the patches are pre-applied into the release image, they cannot be backed out using 'patchrm'.  This isn't generally a problem as the Solaris Update release images are very intensely tested.  One can do a fresh install of the Solaris 10 05/08 (Update 5) release, or upgrade to it from an earlier Solaris release.

The Solaris 10 05/08 Patch Bundle contains the equivalent set of patches to the Solaris 10 05/08 (Update 5) release. The patch bundle does not include the new packages contained in the Solaris 10 05/08 (Update 5) release.  Therefore, new features in Update 5 which depend upon new packages introduced in that release will not be available in the patch bundle.  However, as discussed in a previous blog entry, any change to pre-existing code is delivered in a patch.  This includes features as well as bug fixes.  Therefore some feature enhancements will be available in the patch bundle.  ZFS, for example, is typically self-contained in patches and hence ZFS enhancements will typically be available via the patch bundle as well as via the Update release image. So will most Zones enhancements.  The patch bundle is simply a collection of patches with an install order file (patch_order) and an install script wrapper (installbundle.sh) around 'patchadd'.  Patches in the patch bundle can be backed out using 'patchrm', so long as the '-d' (no save) option wasn't used when applying the patch bundle.

There are a number of "special" or "script" patches included in each Solaris Update release.  These patches are used to correct issues in how patches are pre-applied into the Solaris Update release image and have no purpose whatsoever outside of the Solaris Update release process.  Therefore, these "special" or "script" patches are not released to SunSolve and are not included in the patch bundle.  See the Solaris 10 05/08 Patch Bundle README file for further information on these and other minor differences between the patch set pre-applied in the Solaris 10 05/08 release image and the patch set included in the Solaris 10 05/08 Patch Bundle.

Access 

The Solaris 10 05/08 Patch Bundle is available from the usual patch cluster location. 

Log onto Sunsolve, click on Patches and Updates, then Recommended Patch Clusters and scroll down the box under "Recommended Solaris Patch Clusters, J2SE and Java Enterprise System Clusters" to the Solaris 10 SPARC 05/08 Patch Bundle and Solaris 10 x86 05/08 Patch Bundle entries.  

The cluster is chunked to aid download.  There are 2 chunks for x86 and 3 chunks for SPARC. 

Follow the download instructions to the right of the scroll-down box or read the README file for any chunk.

As with all patch clusters, you need a valid support contract to download the cluster.   The following support contracts include access entitlement to Solaris patches and Patch Clusters (BTW: Software Update = patch), plus a wide range of additional support services:  Solaris Subscriptions, which includes Basic, Standard, Premium, and Solaris Everywhere Service Plans (compare here); Sun Software Service Plans, including Basic, Standard, Premium, and Premium Plus; Sun System Service Plans for Solaris, which includes Bronze, Silver, Gold, or Platinum options (compare here); or a Sun Spectrum Enterprise Service Plan.  See also http://www.sun.com/servicelist/ for country specific details.

Installation

Read the Patch Bundle README file for full installation instructions.

The patch bundle can be installed either on the active boot environment (i.e. the live system) or an inactive boot environment. 

Patching an inactive boot environment is recommended as, depending on the starting patch level of the target system, it may involve less system downtime as only a single reboot is required at the end to activate the boot environment. 

If you patch the active boot environment (i.e. the live system), then depending on the starting patch level of the target system, you may need to reboot an x86 system up to three times (twice at specific points during the installation process and once at the end) and a SPARC system up to two times (once after installing Kernel patch 118833-36 and once at the end).  See the patch bundle README for details.

The Solaris 10 05/08 Patch Bundle includes a new install script, installbundle.sh, which guides users through the installation process. 

The patches are ordered in such a way as to process any reboots required when patching an active boot environment as near the start of the installation process as possible.  This is to facilitate System Administrators by allowing them to get over the interim reboots early in the process and kick off the final patching sequence and let the process complete. 

The screen output and logfiles produced are also designed to be as clear and self-explanatory as possible, providing both overview and drill-down capabilities.

Approximate Installation Time 

How long it will take to install the Solaris 10 05/08 Patch Bundle will depend upon a number of factors:

  • The speed of the hardware and its I/O.
  • Which Solaris 10 release is installed on the target system and what patch level the system is at.  The higher the Solaris 10 Update release or patch level, the quicker the patch bundle will apply.
  • Whether Zones are installed on the system and which type of Zone.  Currently, the time to apply the cluster to each whole root non-global Zone will be approximately linear - i.e. multiple the install time by the number of whole root non-global Zones on the system.  Sparse root non-global Zones will be a little faster. (BTW: Sparse root non-global Zones is the recommended option when creating non-global Zones.)  As mentioned in a previous blog posting, there is a project in development to improve Zones patching performance.

For example, I installed the Solaris 10 x86 05/08 Patch Bundle on a v65x running the original Solaris 10 3/05 "FCS" (First Customer Shipment) release with no additional patch applied (worst case) and no non-global Zones.  I applied the patch bundle to the active boot environment.  Installation took a total of 3 hours and 58 minutes plus 3 reboots (see the Patch Bundle README for an explanation of the reboots when patching an active boot environment).

Conclusion

The Solaris 10 05/08 Patch Bundle will not suit everyone.  It is a large collection of patches and hence is slow to download and install.

As described in a previous blog posting, the Sun Alert patch clusters (available from the same location on SunSolve - see above) provide the minimum amount of change to address the most critical Solaris issues.  The Sun Alert cluster contains all available Solaris patch fixes for Security, Data Corruption, and System Availability issues. New versions of the Sun Alert cluster are posted whenever a new patch to fix a Sun Alert issue becomes available.  Customers should try to keep as current as possible with the contents of the Sun Alert clusters.

For customers who want to bring all their systems to the Solaris 10 05/08 (Update 5) release patch level, installing or upgrading to the Solaris 10 05/08 (Update 5) Release image remains the recommended option where feasible.  The Solaris 10 05/08 Patch Bundle was simply created in response to a demand from customers for an alternative option where upgrading was not feasible due to internal customer policies.

Since Solaris Update releases are intensely tested, the patch bundle provides a good quality patch "baseline" on which to standardize systems.

From customer feedback to date, the next Patch Bundle for the equivalent set of patches for Update 6 is likely to also be a complete set of patches from Solaris 10 3/05 "FCS" (First Customer Shipment - i.e. the original Solaris 10 release) and not an incremental bundle just containing the patch set delta between Updates 5 and 6 as I had previously suggested.  Feel free to post a comment with your preference.

Enjoy!

Tuesday Jan 22, 2008

Patch Automation Tools

First of all, let me say that my personnel experience of Sun's patch automation tools is limited.  I work upstream from the SysNet and Services groups who produce most of Sun's patch automation tools, so I and my team mostly patch from first principles using the basic Solaris patch utilities, patchadd and patchrm.

My team does have some experience of working with some of the patch automation tools.  I've supplemented this with information from SysNet and Services folk.

Sun Connection 1.1.1 Satellite (a.k.a. UCE) and xVM Ops Center 1.0

The official Sun patching tool of choice is now xVM Ops Center, which contains an enhanced version of Sun Connection 1.1.1 Satellite Edition.

Sun acquired Aduva a couple of years ago.  Aduva has a track record of providing patch and update automation tools for multiple Operating Systems.

The next-generation Aduva-based tools are coming on stream. Sun Connection 1.1.1 Satellite is based on Aduva.  Note, "Satellite" has a completely different back end to the Sun Update Connection Hosted edition and Solaris Update Manager, which are based on PatchPro (see below).

I'm hearing good things about the Satellite.  I understand that its initial target market is customers with 50+ systems.

Sun Connection 1.1.1 Satellite Edition is based on Aduva Onstage and Update Connection Enterprise.   A central server (Satellite) at the customer site is used to analyze and update all attached client systems in a fully automated manner.  It builds upon a central Knowledge Database fed by Sun.  It covers the provisioning of patches, packages, config files and scripts.  It is available to customers who pay for it.

Sun Connection 1.1.1 Satellite provides a solution for customers primarily interested in patch and package provisioning.  There is a 10 minute demo introducing you to some of the key features of Sun Connection Satellite at http://frsun.downloads.edgesuite.net/sun/07D01031/SunConnectSatellite.html, or alternatively there is a more detailed 32 minute demo at http://frsun.downloads.edgesuite.net/sun/07D01032/SunConnect.html

Sun Connection Satellite is a component of the xVM Ops Center

xVM Ops Center is a merge of Sun Connection and N1SM.  Here's a BigAdmin article on Patching Solaris using Sun xVM Ops Center.  The monthly patch baselines referred to are the patch sets in the monthly EIS DVD release (see below).

For further information, please see the Sun Connection hub's Product Tour page on BigAdmin.

EIS

EIS stands for Enterprise Installation Standards and originated from Sun field personnel wanting to develop best practice installation standards for systems installed on customer sites.

EIS has proved extremely popular with Sun field personnel and approved partners.  It's widespread adoption was due to it successfully addressing a real need.  I view it's widespread adoption among field personnel and OEMs as proof positive of its efficacy.

The EIS patch baseline goes through QA testing prior to release.  The images installed by Sun's manufacturers on servers are also based upon the EIS patch baseline.  Additional testing by Sun's manufacturers plus feedback from the EIS user community raises confidence in the EIS patch baseline content further.  Since many system installations world-wide use the EIS methodology, any inherent problems will quickly appear and can be dealt with.  In the event of there being issues with the EIS patch baseline recommendations are communicated to the EIS community.

This same EIS set of patches which are considered by Sun Field Engineers as best practice to install on a new system, can also be used to patch existing systems to the same patch level.  The EIS set of patches is based on the Recommended Patch Cluster for the Solaris OS with additional patches included by the Field Engineers for additional products and to address irritating issues which do not meet the criteria for inclusion in the Recommended Patch Cluster.

The EIS patch baseline covers Solaris and other products such as SunCluster, SunVTS, SSP, SMS, QFS, SAM-FS, and includes patches which provide firmware updates.

EIS has traditionally only been available via Sun Services personnel but is now available direct to customers via Sun Connection Satellite.  This provides a good option to customers to patch to a defined and tested patch baseline.  See the Sun Connection blog for further information.

pca

pca is a popular 3rd party tool developed by Martin Paul.  I've only ever heard positive feedback about pca.

pca is available from http://www.par.univie.ac.at/solaris/pca/

To try out pca, just run this on any Solaris machine:

  $ wget http://www.par.univie.ac.at/solaris/pca/pca
  $ chmod +x pca
  $ ./pca

pca is a good solution for customers interested in a simple, easy to use, patch automation tool.

smpatch, Update Manager, and Sun Connection Hosted Edition

smpatch is a command line tool and part of Solaris.   It allows customers to analyze and update Solaris with current patches.  For customers without a valid support contract,  only security and driver patches are available.  For customers with a valid support contract, all patches are available.

updatemanager is a GUI wrapper around smpatch and is also part of Solaris.  It can be used to see what patches/updates are available and to easily select the patches, which the customer wants to install. Again, for customers without a valid support contract,  only security and driver patches are available.  For customers with a valid support contract,  all patches are available.

Sun Connection - Hosted Edition is the internet portal version of  updatemanager.  The customer can register all their servers and  can schedule and review the installation of patches from a central portal.  This is only available to customers who pay for it.

The above tools rely on the "PatchPro" analysis engine to recommend patches to customers.

PatchPro utilizes what are called "Realizations".  These are listed in the patchinfo file in the top directory of a patch.  This allows the patch developer to associate a patch with one or more "Realization Detectors", which determines whether or not it is appropriate to apply a patch to a particular customer environment.  For example, a Realization Detector might only recommend a particular patch if the target system utilizes a particular piece of hardware or software, or if a particular service is enabled.  This provides fine-grained control on patch recommendations.

The vast majority of Realizations simply associate a patch to packages installed on the target system, in the same way patchadd determines whether or not to apply a patch.  That is, if the package name, package version, and platform architecture in the pkginfo file(s) in the patch match at least one package name, package version, and platform architecture on the target system, the patch can be applied, else not.

Errors in writing Realization Detectors cause patch automation tools which utilize the PatchPro analysis engine to occasionally recommend inappropriate patches.  This has impacted the reliability of PatchPro based tools.

Work is underway to write a generic realization detector to match patches to packages.  This will save patch creators from writing their own realization detectors for the common case, simplifying the process and reducing error opportunity.  Patch creators will still be able to write specific Realization Detectors where necessary.

See Instructions for Getting Started with Sun Connection's Update Manager and Sun Hosted solutions and Patch Manager 2.0 FAQs for further information.

TLP

TLP stands for Traffic Light Patching, and is another tool which was developed by Sun Service folk for Sun Service folk to address the need for Patch Automation.

TLP is not directly available to customers.  It's used by Sun Service personnel to determine the appropriate patches to be installed on a customer's system, including things like firmware patches.

TLP has a modular design.   It utilizes the concept of a "baseline" of patches chosen by the user, from the Recommended Patch Set, to the EIS patch set, to a user defined set of patches.  TLP allows a number of different patch analysis engines to be used to determine which patches from the "baseline" to apply to a particular target system.

TLP is popular with customers who use it, as it's reliable and works well. 

TLP was End-Of-Lifed (EOL'd) in September of 2006 and reached End-Of-Service-Life (EOSL) in December 2007.  However, a number of customers have been given extensions on TLP support for transition purposes. 

Sun Services Patch Recommendations 

Most European countries provide a service where customers can submit Explorer logs and the local Sun office provides back a patch bundle.  These services may use SRAS and TLP in the background.

Please contact your local Sun Services office for further details.

I believe the plan will be to consolidate these services into a consistent official worldwide service.

About

This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today