Tuesday Nov 17, 2009

New FREE online Patch Training Courses

I'm delighted to announce the availability of 10 new free online patch training modules.

This is the result of a lot of work from those nice people in Sun Learning Services, the Install Revenue Product Engineering (RPE, a.k.a. Sustaining) team, and my own folk.

The modules concentrate on using Live Upgrade for patching, as well as providing background on Deferred Activation Patching, Kernel patches, and other useful information.

You can access the modules as follows:

I think even experienced Sys Admins will find the modules useful in clarifying patching best practices and providing context and background information on the evolution of patching technology and best practices in Solaris 10.

If you don't like the online course format, or if you want a reference document to refer back to after taking the course, please see the attached .pdf.


Best Wishes,

Gerry Haskins,
Director, Software Patch Services

Wednesday Dec 17, 2008

Definitive interpretation of the "rebootimmediate" and "reconfigimmediate" patch flags

The following is now available as Infodoc 249046:

What follows is an open letter to customers in response to customer confusion over how to handle the "rebootimmediate" and "reconfigimmediate" flags specified in some patches.

Despite the READMEs of patch clusters which contain such patches clearly stating that during a patching session, a reboot is only required in exceptional and documented circumstances, it has come to my attention that some customers are initiating reboots after applying every single patch in a patch set which specifies such flags.  Not surprisingly, such customers are concerned at the length of time this takes.

Open Letter with definitive interpretation of the "rebootimmediate" and "reconfigimmediate" patch flags

To whom it may concern,

Summary: When patching a live boot environment, it is usually OK to apply any number of patches before performing a single reboot at the end, even if multiple patches specify "rebootimmediate" or "reconfigimmediate".  On the rare occasion when it is found that this is not possible, specifically for 118833-36 (SPARC) and 118855-36 (x86) and 118844-14+ (x86), code will typically be inserted into the relevant patches to prevent the application of further patches which could cause problems.  Use of Live Upgrade to patch an inactive boot environment is recommended as it avoids the need for interim reboots for even these atypical patches.  Details below.

The "reboot" metadata flags which may be contained in the patch 'pkginfo' file(s) have the following meaning:

rebootafter - a reboot is required to activate some of the content delivered in the patch, but the system remains in a consistent state until the reboot is performed.

reconfigafter - a reconfiguration reboot is required to activate some of the content in the patch, but the system remains in a consistent state until the reconfiguration reboot is performed.

rebootimmediate - the system is in a potentially inconsistent state until the system is rebooted.  The objects applied in the patch are potentially inconsistent with processes running in memory.  Normal production must not be resumed until a reboot takes place to bring the system back into a fully consistent state.  However, since the footprint of the patch utilities is relatively small, it is normally OK to continue to apply further patches before initiating the reboot.   In cases where this is not OK, the patch in question will typically contain additional code to prevent further patches from being applied until the reboot takes place\*.  Since the system is in a potentially inconsistent state, it's advisable to avoid running any additional processes until the reboot takes place.  If patch automation tools are being used to apply "rebootimmediate" or "reconfigimmediate" patches, it's up to the automation tools' QA to ensure that their additional code footprint does not hit the potential inconsistent system state when applying such patches.

reconfigimmediate - exactly the same as rebootimmediate, except a reconfiguration reboot is required.

\*This is the case with Kernel patch 118833-36 (SPARC) / 118855-36 (x86), whose patch scripts replace 'patchadd' with a no-op telling the user to reboot the system.  The only other known reboot required before further patching can be done is specific to x86, and only if the system is running at a Kernel patch level below 118844-14.  A later revision of 118844, e.g. 118844-20, needs to be applied and the system rebooted to ensure the Kernel running in memory is compatible with library changes supplied in the libc patch 121208-02.  The prepatch script in 121208-02 and -03, and 118855-xx which obsoletes it, contains code to ensure 118844-14 or later is installed and active on the system.  (BTW, 118844-14 wasn't released. 118844-20 is recommended to fulfill the libc compatibility requirement.)

UPDATE, Jan 20, 2009: Murphy's Law strikes again!.  There's currently an issue, CR 6704883, with the "Sun Fibre Channel Device Drivers" patches 125184-05, -06, -07, and -08 (SPARC) and 125185-05, -06, -07, and -08 (x86) as described in Sun Alert 238630.  The fix for this issue is in rev-09 of the patches which is currently available as a T-Patch and will be released shortly.  Rev-09 of the patches uses modloading in its prepatch script to avoid the issue.  In the meantime, a workaround is to apply the affected patches last, immediately prior to rebooting the system.  The patches in the Solaris 10 10/08 patch bundle were specifically ordered to avoid this issue.  Where such issues are found, SunAlerts are published and the issue fixed.

Remember, patches can be downloaded and installed individually.  Therefore, each patch which requires a reboot must specify the reboot requirements.  But if patches are installed collectively in the same patching session, for example, as part of a patch cluster, then the install instructions contained in the cluster README file take precedence - e.g. that reboots are only required \*during\* patching sessions for the specific cases mentioned above.

Since the above patches were created, a significant enhancement has been made to the Solaris patch utilities called Deferred Activation Patching.  This enhancement is not retrospective, so the above historical problematic patches remain.

Deferred Activation Patching

The problem with the above atypical patches is that the new code they deliver may be invoked by the original patchadd code and the utilities it calls \*during\* patch installation.  A patch may patch many packages.  The packages are applied in alphabetic order.  In a Zones environment, the patch is applied to the global zone first, then to each non-global zone.

In the case of 118833-36 (SPARC) / 118855-36 (x86), the new versions of the libdevinfo.so.1 and libsec.so.1 libraries delivered in the patch could be invoked by patchadd and are potentially incompatible with the processes running in memory.

The solution devised in the patch scripts contained in 118833-36 (SPARC) / 118855-36 (x86) is to overlay mount the old objects on top of the newly laid down objects using the loopback filesystem (lofs).  This ensures that the system remains in a consistent state \*during\* the patch process as the old library versions which are compatible with what's running in memory will be called.

To avoid the application of further patches, which patch the same objects as 118833-36 (SPARC) / 118855-36 (x86), from patching the overlay mounted objects instead of the patched objects, 118833-36 (SPARC) / 118855-36 (x86) replace 'patchadd' with a no-op telling the customer to reboot the system before applying any further patches.

During reboot, the loopback filesystem mounts are torn down exposing the patched objects.  Further patching can now continue as the system is in a fully consistent state.

This loopback filesystem mount solution is the basis of Deferred Activation Patching.  After patch 118833-36 (SPARC) / 118855-36 (x86) was released, the solution was perfected and moved to the patch utilities.  The few patches which require application using Deferred Activation Patching specify the SUNW_PATCH_SAFE_MODE=true flag in their pkginfo files.  The solution was enhanced so that any subsequent patch applied prior to a reboot of the system, which patches the same objects as a patch explicitly specifying Deferred Activation Patching, will itself be automatically applied in Deferred Activation Patching mode.   This is known as implicit Deferred Activation Patching and enables other patches to be applied on top of a patch applied using Deferred Activation Patching without the need for an intervening reboot.  When a patch specifying Deferred Activation Patching mode is applied to a system, the user will see lots of loopback filesystem mounts on the system until such time as the reboot takes place.  Upon reboot, the loopback filesystem mounts are torn down, exposing the newly patched objects.

Kernel patch 12001[12]-14 which is included in Solaris 10 8/07 (Update 4), Kernel patch 12712[78]-11 which is included in Solaris 10 5/08 (Update 5), and Kernel patch 13713[78]-09 which is included in Solaris 10 10/08 (Update 6), are currently the only patches which specify application in Deferred Activation Patching mode.  Future Kernel patch included in future Solaris 10 Update releases are the likely candidates requiring application using Deferred Activation Patching.

With the introduction of Deferred Activation Patching, it is highly unlikely that future patches will require an interim reboot before further patches can be applied.

The problems with the system getting into an inconsistent state \*during\* patching (which Deferred Activation Patching resolves) could only occur when patching a live boot environment as it's due to the interaction between newly patched objects which are incompatible with processes running in memory being invoked prior to the system being rebooted.

To avoid this and other issues, Sun strongly recommends the use of Live Upgrade to patch (or upgrade) an inactive boot environment, which dramatically reduces the risk and downtime associated with patching.  For example, even though Deferred Activation Patching resolves the inconsistency issue, patching a live boot environment takes time and the system is out of production.

Using Live Upgrade, the inactive boot environment is patched, potentially while the system is still in production.  Issues such as those described above with Kernel patch 118833-36 (SPARC) / 118855-36 (x86), and 118844-20 (x86) simply don't apply when patching an inactive boot environment as there is no interaction between the objects being patched and the processes running in memory, as all the calls patchadd makes will be to the objects on the live partition, not the patched objects on the inactive partition.  A single reboot is required to boot into the new boot environment.

Another advantage of Live Upgrade is that if a problem arises with the new boot environment for whatever reason, the user can simply reboot back into the old boot environment to enable production to resume and the issues with the now inactive boot environment can be resolved later.

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Friday Jan 04, 2008

Deferred Activation Patching

The generic issue to be solved is the need to be able to patch safely across arbitrary change and is logged as CR 6486471.

The problems of patching a live Solaris 10 boot environment became most acute with the Kernel patch associated with the Solaris 10 11/06 (Update 3) release.

The problem when patching a live boot environment, is that some of the changes delivered in a patch, such as shared objects, may be invoked by processes as soon as they are applied to the live boot environment.   Other objects, such as genunix, will only be activated when the system is rebooted.   Problems can occur where the scope of the change applied is very large compared to that which is running on the live boot environment.  In such cases, the new objects which are invoked, e.g. zoneadmd, may be incompatible with the old objects running in memory, e.g. genunix.  This can cause the live boot environment to get into an inconsistent state during patching.  The problem is most acute on a system running Zones, since the patch utilities need to invoke the zones utilities during patching to patch the non-global Zones.

A solution using loopback file system mounts (lofs) was incorporated into Kernel patch 118833-36 (SPARC) and 118855-36 (x86).  This overlays the patched objects with the original versions which were present on the system, ensuring that the system remains in a consistent state during the application of the patch.  When applied to a live boot environment, these patches require a reboot before any further operation, including the application of any further patches can be performed.  This forced reboot requirement is not good from a system availability standpoint.

The solution was subsequently refined and formalized as Deferred Activation Patching.

Loopback mounts are used to overlay the original objects on top of the patched objects.  This keeps the live boot environment in a consistent state during patching, irrespective of how much change is delivered in the patches.  Once all patches have been applied, the system is rebooted to activate the changes delivered in the patches.  At all points in time, the live boot environment remains in a consistent state.  Patches specifying application in Deferred Activation Patching mode set the SAFEMODE flag in the pkginfo file(s) in their packages.  The Solaris patch utilities will automatically recognize if any subsequent patches applied prior to the reboot touch the same objects as the patch applied in Deferred Activation Patching mode and will automatically install such patches in Deferred Activation Patching mode too.  Patches which don't intersect with a patch specifying Deferred Activation Patching mode will continue to be installed as normal.

Deferred Activation Patching was initially delivered in the Solaris 10 patch utility patch 119254-42 (SPARC) and 119255-42 (x86).  It is recommended that customers use patch utility patch 119254-50 (SPARC) and 119255-50 (x86) or later revision as these address some bugs in Deferred Activation Patching.

The Kernel patch associated with the Solaris 10 8/07 (Update 4) release, 120011-14 (SPARC) and 120012-14 (x86), are the first patches to utilitize Deferred Activation Patching.

Deferred Activation Patching will only be specified in patches which require it.  This is likely to be the Kernel patches associated with future Solaris 10 Update releases. 

See the Deferred Activation Patching article on the BigAdmin Patching Hub for further information.


This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer


« July 2016