Patch Install Downtime Requirements

As mentioned previously, Solaris Live Upgrade can help minimize the downtime associated with patching, by enabling users to patch an inactive boot environment.  When all modifications have been made to the inactive boot environment, a single reboot is required to activate it.  Also, most Special Install Instructions specified in patch READMEs can be ignored when patching an inactive boot environment.

When patching a live boot environment, certain patches require system downtime in order to complete their installation.

Such requirements will be specified in the patch README file and is also specified by the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s).

As mentioned previously, the problem with patching a live boot environment (without Deferred Activation Patching) is that some objects delivered in a patch, such as shared objects, may be invoked immediately while other objects, such as genunix, will only be activated when the system is rebooted.  Also, processes already running in memory may be using the old versions of objects while processes started after the patch(es) were applied will be using the new versions of the same objects (when Deferred Activation Patching isn't specified).  In some cases this is harmless.  In other cases, the system may be in a potentially inconsistent state until it is rebooted.

Some patches require that they be installed in Single User Mode (run level S) when applied to a live boot environment.  This is to ensure that the system is in a quiesced state to avoid the above potential problems.  This will be specified in the patch README file.  Also, the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s) will contain the entry singleuser_required.

Some patches require that the system be rebooted at some convenient point after the patch is applied in order to activate its contents. Either a normal reboot or a reconfigure reboot may be required. This will be specified in the patch README file.  Also, the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s) will contain the entry rebootafter or reconfigafter.  For such patches, the system remains in a consistent state until the reboot takes place.  The reboot is simply to allow the changes supplied by the patch to be activated. If a reconfigure reboot is required, application of the patch will cause the creation of a /.reconfigure file which will result in a reconfigure reboot when the system is rebooted.

Some patches require that the system be rebooted immediately after the patch is applied to a live boot environment.  For such patches, the system is in a potentially inconsistent state until the reboot takes place.  Either a normal reboot or a reconfigure reboot may be required. This will be specified in the patch README file.  Also, the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s) will contain the entry rebootimmediate or reconfigimmediate.  If a reconfigure reboot is required, application of the patch will cause the creation of a /.reconfigure file which will result in a reconfigure reboot when the system is rebooted.  Normally, it is OK to apply further patches to the live boot environment before initiating the reboot.  However, normal operations must not be resumed until after the reboot is performed.  On the rare occasion where the reboot must be instigated before any further patches are applied, such as is the case with Solaris 10 Kernel patch 118833-36 (SPARC) / 118855-36 (x86), such patches will typically contain code to prevent further patches from being applied as an added safety mechanism.

Comments:

Gerry,

Have a question about LU'ing patches into ABE. I love using LU for patching and am starting to get traction with customers in using it.

There are some patches that even when being installed into an ABE won't install as they are flagged for "single-user" mode only or some similar reason. Very frustrating as I would think that LU should be telling the patch installation processing that single-user mode isn't required as the installation is going into an offline root environment. In this case I'm attempting to use something like:

smpatch update -b MYLUVLYLUENV

When this happens, I'll do something like:

smpatch download -i <patch-id> (assuming the patch hasn't already been downloaded)
smpatch add -i <patch-id> -b MYLUVLYLUENV

Here the "smpatch add" command displays a message about going into patch add mode or something like that ... (can't recall the exact message). In this case, the patch which would not install via "smpatch update -b MYLUVLYLUENV" installs just fine. But this is annoying as now I have to be back through my patch install log and see which patches didn't install that are dependent on the patch I just installed using "smpatch add .....".

Wondering if there is something I'm missing. Your blogs are making me think I am.

Posted by Sean O'Neill on January 13, 2008 at 03:56 PM GMT #

Hi Sean!

I'm not very familiar with 'smpatch' as I and my team primarily use the basic Solaris patch utilities, 'patchadd' and 'patchrm'.

I've asked around, and here's what I've been told:

From man smpatch on patched u3 system:

"smpatch supports the Live Upgrade feature of the Solaris operating system (see live_upgrade(5)). Through the add, remove, and update subcommands, described below, smpatch enables you to perform operations on a boot environment (BE). A BE is an operating system image, consisting of a particular set of operating system and application software packages."

So basically 'smpatch -b' calls 'luupgrade -t -n', etc., to patch the ABE.

Now as to whether smpatch will install patches to an ABE if they are specify Single User Mode installation, etc., is unclear to me.

If 'smpatch' has the following:

patchpro.install.types - rebootafter:reconfigafter:standard

...then I suspect that even for -b ABE it will honor the patchpro.install.types.

I would suggest a Bug/RFE be filed against this "feature".

I also received the following suggestion (WHICH IS UNTESTED):

The workaround would be to set patchpro.install.types to a very liberal list. It's not documented, but you can do this on a per-command basis with "-C patchpro.install.types=...".

Personally, I would suggest considering to use Live Upgrade outside of smpatch.

If I get any more information from the 'smpatch' folks, I'll pass it on.

Posted by Gerry Haskins on January 17, 2008 at 11:20 AM GMT #

Hi Sean!

I've filed CR 6652289 "smpatch incorrectly considers patchpro.install.types when invoking Live Upgrade" on this issue.

It would be very useful if you or another external contract customer escalated this CR to get it fixed. (External customer escalations hold more weight than internal escalations, so it's always a good idea to escalate CRs which are hurting you via the support channels.)

My contact in 'smpatch' says: "be warned, smpatch is in the process of being transferred to sustaining for all continued support so its unlikely that this will be fixed without an escalation to back it up. But at least the workaround is easy."

Posted by Gerry Haskins on January 18, 2008 at 03:50 AM GMT #

How can this blog say "Normally, it is OK to apply further patches to the live boot environment before initiating the reboot."? What, then, does "rebootimmediate" mean? Is there an algorithm that a System Administrator can apply to a patch when attempting to discern if "rebootimmediate" will allow further patches?

What does "Normally" mean? Do you have any examples of an abnormal situation?

Posted by Michael Schwager on May 13, 2008 at 10:51 AM IST #

Hi Michael!

Please see the subsequent sentence of the blog entry for an example of an abnormal situation:

"On the rare occasion where the reboot must be instigated before any further patches are applied, such as is the case with Solaris 10 Kernel patch 118833-36 (SPARC) / 118855-36 (x86), such patches will typically contain code to prevent further patches from being applied as an added safety mechanism."

For x86 only, there are a couple of additional cases:

libc patch, 121208, contains a prepatch script to ensure a reboot takes place between the application of Kernel patch 118844-14 and 121208. 121208 is obsoleted by 118855-36, which inherited the prepatch script, so a reboot is required between the application of 118844-14 and 118855-36.

The only other patch which may, on certain hardware, require a reboot before applying further patches is biosdev patch 117435-02. On certain hardware, a reboot is required after applying biosdev patch 117435-02, before applying Kernel patch 118844-27 which transitions the system to grub.

Login to SunSolve and see http://sunsolve.sun.com/search/document.do?assetkey=1-66-200882-1 for further information on the above x86 issues..

When patching the live boot partition, "rebootimmediate" means that the system is in an inconsistent state until the reboot takes place. But the footprint of "patchadd" is quite small, especially in a non-Zones environment, so unless the area of inconsistency is in code which patchadd calls, it's OK to continue to add further patches. Where the area of inconsistency is known to be in code which patchadd calls, code is added to the patch script which introduced the inconsistency to prevent further patching until the reboot occurs.

As mentioned elsewhere in this blog, such restrictions only apply when patching the live boot environment. If you use Live Upgrade or "patchadd -R" to patch an inactive boot environment, such restrictions don't apply, so "rebootimmediate" has no meaning when patching an inactive boot environment.

Posted by Gerry Haskins on May 14, 2008 at 04:51 AM IST #

Thanks Gerry. I realize that I was mixing Deferred Activation Patching in my mind when I was reading the latter half of the blog entry.

So now I'm wondering, how do my questions- and your response- figure into a Deferred Activation patch? Can we confidently patch (without immediately rebooting) after a DAP, and only reboot after all our patches are in? Or does your caveat "unless the area of inconsistency is in code which patchadd calls..." still apply?

We have scripting that installs sets of patches, in series. Therefore, we need to programmatically determine ahead of time if there will be an inconsistency that will prevent further patching until the reboot occurs. Thus we could conduct the reboot at the appropriate point. Is this possible?

The bottom line is, we need something solid and definitive that we can stand on, an algorithm for determining without a doubt, in the moment our script is running, if we have a patch that will require an immediate reboot before further patching or if we can continue patching until we run out of patches to install...

Right now we reboot after every patch that mentions "Reboot/Reconfig Immediately" in its README file, but I happen to be generating a patchset at this moment which contains patches 118918-24, 122660-10, 125503-02, 120011-14, an 127127-11. Potentially 5 reboots. Eeep.

P.S. We expect to go to Live Upgrade later this year.

Thanks again.

Posted by Michael Schwager on May 14, 2008 at 08:59 AM IST #

Michael wrote:

"Can we confidently patch (without immediately rebooting) after a DAP, and only reboot after all our patches are in?"

Yes, Deferred Activation Patching (DAP) resolves the problem of patchadd getting into an inconsistency state during patch application.

The only caveat is whether there are any bugs lurking in the DAP implementation, but I'm pretty confident that it's working correctly. There was one bug, CR 6626977, which is fixed in patch utility patch 119254-49 (SPARC) / 119255-49 (x86). This is why I always recommend making sure that the latest revision of the patch utilities patch is always installed before installing any other patches.

For years, my team, Patch System Test, have applied all patches to the live boot environment of all our test systems every week and only done a single reboot at the end of the application. Apart for the couple of patches mentioned in the blog article above, this has never caused us a problem. Of course, the systems are in a quiet state when we apply the patches.

So you should be able to do the same in your script. If your systems already have 118833-36 (SPARC) and 118855-36 (x86) applied, the process should be relatively simple. Just install the latest patch utilities patch first, then your other patches, then reboot once at the end.

The only other serious caveat I know of regarding patching currently is that I strongly recommend users avoid using "patchadd -M" in a Zones environment due to some bugs and design flaws in the current implementation. "patchadd -M" currently applies all the patches to the global zone before applying any to the non-global zones. This is poor design, as if a patch in the list exits on the global zone with an error, it may leave the global zone significantly out of sync with the non global zones which isn't good. The solution is obvious. Apply each patch to all Zones before proceeding to the next patch. This issue is currently being worked on in conjunction with the project to significantly enhance Zones patching performance. In the meantime, it's recommended that customers in a Zones environment use "patchadd -a -M" (the "a" means it's a dryrun, so no action will be taken) to ascertain a valid install order for an arbitrary list of patches and then feed this list into a "for" loop to "patchadd" (without -M) to apply them. For example:

for i in 'cat OrderedListOfPatchesToApply'
do
patchadd $i
done

See my colleague, Enda O'Connor's, Patch Management Best Practices document, http://www.sun.com/bigadmin/features/articles/patch_management.jsp , on BigAdmin for further information.

Posted by Gerry Haskins on May 14, 2008 at 10:07 AM IST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today