Wednesday Oct 12, 2011

Live Upgrade document updated and simplified

I forgot to let you know, but a couple of months ago, my colleagues, Don O'Malley and Ed Clark updated the Oracle Solaris Live Upgrade (LU) document describing the pre-requisites for Live Upgrade.

The original document was pretty convoluted and required several cups of strong coffee to parse.  The updated version is a little easier to understand, even without caffeine.

Thanks also to Beth Barrett, Rick Ramsey, and Jon Bowman who helped make this happen.

Friday Oct 15, 2010

Using Live Upgrade in complex environments

One of my senior engineers, Enda O'Connor, has written a document on Patching Solaris using Advanced Live Upgrade Strategies for Zones and Clusters which I hope you will find useful.

Enjoy!

Thursday Sep 09, 2010

Solaris 10 9/10 (Update 9) released

Solaris 10 9/10 (Update 9) has been released.  See here for information and here for the download (remember to accept the license agreement at the top).  There's also a podcast and a dedicated Solaris blog.

A number of technical articles have been released, including George Wilson's video overview of ZFS enhancements in Solaris 10 9/10.

As with all Solaris Updates, Solaris 10 9/10 contains all available bug fixes which were available at the time that its contents were finalized, pre-applied into the Solaris Update image. 

It also contains a significant number of feature enhancements as described in the above links.

The corresponding Solaris Update Patch Bundle is currently in test and I expect that it should be released in a similar timeframe to previous Updates.  See http://blogs.sun.com/patch/entry/solaris_10_10_08_patch  for information on Solaris Update Patch Bundles.

All standard patches in Update 9 have already been released to SunSolve and My Oracle Support (MOS).  I've updated the Solaris 10 Kernel PatchID Sequence entry below with the Kernel PatchIDs for Solaris 10 9/10 (Update 9).

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  See the SPARC and x86 patch lists. 

<pet peeve>

Please note it is incorrect to refer to Kernel Patch 142909-17 (SPARC) / 142910-17 (x86) as the "Update 9 Kernel patch".  It is the latest Kernel Patch included in Update 9, but this Kernel patch can equally be applied to all previous Solaris 10 releases.   Solaris Updates are built from patches (and a few new packages), patches are not built from Solaris Updates.

</pet peeve>

Tuesday Dec 15, 2009

Veritas now supports Solaris Live Upgrade

As I've mentioned in a previous posting, Veritas now supports Solaris Live Upgrade.   Please see here for details.

Tuesday Nov 17, 2009

New FREE online Patch Training Courses

I'm delighted to announce the availability of 10 new free online patch training modules.

This is the result of a lot of work from those nice people in Sun Learning Services, the Install Revenue Product Engineering (RPE, a.k.a. Sustaining) team, and my own folk.

The modules concentrate on using Live Upgrade for patching, as well as providing background on Deferred Activation Patching, Kernel patches, and other useful information.

You can access the modules as follows:

I think even experienced Sys Admins will find the modules useful in clarifying patching best practices and providing context and background information on the evolution of patching technology and best practices in Solaris 10.

If you don't like the online course format, or if you want a reference document to refer back to after taking the course, please see the attached .pdf.

Enjoy!

Best Wishes,

Gerry Haskins,
Director, Software Patch Services

Tuesday Mar 10, 2009

Improvements to Patch Cluster pages on SunSolve

My team and I have been working with the SunSolve team on improvements to the Patch Cluster pages on SunSolve.  These improvements went live on April 20, 2009.

The old "Recommended Patch Clusters" and "Recommended and Security Patches" pages have been combined into a single Patch Cluster & Patch Bundle Downloads page.

A Notice Board section at the top of the page will be used to alert customers to current issues.

Click on the cluster headings to see a brief description of the purpose of the cluster, with links to view the cluster README as well as a download link.   The date the cluster was last updated and the size of the cluster are also shown.

No change has been made to the underlying cluster file names, so scripts using 'wget' to access the patch clusters should be unaffected.

This is part of an ongoing effort to improve our patch presentation to customers.

As before, customers need a valid support contract in order to be able to access patch clusters. 

If you are not registered in Member Support Center, simply log into SunSolve and associated one or more support contracts with your Sun Online Account using the "Change Contract" option in the top right hand menu.

If you are registered in Member Support Center, your contracts will be automatically associated with your account (and the "Change Contract" option will not be shown when you log into SunSolve).

Wednesday Feb 11, 2009

Now possible to upgrade directly from Solaris 8 SPARC to latest Solaris 10 release

Thanks to my colleague Enda O'Connor, who has made p7zip available for Solaris 8 SPARC, it's now possible to upgrade directly from Solaris 8 SPARC to the latest Solaris 10 Update releases such as Solaris 10 5/08 and Solaris 10 10/08. 

See document 1019995.1 on My Oracle Support (MOS), http://support.oracle.com

Previously, due to the lack of p7zip on Solaris 8, customers needed to perform an interim upgrade to Solaris 9 or an earlier Solaris 10 release before upgrading to the latest Solaris 10 release.

Friday Jul 04, 2008

Solaris 10 Live Upgrade Zones Starter Patch Bundle

The Solaris 10 Live Upgrade Zones Starter Patch Bundle has been released.  It is designed to make it simpler for customers running on systems below Solaris 10 5/08 (Update 5) to apply the pre-requisite patch level needed to be able to utilize basic Live Upgrade functionality in a Zones environment.  These patches need to be applied to the live boot environment to enable Live Upgrade to work correctly in a Zones environment.

Aside: Customers with systems running Solaris 10 5/08 (Update 5) or later already have all the  pre-requisite patches pre-installed on the live boot environment and hence do not need to apply this patch bundle.

After this, Live Upgrade itself can be used to create an inactive boot environment and apply any additional patches referenced in SunSolve document 206844 'Solaris[TM] Live Upgrade Software: Minimum Patch Requirements' (formerly Infodoc 72099) to provide advanced Live Upgrade functionality such as support for ZFS Root. The document is available from: http://sunsolve.sun.com/search/document.do?assetkey=1-61-206844-1

The Solaris 10 Live Upgrade Patch Bundle is available from the normal patch cluster download center on SunSolve.  To download the patch bundle, login to SunSolve, http://sunsolve.sun.com , click on the Patches and Updates link, click on Recommended Patch Clusters, and scroll down the window under the heading "Recommended Solaris Patch Clusters, J2SE and Java Enterprise System Clusters" to find the "Solaris 10 SPARC Live Upgrade Zones Starter Patch Bundle" or "Solaris 10 x86 Live Upgrade Zones Starter Patch Bundle".  As always, you need a valid support contract to access patch clusters.  See previous postings for further information on support contracts.

Thursday Jun 19, 2008

More info on patching using Live Upgrade

My colleague, Enda O'Connor, has written another useful article on Big Admin about patching using Live Upgrade, restrictions, and how-to use Live Upgrade to upgrade/patch from Solaris 8 or Solaris 9 to Solaris 10.  See Doc ID 1019995.1 on MOS.

BTW: Searching "Live Upgrade" under 'Search Knowledge Base' on MOS brings up some other good LU articles too.

Tuesday Apr 29, 2008

Patch Management Best Practices

Please see the Patch Management Best Practices guide which my colleague, Enda O'Connor, has published on the BigAdmin Patching Hub.  I hope you'll find it useful.

Enda is a senior engineer in Patch System Test and he is far more technical than I am.

Enda has more practical experience of patching Solaris 10 Zones environments than anyone else in Sun.

Enjoy!

Thursday Jan 10, 2008

Patching Strategies

As mentioned in my initial posting, there isn't a "one size fits all" patching strategy for all customers to use in all circumstances.

Perhaps the most common question which customers ask is, "What patches should I apply to my system ?"

The answer, unfortunately, is "It depends."

Many factors determine what patching strategy is appropriate for a particular system.  These may include:

  • Risk profile of the customer.  For example, Financial institutions tend to be very risk adverse.  Their change control processes can be onerous.
  • Criticality of the system.  Is it Life Critical, Mission Critical, Business Critical, or relatively expendable ?
  • Risk profile of the system.  For example, is it behind a firewall, is it vulnerable to Denial Of Service attacks, etc. ?
  • Cost of planned downtime (for proactive patching and maintenance) versus the cost of unplanned downtime (for reactive break-and-fix patching and maintenance).
  • Available Maintenance Windows
  • Upgrade strategy - Is the customer still on older versions such as Solaris 8 or Solaris 9 or is there a desire to leverage some of the cool new software features (e.g. Containers (Zones), ZFS, DTrace, etc.) or support for cool new hardware available in Solaris 10 ?
  • Desire to keep a relatively homogeneous Operating Environment across similar servers
  • etc.

While I can discuss some evolving thoughts on patching strategies here, please note that Sun Services offer comprehensive solutions tailored for the needs of specific customers.  The thoughts expressed here are not a substitute for careful analysis of the specific needs of individual customers.

Minimizing Risk

Risk minimization is a key consideration for many customers when deciding on a patching strategy.

Change implies risk.

There are industry studies which show that for every x number of bug fixes or lines of code changed, a new bug or regression is introduced.

One might logically conclude therefore, that the more change that is applied to a system, the more the risk of introducing a regression.  Hence, one might conclude that applying the minimum number of patches and hence the minimum amount of change would minimize risk.

But that's just one factor.

It's also important to consider the test coverage of the various change delivery options.  This includes test coverage by Sun as well as test coverage by the customer, channel partner, or other vendor.

Always install the latest patch utilities patch first

Always install the latest version of the Solaris patch utilities patch before installing any other patches.  This is important to ensure that you have all the latest fixes to the patch utilities.  The patch utility patches are always contained in the Solaris Recommended and SunAlert patch clusters and are always installed first, along with any patches which they themselves require.

The patch utilities patches are currently:

  Solaris 10 SPARC:    119254
  Solaris 10 x86:           119255
  Solaris 9 SPARC:      112951
  Solaris 9 x86:             114194
  Solaris 8 SPARC:      110380
  Solaris 8 x86:             110403

Depending on the OS version, several other patches may be required to avoid issues which can impact correct patch application.  Such patches are listed in the "Latest patch updates" section on the SunSolve home page.

Solaris Patch Management: Recommended Strategy 

The Solaris Patch Management: Recommended Strategy available from http://docs.sun.com and linked off the SunSolve "Patches and Updates" page is a good starting point. 

Perhaps surprisingly, it shows from an analysis of customer Explorer data that the more patches which are applied to a system, the more downtime that system will experience.  This is largely because, as discussed in the preceding posting, a number of patches require downtime in order to be installed on a live boot environment. 

However, in many cases the cost of unplanned downtime to fix issues is much, much higher than the cost of planned downtime to facilitate preventative patching to prevent issues from occurring in the first place. 

The trick is to know which patches are likely to prevent issues on a particular system.

Recommended and Sun Alert Patch Clusters 

When deciding what patches to apply to Solaris, the Recommended and the Sun Alert Patch Clusters, which are available from SunSolve to customers with a valid support contract, are a good starting point.  They provide:
  • The latest revision of patch utilities patch
  • Solaris patches which address Sun Alert issues - that is, patches which address Security, Data Corruption, or System Availability issues.
  • Any patch which is required by either of the above.

The main difference between the Recommended Patch Cluster and the newer Sun Alert Patch Cluster is that the Sun Alert Patch Cluster contains the lowest revision of patches which address Sun Alert issues while the Recommended Patch Cluster contains the latest available revision of such patches.  Both are good options.

Note, the Recommended and Sun Alert Clusters only contain patches for the Solaris OS.  They do not contain patches for middleware or application layer products such as Java ES, SunStudio, etc. 

Both the Recommended and the Sun Alert Patch Clusters come with an install_cluster script and a patch_order file listing the order in which the patches are to be installed.  See the Cluster README files linked off the "Patches and Updates" page on SunSolve for further information.   (On http://sunsolve.sun.com/show.do?target=patches/patch-access , "Solaris 10 x86" is the Solaris 10 x86 Recommended Cluster and "Solaris 10 x86 Sun Alert Patch Cluster" is self-explanatory.)

Applying the Solaris Recommended or Sun Alert patch cluster at each available maintenance window, plus any patches for fixes for bugs which you as a customer have filed yourself, is a good approach to proactive patching.

In between maintenance windows, monitor new Sun Alerts which are issued and determine whether your systems are vulnerable to the issue.  If the risk of the issue occurring is low or the consequences of the potential problem manageable, you may decide that it's OK to wait until the next maintenance window before applying the patch or taking whatever other action is recommended in the Sun Alert.  If the risk of the issue occurring is high and the potential problem severe, consider applying the patch or taking whatever other action is recommended in the Sun Alert as soon as possible.

Apart from these Solaris patch clusters, other patch clusters are available on http://sunsolve.sun.com/show.do?target=patches/patch-access for other products, such as J2SE and Java ES.

Installing or Upgrading to the latest Solaris Update Release

Each bi-weekly build of the next Solaris Marketing Release ( "Nevada" ) and the next Solaris 10 Update Release is intensely tested by a large number of test teams throughout Sun.  Each team has a particular focus, from functional testing of new features, regression testing of pre-existing features, performance improvement testing (each release should be faster than the last), new hardware testing, hardware regression testing, Desktop, Globalization, Accessibility, SunCluster, Java Enterprise System, patch testing, etc.

Due to the intensive testing of Update releases, installing or upgrading a system to the latest available Update Release should be seriously considered by customers wishing to minimize risk. 

While each Update release contains significant amount of code change, it has been intensely tested as a unit.  As previously mentioned, each Update contains all the available bug fixes at the time it was built.  Therefore, pre-existing functionality should be more stable and more performant in each successive Update.  The latest available Update should therefore provide a good stable baseline for customers. 

For example, Solaris 10 8/07 (Update 4) not only introduces cool new software features and support for cool new hardware, it also contains many fixes and enhancements to pre-existing functionality such as Containers (Zones), such as the ability to Upgrade Zones, significantly improving the maintainability of Zones.

The complexities of patching the live boot environment of a pre-Update 4 Zones systems can be avoided by upgrading to Update 4 instead.

"Dim Sum" Patching

As previously mentioned, new bug fixes as well as features "soak" in the next Marketing Release of Solaris under development (currently "Nevada";) to shake out any bugs in the code, before the code changes are allowed to be put back into an older release of Solaris for inclusion in a patch and, if the patch is for Solaris 10, included in the next Update Release.

 

In this way, patches leverage the intensive testing done on the Marketing and Update releases.  Indeed, Solaris 10 patches leverage this intensive testing twice: once in the Marketing Release "soak" test, and again when the bug fix is included in builds of the next Solaris 10 Update.

The bug fix in each patch is verified by the responsible engineers in-house using a test case and/or by providing the T-Patch (Test Patch) to escalating customers to verify that it fixes the issue.  In addition, the patch will be tested by the Patch System Test group and potentially by other test teams such as the Desktop QA or Hardware QA teams.  Only when all verification and testing has been successfully completed will the patch be released to SunSolve.  See an Overview of Patch Testing on SunSolve for further details.

Patch System Test test the patch both individually with any required patches, and cumulatively along with all other available patches. Testing the patch on its own helps ensure that all patch requirements have been correctly specified.  Testing the patch in combination with all other available patches helps ensure that there are no bad interactions between patches.  Testing these boundary conditions gives confidence that other patch combinations should work.

Extreme lengths are also taken in the code development and putback approval processes to ensure that patch requirements are correctly specified and that the change is compatible, well designed, and will not introduce regressions.

Nevertheless, if a customer takes individual patches which they feel are appropriate to their system, outside of a defined patch cluster such as the Recommended or Sun Alert Patch Cluster, they may end up running a code combination which has never been tested as a unit. 

The various checks and balances in the patch process should be fully sufficient to ensure this code combination is stable and functional.  But from a risk management perspective, running code which may not have been tested as a unit remains a finite risk.

This is what Bart Smaalders refers to as "Dim Sum" patching.

Most customers have practiced "Dim Sum" patching for years and, in general, it works very well.  Even with the massive amount of code changes included in Solaris 10 Update Releases compared to Solaris 8 or Solaris 9 Update Releases, there have been very few issues as a result of "Dim Sum" patching.

But using the latest available Solaris Update release or Recommended or SunAlert Cluster or EIS CD (via Sun Connection 1.1.1 Satellite or xVM Ops Center 1.0) or other set of patches as a baseline has the advantage that that baseline has been tested to varying degrees as a unit, with Solaris Update releases the most intensely tested of those options.

This is a case where taking more change by installing or upgrading to the latest Solaris Update may actually imply less risk.

Customer Testing

Testing by Sun is just one factor.  Testing by the customer, channel partner, or other vendor also plays a significant part in managing risk.

If the customer has a test set up which exactly mirrors their live production environment, with tests which mimic normal and peak loads, then their confidence level in any patching strategy they choose can be very high.

The less sophisticated the customer test environment, the more the customer is relying on Sun's Development and QA processes to  catch all the issues.

Patch Quality 

The good news is that the Sun's Development processes are meticulous and mature and the QA processes sophisticated and effective. 

That doesn't stop all issues escaping to customers but, in general, the quality of patches is very high. 

For example, out of approximately 4,500 patches released by Sun in 2007, only 70 have been subsequently withdrawn due to serious issues with them.

Patch Maturity

A number of customers like to wait a set period of time after a patch has been released before considering installing it to see if Sun or other customers find issues with it.

This is a reasonable strategy.

Some customers wait until 6 weeks after a patch has been released before applying it.  Data analysis shows that there is no particular significance in this time period.

Data analysis shows that on the rare occasion where patches which are withdrawn from SunSolve after their release due to serious issues with them, the length of time between when a patch was released and when it was withdrawn shows no clear period of time before which a patch can be considered unsafe or after which a patch can be considered safe.  Some patches are withdrawn within the first 3 weeks of release.  Others not until 18 months or 2 years later. 

However, it is reasonable to assume that patches which aren't withdrawn for a significant period of time only have a serious issue is a rare configuration which most customers won't encounter.

Friday Jan 04, 2008

Using Solaris Live Upgrade for patching

Solaris Live Upgrade can be used to patch a system as well as to upgrade from an earlier Marketing or Update release of Solaris.

Live Upgrade avoids many of the problems encountered when patching a live Solaris 10 boot environment as Live Upgrade modifies an inactive boot environment, rather than the live boot environment.  This means that the live boot environment remains in a consistent state throughout the modifications.   Once the inactive boot environment has been patched to the appropriate level, the system is rebooted to activate the modified boot environment.   Live Upgrade has the additional advantage that it provides a ready-made fallback option - simply reboot back into the old boot environment if you encounter problems with the modified boot environment.

The downside of Live Upgrade is that it requires a number of prerequisite patches to be installed on the live boot environment before it can be used.  See Infodoc 72099 available from SunSolve for a list of prerequisite patches.  This is OK for pre-Solaris 10 systems and Solaris 10 systems without Containers (Zones).

However, for Solaris 10 systems with non-global Zones, this list of pre-requisite patches includes complex Kernel patches such as 118833-36 (SPARC) and 118855-36 (x86).  This somewhat limits the usefulness of Live Upgrade as a Solaris 10 patching aid for systems with non-global Zones running a version of Solaris older than Solaris 10 11/06 (Update 3) or otherwise at a Kernel patch level of less than 118833-36 (SPARC) / 118855-36 (x86).

But for other environments, Solaris Live Upgrade provides a good method to patch systems.

For further information, please see the following articles on the How To page of the Big Admin Patching Hub:

How to Use Solaris Live Upgrade to Install Patches On Your System

How to Patch a System With RAID-1 Volumes by Using Solaris Live Upgrade

How to Patch and Upgrade a System by Using Solaris Live Upgrade When Non-global Zones are installed

About

This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today