Zones Parallel Patching feature now available!

The Zones Parallel Patching feature is now available in the latest Solaris 10 patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

This is available for use on all Solaris 10 systems. 

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

Prior to this feature, each non-global zone was patched sequentially, leading to unnecessarily long patching times for zones systems.  (Sequential patching remains the default behavior unless the config file is edited to enable Zones Parallel Patching.)

With this feature invoked, the global zone continues to be patched first, but then the non-global zones can be patched in parallel, leading to significant performance gains in patching operations on Zones systems.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

Here's the relevant note from the patch README file:

NOTE 10: 119255-66 is the first revision of the patch utilities to deliver "zones parallel patching".
          This new functionality allows multiple non-global zones to be patched in parallel by patchadd.
          Prior to revision 66, patchadd would patch all applicable non-global zones sequentially,
          that is one after another. With zones parallel patching, a sysadmin can now set the number
          of zones to patch in parallel in a new configuration file for patchadd called /etc/patch/pdo.conf.

         The two factors that affect the number of non-global zones that can be patched in parallel are
         1. Number of on-line CPUs
         2. The value of num_proc in /etc/patch/pdo.conf

          If the value of num_proc is less than or equal to 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in
          parallel to num_proc. If the value of num_proc is greater than 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in parallel
          to 1.5 times the number of on line CPUs. Note that patchadd will patch all applicable non-global
          zones on a system, the above description outlines only how patchaadd determines the
          maximum number of job slots to be used during parallel patching of non-global zones.

          An example of this in operation would be where:
          num_proc=8
          and number of on line CPU's is 4

          In this case the maximum setting for num_proc would be 6, that is the maximum number
          of zones that could be patched in parallel is 6.  If there are more than this number of non-global zones on the
          system, the first 6 will be patched in parallel, then the remaining non-global zones will be patched
          as processes finish patching the first 6 non-global zones.   Only one patch process will be used for each
          non-global zone, so if there are less than 6 non-global zones on the system, then only the number of processes
          equal to the number of non-global zones will be initiated.

          Please see comments in /etc/patch/pdo.conf for more details on setting num_proc.

I would like to thank Ed Clark and Enda O'Connor from my own team for all their work in developing and testing Zones Parallel Patching.

I would also like to thank Jon Bowman, Arindam Sarkar, and the rest of the RPE (Sustaining) Install team for all their work in getting this feature integrated into the patch utilities and delivered to production.

I would also like to thank our selected key customers who kindly Beta tested the feature for us.

I believe this feature is an important milestone in improving our customers' patching experience in a Zones environment as it addresses a long standing customer complaint on Zones patching performance.

Enjoy!

Comments:

This rocks -- thank you very much to everyone who made this happen. =-)

Even a modest improvement will be a boon to our patch cycle. We can also apply more patches per run. Good stuff!

I'll be interested to see how the advantage multiplies with PSARC 2009/173. That's also independent, per-container.

Posted by Craig Bell on June 17, 2009 at 01:24 PM IST #

Hi, great, this is long awaited. But why is there
an /etc/patch/pdo.conf in addition to /etc/patch/patch.conf and /etc/patch/secret.conf. It would be great if there is only one configfile, e.g. /etc/default/patch right there in /etc/default. This would streamline with other defaults configfiles.
Rolf

Posted by Rolf M Dietze on June 18, 2009 at 10:45 AM IST #

Hi
the decision to put the file in /etc/patch was that is because other patching related utilities store config files there as well.

patch.conf and secret.conf are owned used and maintained by a seperate utility, ie smpatch. The code to maintain these are not in the core patching utilites, ie patchadd, and these utilites do not exist in all metaclusters either.
So we cannot use config files that are owned by other tools which can modify these files as they so want, the patch utilities do not have access to this code base ( smpatch ).
pdo.conf is a config file specifically for the core patch utilites, patchadd and patchrm.
Enda

Posted by Enda O'Connor on June 18, 2009 at 11:36 AM IST #

[Trackback] One of the key issues at Solaris 10 deployments with many zones was the speed of patching. With the patches 119254-66 for Solaris SPARC and 119255-66 for Solaris x86 you can patch zones in parallel and thus reduce the time needed for this task:While ...

Posted by c0t0d0s0.org on June 18, 2009 at 01:50 PM IST #

Thanks a lot for this functionality

Had a query will this work with zones upgrade on attach.

What we have been doing is detach the zones; jump the global to the latest release + recommended patch cluster and then upgrade on attach the zones on the global.

Will this patch help to upgrade on attach the zones in parallel.

Posted by kocp on June 18, 2009 at 08:27 PM IST #

Hi Craig!

Glad you like it!

PSARC 2009/173, a.k.a. Turbo Packaging, will dramatically increase the speed of installation, upgrade, and zones creation (by ca. x1.5).

However, it has little impact on patching - perhaps just 10% improvement.

But it's all goodness.

Best Wishes,

Gerry.

Posted by Gerry Haskins on June 19, 2009 at 03:40 AM IST #

Hi kocp!

You raise an interesting subject!

True patching using 'patchadd' and Zones "Update on Attach" are two different technologies.

The primary purpose of Zones "Update on Attach" is Zones migration from one server to another. It can certainly be used for patching but there are limitations you need to be aware of as outlined below.

The Zones Parallel Patching enhancement to 'patchadd' (which by the way works well in conjunction with higher level patch automation tools such as xVM OC) patches non-global zones in parallel. The global zone is still patches first.

"Update on Attach" is using Zones functionality rather than 'patchadd' functionality. When a non-global zone is attached to a system, it updates the non-global zone to the same software level as the global zone.

But the global zone still needs to be patched first, and since Zones Parallel Patching only speeds up non-global zone patching, the two enhancements don't overlap.

So no, the Zones Parallel Patching enhancement doesn't further speed up Zones "Update On Attach".

Zones "Update On Attach" is typically faster than Zones Parallel Patching (assuming a large number of patches are being applied).

However, Zones "Update On Attach" has certain limitations as I discussed in a previous blog posting - see the Zones Update On Attach section in http://blogs.sun.com/patch/date/20081204.

In particular, Zones "Update On Attach" only works for packages which are SUNW_PKG_ALLZONES=true - i.e. typically OS level packages, and not application packages.

So when to use Zones Parallel Patching in 'patchadd' and when to use Zones "Update On Attach" ?

Here's what my senior engineer, Enda O'Connor, says:

"The Zone Update On Attach doc should be live in a day or two
http://www.sun.com/bigadmin/features/articles/zone_attach_patch.jsp

I am just reviewing the link above for any issues introduced in publishing, then it will go live. Might help [customers] understand how the technology works, applying a cluster via patching and via zones update on attach is not quite the same really."

Furthermore, Enda says:

"It really depends on the patches being applied, i.e. applying a firefox patch via Update On Attach would not work if you wanted it to apply to the global and all zones as well.

One has to understand how Update On Attach works and then apply that to the list of patches to see if it gets them to a desirable state.

There is no black or white answer here.
I'd recommend Zones Parallel Patching using 'patchadd' as it has a known outcome all the time, [whereas] Update On Attach makes it's own internal determination based on a number of things, that can vary from system to system ( e.g. inherited directories ).

But if time to patch is critical then if they do proper testing to validate things, and are happy then by all means [use] Update On Attach.

But using Update On Attach without:

1. Understanding how it determines what packages to update

2. Not inspecting the patches being applied.

...will most likely lead to grief at some point."

And my other senior engineer, Ed Clark, says:

"In terms of giving guidance on which technology to use, there are a number of considerations -- two of these considerations are:

1. Using Update On Attach to update sparse zones can require significantly more disk storage space [than] would be needed by applying patches with 'patchadd' (3-4 times as much space would not be uncommon i think), due [to] Update On Attach copying fully populated global zone 'undo' files into the [non-global] zones, as opposed to having patchadd build sparsely populated 'undo' files in the [non-global] zones.

2. If a customer is really concerned about the ability to back out patches reliably, then 'patchadd' is a lower risk option than Update On Attach -- 'patchrm' of a patch from a non-global zone that has a copy of the global zones 'undo' pkg data (as is the case after Update On Attach) may potentially have unexpected side effects." [although we have yet to see any actual cases of negative results from this.]

Posted by Gerry Haskins on June 19, 2009 at 07:28 AM IST #

Hi Gerry,

Thanks a lot for the detailed explanation. We have been using Update on attach as we mostly just patch the OS patches.

We were able to update on attach zones created under U5 to U7 parallelly, we did install the patch 119254-66 before we tried so am not sure if this works without the patch also.

what we did was
a) detach the zones from the U5 global.
Note: The zoneroots and data are on SAN storage
b) Jump the U5 box to U7 + recommended patch cluster + 119254-66
c) create the zones using
zonecfg -z <zonename> create -a /zone-root
d) ran
for zone in `zoneadm list -c | grep -v global` ; do zoneadm -z $zone attach -u & ;
done

It looks like it has worked for now ... all the zones have an /var/sadm/system.../update_log which does not have any errors

Posted by kocp on June 21, 2009 at 06:18 AM IST #

I tested this patch on Sun Fire V240 (2x1GHz, 2x36GB, 2GB RAM) + Solaris 10 10/08 (clean install) + 3 sparse root local zones.

Without patch tool patch recommended patch set installed in 2hours. With patch tool patch it took 1h30min. That is 25% faster.

I think one could get better results with faster disks and more CPUs.

Posted by Bergman on June 24, 2009 at 12:55 AM IST #

Hi Bergman!

Yes, it's largely IO constrained, so the better the I/O subsystem, the better it'll perform.

Best Wishes,

Gerry.

Posted by Gerry Haskins on June 24, 2009 at 04:45 PM IST #

After installing 119255-66, there is an entry in /etc/patch/pdo.conf:

"Factors determining processes to be forked in the man page of pdo.conf(4)"

I can't seem to find this man page - is this deployed in another patch ?

Thanks

Mark.

Posted by Mark on June 25, 2009 at 01:41 AM IST #

This sounds good, and I would like to try it out, but I have a question.
Does the "Number of on-line CPUs" take care of cpu cores as well? If I have 2 x 64core cpu server ( mean 2 physical cpu, and each physical cpu has 64 cores), what number I should specify for "Number of on-line CPUs"; 2 of 128?

Please email me your reply. Thanks.

Posted by Manoj on August 14, 2009 at 08:00 AM IST #

In parallel patching, cpu cores, threads are all taken into account, so in the example of 2 x 64 core above from Manoj, then parallel patching sees 128 cpus's.
ie for any sun4v architecture parallel patching will see the number of cpu's as defined by prtdiag, ie 5140 will have up to 128 cpu's.

The manpage for parallel patching ( pdo.conf in particular ) will be delivered in update 8, so once update 8 releases a patch for the man pages will become available, which will be installable on FCS upwards.

Also BTW, "zone update on attach" and "parallel patching" are two completely unrelated features in terms of functionality, that is to say, one will not impact the other in anyway.
Enda

Posted by Enda on August 14, 2009 at 11:36 AM IST #

Hi Edma,
Thanks so much for clarifying my doubts. It does make sense to see 128 cpus on a system with 2 x 64 cores; that's what prtdiag sees. This clarification is a big help to me on patching as we have servers with over 20-25 zones on each. I have installed patch 119254-67, and going to begin my test shortly.

Thanks again,
Manoj

Posted by Manoj on August 18, 2009 at 08:59 AM IST #

Hi Enda,
Sorry, I misspelled your name in my last posting.

Posted by Manoj on August 19, 2009 at 08:24 AM IST #

This is great news. Does parallel patching works whole zone as well? Thanks.

Posted by Ronald Kwok on August 31, 2009 at 10:09 AM IST #

Hi Ronald!

Yes!

Best Wishes,

Gerry.

Posted by Gerry Haskins on September 01, 2009 at 01:41 PM IST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today