Why people don't patch and upgrade?!?

Please see the JAPANESE version of this blog post here!!

Discussing the topic "Why Upgrade" or "Why not Upgrade" is not always fun. Actually the arguments repeat from customer to customer. Typically we hear things such as:

  • A PSU or Patch Set introduces new bugs
  • A new PSU or Patch Set introduces new features which lead to risk and require application verification 
  • Patching means risk
  • Patching changes the execution plans
  • Patching requires too much testing
  • Patching is too much work for our DBAs
  • Patching costs a lot of money and doesn't pay out

And to be very honest sometimes it's hard for me to stay calm in such discussions. Let's discuss some of these points a bit more in detail.

  • A PSU or Patch Set introduces new bugs
    Well, yes, that is really true as there's no software containing more than some lines of code being bug free. This applies to Oracle's code as well as too any application or operating system code. But first of all, does that mean you never patch your OS because the patch may introduce new flaws? And second, what is the point of saying "it introduces new bugs"? Does that mean you will never get rid of the mean issues we know about and we fixed already? Scroll down from MOS Note:161818.1 to the patch release you are on, no matter if it's 10.2.0.4 or 11.2.0.3 and check for the Known Issues And Alerts.
    Will you take responsibility to know about all these issues and refuse to upgrade to 11.2.0.4? I won't.

  • A new PSU or Patch Set introduces new features
    Ok, we can discuss that. Offering new functionality within a database patch set is a dubious thing. It has advantages such as in 11.2.0.4 where we backported Database Redaction to. But this is something you will only use once you have an Advanced Security license. I interpret that statement I've heard quite often from customers in a different way: People don't want to get surprises such as new behaviour. This certainly gives everybody a hard time. And we've had many examples in the past (SESSION_CACHED_CURSROS in 10.2.0.4,  _DATAFILE_WRITE_ERRORS_CRASH_INSTANCE in 11.2.0.2 and others) where those things weren't documented, not even in the README. Thanks to many friends out there I learned about those as well. So new behaviour is the topic people consider as risky - not really new features. And just to point this out: A PSU never brings in new features or new behaviour by definition!

  • Patching means risk
    Does it really mean risk? Yes, there were issues in the past (and sometimes in the present as well) where a patch didn't get installed correctly. But personally I consider it way more risky to not patch. Keep that in mind: The day Oracle publishes an PSU (or CPU) containing security fixes all the great security experts out there go public with their findings as well. So from that day on even my grandma can find out about those issues and try to attack somebody. Now a lot of people say: "My database does not face the internet." And I will answer: "The enemy is sitting already behind your firewalls. And knows potentially about these things."
    My statement: Not patching introduces way more risk to your environment than patching. Seriously!

  • Patching changes the execution plans
    Do they really? I agree - there's a very small risk for this happening with Patch Sets. But not with PSUs or CPUs as they contain no optimizer fixes changing behaviour (but they may contain fixes curing wrong-query-result-bugs). But what's the point of a changing execution plan? In Oracle Database 11g it is so simple to be prepared. SQL Plan Management is a free EE feature - so once that occurs you'll put the plan into the Plan Baseline. Basta! Yes, you wouldn't like to get such surprises? Then please use the SQL Performance Analyzer (SPA) from Real Application Testing and you'll detect that easily upfront in minutes. And not to forget this, a plan change can also be very positive!
    Yes, there's a little risk with a database patchset - and we have many possibilites to detect this before patching.

  • Patching requires too much testing
    Well, does it really? I have seen in the past 12 years how people test. There are very different efforts and approaches on this. I have seen people spending a heck of money on external licenses or on project team staffing. And I have seen people sailing blindly without any tests just going the John-Wayne-approach.
    Proper tools will allow you to test easily without too much efforts. See the paragraph above. We have used Real Application Testing in so many customer projects reducing the amount of work spend on testing by over 50%.
    But apart from that at some point you will have to stop testing. If you don't stop you'll get lost and you'll burn money. There's no 100% guaranty. You will have to deal with a little risk as reaching the final 5% of certainty will cost you the same as it did cost to reach 95%. And doing this will lead to abnormal long product cycles that you'll run behind forever. And this will cost even more money.

  • Patching is too much work for our DBAs
    Patching is a lot of work. I agree. And it's no fun work. It's boring, annoying. You don't learn much from that. That's why you should try to automate this task. Use the Database's Lifecycle Management Pack. And don't cry about the fact that it costs money. Yes it does. But it will ease the process and you'll save a lot of costs as you don't waste your valuable time with patching. Or use Oracle Database 12c Oracle Multitenant and patch either by unplug/plug or patch an entire container database with all PDBs with one patch at one time in one task.
    We have customer reference cases proofing it saved them 75% of time, effort and cost since they've used Lifecycle Management Pack. So why don't you use it?

  • Patching costs a lot of money and doesn't pay out
    Well, see my statements in the paragraph above. And it pays out as flying with a database with 100 known critical flaws in it which are already fixed by Oracle (such as in the Oct 2013 PSU for Oracle Database 12c) will cost ways more in case of failure or even data loss. Bet with me?

Let me finally ask you some questions.

What cell phone are you using and which OS does it run? Do you have an iPhone 5 and did you upgrade already to iOS 7.0.3? I've just encountered on my iPhone 5 that the alarm (which I rely on when traveling) has gotten now a dependency on the physical switch "sound on/off". If it is switched to "off" physically the alarm rings "silently". What a wonderful example of a behaviour change coming in with a patch set. Will this push you to stay with iOS5 or iOS6? No, because those have security flaws which won't be fixed anymore.And I tell you how I have found out that: I have tested the alarm after updating to iOS 7.0.3 as I heavily rely on that feature. I don't test everything as my time and my resource won't allow that - but I usually check the most important things.

What browser are you surfing with? Do you use Mozilla 3.6? Well, congratulations to all the hackers. It will be easy for them to attack you and harm your system. I'd guess you have the auto updater on.  Same for Google Chrome, Safari, IE. Right?

-Mike



Comments:

"Because getting outage on production systems is hard, getting the money for features that could mitigate disruption is harder, and re-architecting our systems is even more disruptive than patching."

You're welcome.

Posted by guest on November 21, 2013 at 11:36 AM CET #

You've wrote:
"Because getting outage on production systems is hard, getting the money for features that could mitigate disruption is harder, and re-architecting our systems is even more disruptive than patching"

My reply:
Getting an outage because of an already fixed issue is even harder. Why the heck do you think we release PSUs with the most critical fixes? Including security fixes?
How about the risk of NOT applying such an important patch and then aparently hitting such an issue? Believe me, I have seen this too often. People telling me about the risk of patching completely ignoring the risk and trouble and costs of hitting an issue we have fixed already and marked as "Important for Everybody".
Furthermore, PSUs don't contain features.
And furthermore, 11.2.0.4 for instance has 7000 fixes included - sailing without fixes is similar to driving a Porsche at 260km/h speed at a German Autobahn with the tires of a Fiat Panda (no offense to Fiat here!!) ... you can try that but I wouldn't recommend it. And please don't cry if you crash into the woods with your lovely fast and fancy Porsche :-)

Posted by Mike on November 22, 2013 at 03:22 PM CET #

People prefer to postpone their testing (yes patches HAVE to be tested !) to the next release / version upgrade rather than running a round of testing every 3 months.

Posted by Hemant K Chitale on January 09, 2014 at 10:43 AM CET #

Thanks for your comment - but let me ask you if you test the patches for your cell phone or your desktop's OS as well? I think we both know the answer.

When Oracle releases a PSU containing high-risk fixes and security fixes and one is telling me that this has to be tested I usually ask two things:
- how does your testing look for such a PSU?
- are you the person taking the risk of exposing your database to known security issues which got fixed by the vendor (us, Oracle), got highly recommended to apply ASAP (by us, Oracle) - and you don't apply them for some reason? Do you take this risk???

Actually I hear this for many many years. I heard it 17 years ago when I started in Oracle Support. And I still hear it. And I fully disagree. You don't have to test a PSU. If the PSU does something strange to your database log it to Oracle and escalate it immediatelly. But don't take the risk of exposing your database(s) to security threats or potential cruel mean bugs we know about and we fixed for you to protect your database(s) :-)

I can see that patching is no fun job. You don't learn much about it. That's why we offer solutions to automate it. And if you don't like our solutions to automate it then use your own (I have customers who wrote their own).

Cheers, Mike

Posted by Mike on January 09, 2014 at 10:56 AM CET #

Post a Comment:
Comments are closed for this entry.
About

Mike Dietrich - Oracle Mike Dietrich
Senior Principal Technologist - Database Upgrade Development Group - Oracle Corporation

Based near Munich/Germany and spending plenty of time in airplanes to run either upgrade workshops or work onsite with reference customers. Acting as interlink between customers and the Upgrade Development.

Contact me either via XING or LinkedIn

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
2
3
4
5
6
9
10
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Slides Download Center
OOW Slides Download
Visitors since 17-OCT-2011
White Paper and Docs
Oracle Blogs
Workshops
Viewlets and Videos
This week on my Rega/iPod/CD
Workshop Map
Upgrade Reference Papers