Top 5 Myths About Patching Apps Environments

When I was younger, I thought I could change this world.  Now I no longer think so but for emotional reasons I must keep on fighting a holding action.
~ Robert Anson Heinlein

A rational person might contend that actions follow attitudes which follow beliefs.  Bad news:  there's a substantial amount of psychological literature that suggests that this isn't how we really tick.

There's an equally large body of empirical data that suggest that beliefs are relatively intractable.  Once established, certain beliefs don't change, regardless of the best data, clear reasoning, and eloquence that can brought to bear on the subject.  Recent fMRI studies suggest that brain structures may develop in such a way that people with strongly-held beliefs are actually unable to process new information that contradicts those beliefs.

The broader implications of this are kind of depressing to contemplate.  But I digress.

For the narrow purposes of this discussion, I'm compelled to make an admittedly-quixotic attempt to persuade you that the benefits of keeping your E-Business Suite environment up-to-date far outweigh the costs.

"We Can't Upgrade Because..."
  1. It requires too much downtime
  2. Testing is too expensive for end-users
  3. It's too complicated
  4. We don't have enough staff
  5. It ain't broken; why fix it?
Myth #1:  It Requires Too Much Downtime

This seems rational on the surface.  After all, a downtime for patching seems worse than no downtimes.  

Remember that we issue patches for four major reasons: 
  1. To add new functionality
  2. To improve stability
  3. To improve performance
  4. To improve security
The corollary is that an unpatched system may have fewer or less-sophisticated features, and be slower, less stable, and less secure than a fully-patched one.  Questions to consider if you believe this myth:
  • How much downtime is currently caused by unplanned outages due to unpatched stability bugs?
  • How much downtime is needed to bounce your application servers due to unpatched JVM memory leaks?
  • How much downtime would be required if an attacker takes advantage of an unpatched security risk?
There are a number of key ways to reduce downtimes.  One of these is to use shared filesystems in environments with multiple application servers.  There are a number of other ways, too.  These are beyond the scope of this article, so I've covered the top seven way of reducing downtimes in this article.

Myth #2:  Testing is Too Expensive For End-Users

Many organizations recruit business users to participate in User Acceptance Tests for patches.  If you do this, it's true that this may reduce participants' productivity for the duration of your testing phase. 

If you don't keep your environment up-to-date, the key consideration are: 
  • How much productivity is lost for all end-users -- not just the testers -- due to unpatched performance, stability, or security bugs?
  • How much could productivity be improved by new features?

Myth #3:  It's Too Complicated

All DBAs know the drill:  before patching, print all patch READMEs and spread them in neat piles across a big surface.  Read them, highlight them, then reread them.  Download more patches.  Repeat. 

Every patch has some set of prerequisites.  These prerequisites lead to other prerequisites.  The key thing to remember is that it's always easier to patch an up-to-date system than something that's much older.  The bigger the gap between your current system and target patch level, the more complex the upgrade will be. 

The best way of reducing the complexity of upgrades is to keep your system up-to-date.

Myth #4:  We Don't Have Enough Staff

I need to be candid here:  there's a kernel of truth in this one.  This is a direct outcome of believing Myth #3.  If you haven't patched your E-Business Suite Release 11i environment since it was installed in 2001, then you may have a big and complex upgrade project on your hands.  This might take more staff than you have.

So, the best way of avoiding the truth of this one is to keep up-to-date with your patching.  If you're up-to-date, applying a given patch is less complex and requires fewer staff.

Myth #5:   It Ain't Broken; Why Fix It?

With apologies to English teachers everywhere, that old axiom has a seductive ring of truth to it.  After all, your system seems to be running just fine, thank you.  Why bother potentially destabilizing something that everyone's happy with?

Repeating the key points made in Myth #1:  new releases are issued to provide new functionality and improve stability, performance, and security.  If you're running an older release, it has -- by definition -- issues in these areas that you may not have noticed yet.

The Worst-Case Scenario

If I were an IT manager running an older E-Business Suite release, say 11.5.2, this would be the nightmare scenario that would keep me awake at night:

My business requirements changes due to an acquisition or a change in business practices.  These changes trigger a different load or usage profile on my E-Business Suite environment.  This triggers a Severity 1 outage.  My production system is down.

I call Oracle Support for help.  Good news: they can reproduce the problem on 11.5.10.CU2.  Bad news: due to technical dependencies (and supported by the fact that 11.5.2 is out of Premier Support status), they can only issue patches on top of 11.5.10.CU2.  There's no way of getting that patch backported to 11.5.2.

Now I'm faced with a huge upgrade from 11.5.2 to 11.5.10.CU2.  My production system is still down, and it will be down for the duration.  Users are burning me in effigy in the parking lot.  Business impact analysis, user acceptance testing, load-testing, fail-over testing -- all of those activities are jettisoned in my panicked attempts to get my environment running again.


It's Never Too Late

At this point, I suspect I've lost most of my readers.  By now, they've clicked on the latest YouTube video or something more cheerful. 

For the handful of my remaining readers:  if you're in a hole, it's never too late to stop digging.  Less metaphorically:  just because you're behind in patching doesn't mean that you should just give up entirely. 

We can help you put together a strategy for getting up-to-date.  Your first call should be to your Oracle account manager.  He or she can engage specialists in Sales Consulting or Consulting to help.  There are also excellent groups in our Advanced Customer Services organization (a.k.a. Field Support, On-Site Support, Premium Support) who offer a number of of packaged and customizable support offerings in the patching realm. Alternately, there are a number of excellent third-party consultants who can help you with this.  Or, if you're really intent on doing it yourself, there are forums on the Oracle Technology Network and elsewhere where you can brainstorm possible upgrade strategies.

Whatever path you choose:  start now.  It's never too late.

Related
Comments:

Don't forget us! ACS (Advanced Customer Services) a.k.a Field Support, On-Site Support, Premium Support, etc... We have a lot of packaged and customizable support offerings in the patching realm.
http://www.oracle.com/support/advanced-customer-services/index.html

And more specific information on Patch Assessments at:
http://se.ie.oracle.com/proj_patch/index.shtml
(Sorry Internal Only)

Posted by Paul Emblin on June 08, 2007 at 02:12 AM PDT #

Whoops!  Didn't mean to leave ACS out, Paul.  ACS is a very useful option for customers to consider, of course.  I've updated the article.Regards,Steven 

Posted by Steven Chan on June 08, 2007 at 03:17 AM PDT #

Steven, as always, your comments are very timely. I had this very conversation with our CIO just a few hours before your post, but failed to state the case as well as you did here. Think I'll just email him the url...

Thanks,
--Floyd--

Posted by Floyd Teter on June 08, 2007 at 07:03 AM PDT #

Mark,I completely agree with you.  Having a policy for keeping your E-Business Suite environments doesn't mean that you ignore testing and QA of patches in your environment.   In fact, the more often you patch, the more systematic your testing processes should be.I'm sorry to hear about your experiences with those two recent patches.  I trust that your problem reports for these two issues have resulted in new, working patches?  I'll plan for a future article on testing policies and strategies.  Thanks very much for the suggestion.Regards,Steven

Posted by Steven Chan on June 20, 2007 at 02:37 AM PDT #

Whilst I agree with the principle of trying to keep your systems reasonably up to date, you should also recognise that Oracle do not have a perfect track record. I have two very recent examples of patches. One would render Oracle Purchasing unusable - this has taken nearly three months to get fixed. The other is a database patch that breaks EBS on Windows x64 - again a three month lead time to fix. Try explaining three months of downtime with a "well at least the system is up to date". It's a case of the operation worked but the patient died anyway.

Hence the absolute need for proper testing of ANY patch prior to application.

Posted by Mark Allcock on June 20, 2007 at 04:10 AM PDT #

Steven,

Another excellent article that we will continue to refer to as you clearly articulate the potential impacts of continuing to believe in these myths.

In the vein of retentive-DBAs everywhere, you have a line in the top of the patch article that refers to the "top ten way"... You might want to change it to the "top seven ways"...

Just keeps me awake at night... :-)

Great work as always.

Posted by John Stouffer on June 21, 2007 at 02:11 AM PDT #

Allan,Aye, there's the rub.  The minute an organization makes invasive customizations to any packaged application, -- including the E-Business Suite -- they are committed to the added costs and delays arising from having to manually reapply their own changes every time the packaged application is updated.  At minimum, we release our Critical Patch Updates every quarter.  Given that we release many, many other patches to the E-Business Suite throughout the year, this means that your in-house customization team will likely be working full-time on reapplying your customizations to every patched environment.  If I were an IT manager, patching complications are the biggest argument against invasive customizations.  OAF and Forms Personalizations
are preserved across patch updates, but not invasive customizations,
making the former the preferred method of implementing changes to the
E-Business Suite. 

Regards,Steven

Posted by Steven Chan on June 28, 2007 at 01:17 AM PDT #

Sigh, so much for the myths and now for the one that isn't a myth. We customized damnit!

Posted by Allan Nelson on June 28, 2007 at 02:52 AM PDT #

The premise of your myths is the new functionality is magically better than the old and will save your organization a ton of time. You don't dispute that patching takes a long time but counter with it will be better in the end. You don't dispute that it takes a long time to test, but say it will be better when it's over. Phooey.

In a typical organization in the real world, the company has it's procesess and procedures down and is relatively efficient at doing those processes. Add in new functionality and those users productivity goes down the tubes for 6 months until they get back up to speed. And lets get serious, you're telling me that AP payments in R12's HTML nightmare with it's 5 tabs and 60+ clicks is more efficient than the forms version?

Upgrading from 11i to R12 is a nightmare at best unless you want to spend a half-million bucks on Oracle Consulting. I can understand why people are still on 10.7.

Posted by Jeff Hunter on July 03, 2009 at 09:53 AM PDT #

Hi, Jeff,

I appreciate your skepticism. Generalities are simply that -- meant to apply to the majority. There will always be individual cases that disprove them.

It's true that new functionality always comes at a cost in retraining and productivity loss. I don't make any claims about whether the cost-benefits analysis will always justify an upgrade. I lack the empirical data to support such an analysis. This is something that individual organizations need to assess for themselves.

Indeed, I've heard from customers who are still running EBS 10.7. If an organization is comfortable with running desupported EBS releases, with possible stability or security issues, running on desupported Oracle technology like the 8i database, well, there's not much that I can say that will change their minds.

Regards,
Steven

Posted by Steven Chan on July 06, 2009 at 04:06 AM PDT #

Hi Steven,

I do realize that I'm putting this comment on a relatively old post.

Many of the E-Business Suite patches deliver branched versions of code. What we do currently is keep a track of the branched versions delivered by a patch manually in a spreadsheet - I cannot find any documentation on My Oracle Support on the maintenance aspects of such branched versions of code scattered over an installation. Is there a table/view that helps determine conclusively what are the current branched versions of code ?

I'd appreciate if you shed some light on this.

Thanks,
Rakesh

Posted by Rakesh Tripathi on January 11, 2010 at 09:43 PM PST #

Hi, Rakesh,

No, as far as I'm aware, these resources are not available externally. The easiest way of catching up for a given product family is to apply their latest Family Pack.

Regards,
Steven

Posted by Steven Chan on January 12, 2010 at 01:29 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today