« Comparing Bandwidth Requirements between Release 11i and 12 | Main | UK OUG: Apps DBA for OEBS SIG Meeting »

Top 5 Myths About Patching Apps Environments

When I was younger, I thought I could change this world.  Now I no longer think so but for emotional reasons I must keep on fighting a holding action.
~ Robert Anson Heinlein

A rational person might contend that actions follow attitudes which follow beliefs.  Bad news:  there's a substantial amount of psychological literature that suggests that this isn't how we really tick.

There's an equally large body of empirical data that suggest that beliefs are relatively intractable.  Once established, certain beliefs don't change, regardless of the best data, clear reasoning, and eloquence that can brought to bear on the subject.  Recent fMRI studies suggest that brain structures may develop in such a way that people with strongly-held beliefs are actually unable to process new information that contradicts those beliefs.

The broader implications of this are kind of depressing to contemplate.  But I digress.

For the narrow purposes of this discussion, I'm compelled to make an admittedly-quixotic attempt to persuade you that the benefits of keeping your E-Business Suite environment up-to-date far outweigh the costs.

"We Can't Upgrade Because..."
  1. It requires too much downtime
  2. Testing is too expensive for end-users
  3. It's too complicated
  4. We don't have enough staff
  5. It ain't broken; why fix it?
Myth #1:  It Requires Too Much Downtime

This seems rational on the surface.  After all, a downtime for patching seems worse than no downtimes.  

Remember that we issue patches for four major reasons: 
  1. To add new functionality
  2. To improve stability
  3. To improve performance
  4. To improve security
The corollary is that an unpatched system may have fewer or less-sophisticated features, and be slower, less stable, and less secure than a fully-patched one.  Questions to consider if you believe this myth:
  • How much downtime is currently caused by unplanned outages due to unpatched stability bugs?
  • How much downtime is needed to bounce your application servers due to unpatched JVM memory leaks?
  • How much downtime would be required if an attacker takes advantage of an unpatched security risk?
There are a number of key ways to reduce downtimes.  One of these is to use shared filesystems in environments with multiple application servers.  There are a number of other ways, too.  These are beyond the scope of this article, so I've covered the top seven way of reducing downtimes in this article.

Myth #2:  Testing is Too Expensive For End-Users

Many organizations recruit business users to participate in User Acceptance Tests for patches.  If you do this, it's true that this may reduce participants' productivity for the duration of your testing phase. 

If you don't keep your environment up-to-date, the key consideration are: 
  • How much productivity is lost for all end-users -- not just the testers -- due to unpatched performance, stability, or security bugs?
  • How much could productivity be improved by new features?
Myth #3:  It's Too Complicated

All DBAs know the drill:  before patching, print all patch READMEs and spread them in neat piles across a big surface.  Read them, highlight them, then reread them.  Download more patches.  Repeat. 

Every patch has some set of prerequisites.  These prerequisites lead to other prerequisites.  The key thing to remember is that it's always easier to patch an up-to-date system than something that's much older.  The bigger the gap between your current system and target patch level, the more complex the upgrade will be. 

The best way of reducing the complexity of upgrades is to keep your system up-to-date.

Myth #4:  We Don't Have Enough Staff

I need to be candid here:  there's a kernel of truth in this one.  This is a direct outcome of believing Myth #3.  If you haven't patched your E-Business Suite Release 11i environment since it was installed in 2001, then you may have a big and complex upgrade project on your hands.  This might take more staff than you have.

So, the best way of avoiding the truth of this one is to keep up-to-date with your patching.  If you're up-to-date, applying a given patch is less complex and requires fewer staff.

Myth #5:   It Ain't Broken; Why Fix It?

With apologies to English teachers everywhere, that old axiom has a seductive ring of truth to it.  After all, your system seems to be running just fine, thank you.  Why bother potentially destabilizing something that everyone's happy with?

Repeating the key points made in Myth #1:  new releases are issued to provide new functionality and improve stability, performance, and security.  If you're running an older release, it has -- by definition -- issues in these areas that you may not have noticed yet.

The Worst-Case Scenario

If I were an IT manager running an older E-Business Suite release, say 11.5.2, this would be the nightmare scenario that would keep me awake at night:

My business requirements changes due to an acquisition or a change in business practices.  These changes trigger a different load or usage profile on my E-Business Suite environment.  This triggers a Severity 1 outage.  My production system is down.

I call Oracle Support for help.  Good news: they can reproduce the problem on 11.5.10.CU2.  Bad news: due to technical dependencies (and supported by the fact that 11.5.2 is out of Premier Support status), they can only issue patches on top of 11.5.10.CU2.  There's no way of getting that patch backported to 11.5.2.

Now I'm faced with a huge upgrade from 11.5.2 to 11.5.10.CU2.  My production system is still down, and it will be down for the duration.  Users are burning me in effigy in the parking lot.  Business impact analysis, user acceptance testing, load-testing, fail-over testing -- all of those activities are jettisoned in my panicked attempts to get my environment running again.

It's Never Too Late

At this point, I suspect I've lost most of my readers.  By now, they've clicked on the latest YouTube video or something more cheerful. 

For the handful of my remaining readers:  if you're in a hole, it's never too late to stop digging.  Less metaphorically:  just because you're behind in patching doesn't mean that you should just give up entirely. 

We can help you put together a strategy for getting up-to-date.  Your first call should be to your Oracle account manager.  He or she can engage specialists in Sales Consulting or Consulting to help.  There are also excellent groups in our Advanced Customer Services organization (a.k.a. Field Support, On-Site Support, Premium Support) who offer a number of of packaged and customizable support offerings in the patching realm. Alternately, there are a number of excellent third-party consultants who can help you with this.  Or, if you're really intent on doing it yourself, there are forums on the Oracle Technology Network and elsewhere where you can brainstorm possible upgrade strategies.

Whatever path you choose:  start now.  It's never too late.

Related

Comments (8)

Paul Emblin:

Don't forget us! ACS (Advanced Customer Services) a.k.a Field Support, On-Site Support, Premium Support, etc... We have a lot of packaged and customizable support offerings in the patching realm.
http://www.oracle.com/support/advanced-customer-services/index.html

And more specific information on Patch Assessments at:
http://se.ie.oracle.com/proj_patch/index.shtml
(Sorry Internal Only)

Steven Chan:

Whoops!  Didn't mean to leave ACS out, Paul.  ACS is a very useful option for customers to consider, of course.  I've updated the article.

Regards,
Steven 

Steven, as always, your comments are very timely. I had this very conversation with our CIO just a few hours before your post, but failed to state the case as well as you did here. Think I'll just email him the url...

Thanks,
--Floyd--

Steven Chan:

Mark,

I completely agree with you.  Having a policy for keeping your E-Business Suite environments doesn't mean that you ignore testing and QA of patches in your environment.   In fact, the more often you patch, the more systematic your testing processes should be.

I'm sorry to hear about your experiences with those two recent patches.  I trust that your problem reports for these two issues have resulted in new, working patches? 

I'll plan for a future article on testing policies and strategies.  Thanks very much for the suggestion.

Regards,
Steven


Whilst I agree with the principle of trying to keep your systems reasonably up to date, you should also recognise that Oracle do not have a perfect track record. I have two very recent examples of patches. One would render Oracle Purchasing unusable - this has taken nearly three months to get fixed. The other is a database patch that breaks EBS on Windows x64 - again a three month lead time to fix. Try explaining three months of downtime with a "well at least the system is up to date". It's a case of the operation worked but the patient died anyway.

Hence the absolute need for proper testing of ANY patch prior to application.

Steven,

Another excellent article that we will continue to refer to as you clearly articulate the potential impacts of continuing to believe in these myths.

In the vein of retentive-DBAs everywhere, you have a line in the top of the patch article that refers to the "top ten way"... You might want to change it to the "top seven ways"...

Just keeps me awake at night... :-)

Great work as always.

Steven Chan:

Allan,

Aye, there's the rub. 

The minute an organization makes invasive customizations to any packaged application, -- including the E-Business Suite -- they are committed to the added costs and delays arising from having to manually reapply their own changes every time the packaged application is updated. 

At minimum, we release our Critical Patch Updates every quarter.  Given that we release many, many other patches to the E-Business Suite throughout the year, this means that your in-house customization team will likely be working full-time on reapplying your customizations to every patched environment. 

If I were an IT manager, patching complications are the biggest argument against invasive customizations.  OAF and Forms Personalizations
are preserved across patch updates, but not invasive customizations,
making the former the preferred method of implementing changes to the
E-Business Suite. 



Regards,
Steven

Allan Nelson:

Sigh, so much for the myths and now for the one that isn't a myth. We customized damnit!

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

About This Entry

This page contains a single entry from the blog posted on June 7, 2007 5:54 PM.

The previous post in this blog was Comparing Bandwidth Requirements between Release 11i and 12.

The next post in this blog is UK OUG: Apps DBA for OEBS SIG Meeting.

Many more can be found on the main index page or by looking through the archives.

Archives

Subscribe to Updates

Powered by
Movable Type and Oracle