The Software Vendor's Dilemma
By Steven Chan (Oracle Development) on Jan 10, 2007
Last year was another big one for us in terms of releases, and there are more new certifications and patches in the pipeline for the E-Business Suite. This bounty creates challenges for IT managers. In the last month or so, I've been involved in a number of customer dialogues where I've had to discuss The Software Vendor's Dilemma and its concomitant implications for IT management policies.
For the limited purposes of this discussion, let's make the following assertions:
- All software has bugs of some kind
- A given bug will impact some customers more than others
- Some bugs will be hidden until other bugs are fixed
- Some bug fixes will introduce new bugs
- Customers incur direct and indirect costs when applying bug fixes
- If you don't issue bug fixes quickly, some users will complain
- If you do issue bug fixes quickly, other users will complain
Assuming that your software vendors provide patches, that leads naturally and inevitably to the IT Manager's Dilemma:
- If you don't apply patches, your end-users will complain
- If you do apply patches, your end-users will complain... about maintenance downtimes and new rounds of User Acceptance Tests
So far, I've been speaking in generalities. Let's get concrete and talk about what we do here in the Oracle Applications Technology Group. When we fix E-Business Suite technology stack bugs, there are several major ways of getting those fixes into your hands:
- A small patch that fixes a single problem for a specific technology stack component
- A collection of patches for that particular technology stack component
(e.g. for AutoConfig)
- A collection of patches for a set of interdependent technology stack components
(e.g. Oracle Applications Framework)
- A collection of all patches released for all components in the Applications Technology Group product family
Drawing the Line
Within the limited context of this discussion, one of our responsibilities here in the Applications Technology Group is to ensure that we get fixes to you as quickly as possible. A given fix may appear in any one -- or all -- of the categories above.
It's up to you to assess which categories of patches you should apply. We don't have the right to dictate what patches you must install. This is why we flag a given patch as recommended, or highly recommended, but never mandatory.
Likewise, we don't specify when you should apply them, how to test them, how to mitigate your operational risks, nor how to convince your management and business stakeholders to agree to the resulting maintenance downtimes and upgrades. These important business questions are outside of Oracle Development's scope to answer, since their answers depend upon your own operational norms, business priorities, practices, processes, and beliefs. That's the meat-and-potatoes of an IT manager's core responsibilities.
Some Considerations for Managing Applications Patches
I'm assuming that you already have a business framework for assessing and managing a patch's costs, benefits, and risks. Here are some additional things from our own Oracle patching processes that you might wish to consider for E-Business Suite technology stack patches:
- Here in Oracle, all Apps patches go through multiple staging environments, each with their own tests and exit criteria, before being deployed into production. This may seem blindingly obvious, but I'm repeatedly surprised to hear that some of our large customers don't follow this kind of process to minimize risk.
- DBAs, system administrators, and IT managers will have differing perspectives on the value and priority of a given patch, and input from all levels is often required to make a good operational decision. And don't forget your end-users: ignore their input at your own peril.
- All of the quarterly Critical Patch Updates are highly recommended, and we apply all of these -- without fail -- to all of our own Oracle environments
- Individual "one-of-a-kind" emergency patches receive less testing than ATG Family Pack Rollup patches, so we only apply these to Oracle production environments when absolutely necessary, i.e. when the business benefits clearly outweigh the risks
- We apply all ATG Family Pack Rollup patches to all critical environments as soon as they're released. No questions, no exceptions. These Rollup patches are cumulative and are subjected to the highest level of testing that we can bring to bear on a patch.
- We use AutoConfig to minimize the overhead and hassle of managing a given patch's impact on configuration files. And yes, even though we tell you not do so so, sometimes we have to customize those configuration files. If you must customize a configuration file, follow our guidelines so that AutoConfig preserves your customizations.
- Not all READMEs are created equal. Our internal Oracle IT staff are not shy about telling us when a patch's README is ambiguous, misleading, or incomplete. Their feedback can be rather pointed sometimes. Likewise, if a patch's README leaves you baffled, log a Service Request via Metalink and get a definitive answer before applying a particular patch.
I don't know much, but I know these things to be true: nobody likes going to the dentist, or mowing the lawn, or cleaning the toilet, but it's still something that you need to do regularly. The longer you put it off, the worse it will be.
The same goes for patching your E-Business Suite environments. Having a clearly-defined patch prioritization and management process and scheduling a few maintenance windows a year to apply Critical Patch Updates and E-Business Suite Family Packs is a lot less painful than having a Severity 1 patching crisis on the night before Thanksgiving.
If you have tips on navigating the Scylla and Charybdis of E-Business Suite patching, I'd be delighted if you hit the Comment link and shared them with our readers.
- The Blue Bridge of Death
- Products and Families and Versions - Oh, My!
- New Techstack Baseline for Release 11i, Redux
- Inside Oracle's Own Global Operations Data Center