On Database Patching and Support: A Primer for E-Business Suite Users
By Steven Chan - EBS-Oracle on Feb 20, 2009
[Update Jan. 31, 2013: The database patchset grace period has increased to 24 months for some patchsets. See this article for more details]
The Oracle Server Technologies division has issued some important updates to their support policies in the following document:
- Database, FMW, EM Grid Control, and OCS Software Error Correction Support Policy (Metalink Note 209768.1)
These changes affect support policies for the database, Oracle Enterprise Manager, Fusion Middleware, and Collaboration Suite. These changes are important enough to warrant an in-depth discussion about the implications of the database-related updates for E-Business Suite customers. This article also discusses the E-Business Suite database certification process and the safety of applying interim patches to your Apps environments. I'll cover the Apps-specific implications for the other technology products in a future article.
What Are the Important Changes?
The Server Technologies division periodically produces database patchsets: integrated, cumulative, and fully-tested collections of fixes issued between product releases. These are usually produced one or twice a year.
The Server Technologies division also releases three other types of bug fixes between patch sets:
- Quarterly Critical Patch Updates (security fixes)
- Patch bundles
- An interim patch (sometimes called a "one-off") for non-Windows platforms
Server Technologies provides bug fixes for the latest patchset and the previous patchset (i.e. N and N-1 patchset releases) for a given database release. For example, bug fixes are available for the 10g Release 2 Database patchsets:
- The latest (N) 10gR2 patch set: 10.2.0.4
- The previous (N-1) 10gR2 patch set: 10.2.0.3
The important change for E-Business Suite users is that database fixes are now produced for the previous (N-1) patchset for twelve months after the release of the latest (N) patchset. This is called the grace period. The actual calendar dates of the grace periods for database releases varies for individual operating systems. They're listed here:
- Release Schedule of Current Database Patch Sets (Metalink Note 742060.1)
What Is the Impact on EBS Customers?
The E-Business Suite division certifies major database releases, patchsets, and Critical Patch Updates whenever they're released. These certifications are performed on the released database code, not the beta internal drops. These certifications take time, of course. If our certification tests find Apps compatibilities issues with a given patchset, we need to loop back to the Server Technologies to obtain interim patches. The patchset and interim patches are then retested with the E-Business Suite.
Let's look at two of the major database codelines. This illustrates the impact of the new Server Technologies support policy on EBS environments.
For the 11gR1 database codeline:
- 188.8.131.52 was released in October, 2008
- 184.108.40.206's grace period ends in October, 2009
- 220.127.116.11 was certified with Apps 11i in December, 2008
- 18.104.22.168 was certified with Apps 12 in February, 2009
- Apps 11i customers can get get patches for 22.214.171.124 for 10 months
- Apps 12 customers get patches for 126.96.36.199 for 8 months
For the 10gR2 database codeline:
- 10.2.0.4 was released in February, 2008
- 10.2.0.3's grace period ends in February, 2009
- 10.2.0.4 was certified with Apps 11i in July, 2008
- 10.2.0.4 was certified with Apps 12 in August, 2008
- Apps 11i customers can get patches for 10.2.0.3 for seven months
- Apps 12 customers can get patches for 10.2.0.3 for six months
The Server Technologies division understands that you coordinate upgrades to your production EBS environments with financial reporting cycles, business events, and other major EBS updates and architectural changes. They understand that your upgrade cycles can often exceed the database grace periods. In circumstances like these -- i.e. outside of the grace period -- the Server Technologies division has committed to being flexible about approving requests for interim patches for the N-1 release for critical issues affecting your EBS production environments.
Another thing you should consider is that new interim patches will not be produced for older patchsets, e.g. the N-2 release or older. In the case of the 10gR2 codeline, this means that while you can still download existing interim patches for, say, 10.2.0.2, no new interim patches will be produced for those older releases.
A prudent DBA would conclude that the safest policy is to keep your E-Business Suite environment reasonably up-to-date with database patchsets. Even if you can't apply the latest patchset available, it's wise to try and keep up with the N-1 patchset, at minimum.
What's Involved in an EBS Database Certification?
I mentioned that EBS certifications with new database releases or patchsets take time. That's because new database releases are a big deal around here. We invest a lot of resources in certifying major Oracle database releases and patchsets with E-Business Suite 11i and 12. Major new database versions such as the first 11gR1 188.8.131.52 release get a lot of attention. New patchsets such as the 184.108.40.206 or 10.2.0.4 patchsets get a lot of attention, too.
These certifications involve my own certification teams, E-Business Suite product family teams (e.g. Financials, Order Management, Supply Chain), our Architecture teams, our Applications Technology Group (e.g. AOL/J, RapidClone, OA Framework), our Applications Platforms Group, our Applications Performance Group, Oracle Support, and of course, teams within the Server Technologies Division themselves.
This is a bit of a longwinded way of illustrating that when we announce a particular database release's certification with the E-Business Suite, it's been tested by many different E-Business Suite products and tools in a common set of environments with a reference product family and technology stack baseline.
Interim Patches Aren't Certified with EBS
The uncomfortable reality of software release cycles means that certain interim database patches will fall outside of that EBS certification process. Our Apps certification teams don't apply or certify interim database patches unless they're also included in something like the quarterly Critical Patch Updates.
Here's a typical release and certification cycle:
- We certified the 220.127.116.11 database with Apps 11i in March, 2008.
- We certified the 18.104.22.168 patchset with Apps 11i in December, 2008.
- Between March and December, interim database patches were released to fix specific issues found in 22.214.171.124.
- Those interim patches were not tested with the E-Business Suite.
Let's say that you upgraded your EBS database to 126.96.36.199 in June. In the process of testing some of your EBS extensions, you found a generic database bug. Oracle Support identifies an interim patch that could potentially fix that problem.
This triggers a common dilemma for some Apps sysadmins: you can stay with the certified configuration and live with the problem, or you can add an uncertified database patch to your environment, potentially destabilizing things.
Is It Safe to Apply Interim Database Patches?
Yes, it's generally safe to apply interim database patches to your E-Business Suite environments. Remember that an E-Business Suite environment is just like any custom application built on Oracle databases, so interim patches don't pose an unusually high risk for EBS environments in particular. Generally speaking, if it's safe for a generic Oracle database user, it's safe for an E-Business Suite environment, too.
Also remember that interim database patches are typically small and focused on resolving a specific issue. In other words, the solution will likely not make things any worse than they already are. One can argue that certain interim database patches, especially those addressing performance optimization or memory management issues, may be more important to apply than some Apps product patches.
That said, there are some important things you should consider:
- Are your reported symptoms identical to the symptoms that the interim patch addresses?
- Is this a critical issue for your environment or users?
- Can you test your environment thoroughly to ensure that there are no other unexpected side-effects of this patch?
If you can't answer "Yes" to all of these questions, it might be worth reflecting on the wisdom of applying that interim patch. For example, if you're experiencing a problem that only partially resembles the symptoms resolved by an interim patch, it's possible that that patch won't fix your problem. If that's the case, it might be worthwhile hunting down a different patch that more-closely targets your problem.
Ultimately, this all boils down to testing. If you're considering applying an interim database patch, you should remember that it's received limited testing by the Server Technologies group but likely hasn't been tested by the E-Business Suite division at all. So, the onus is on you to verify whether the patch fixes your problem while being otherwise benign to the rest of your Apps environment.
Relax, It's Supported
The reassuring part of all of this is that interim database patches are supported with the E-Business Suite -- even if they're not explicitly certified. In other words, if you apply an interim database patch to your certified Apps database configuration, both the Server Technologies and EBS Support divisions will do everything possible to debug issues with that patch.
If the patch triggers other unexpected side effects, we will help you diagnose your issues, find workarounds, or request additional database or E-Business Suite fixes, as needed.
These new Server Technologies support policies have important operational and planning implications for E-Business Suite customers. You need to evaluate these new policies against your own practices around upgrades to new database releases, patchsets, and interim patches.
This area gets a lot of debate internally at Oracle and you are invited to participate in that dialogue. Feel free to sound off in the comments, below. I'll pass on notable feedback to our colleagues in the Server Technologies division, as appropriate.