On Database Patching and Support: A Primer for E-Business Suite Users

[Update Jan. 31, 2013: The database patchset grace period has increased to 24 months for some patchsets.  See this article for more details]

The Oracle Server Technologies division has issued some important updates to their support policies in the following document:

These changes affect support policies for the database, Oracle Enterprise Manager, Fusion Middleware, and Collaboration Suite.  These changes are important enough to warrant an in-depth discussion about the implications of the database-related updates for E-Business Suite customers.  This article also discusses the E-Business Suite database certification process and the safety of applying interim patches to your Apps environments.  I'll cover the Apps-specific implications for the other technology products in a future article.

What Are the Important Changes?

The Server Technologies division periodically produces database patchsets:  integrated, cumulative, and fully-tested collections of fixes issued between product releases.  These are usually produced one or twice a year.

The Server Technologies division also releases three other types of bug fixes between patch sets:

  1. Quarterly Critical Patch Updates (security fixes)
  2. Patch bundles
  3. An interim patch (sometimes called a "one-off") for non-Windows platforms

Server Technologies provides bug fixes for the latest patchset and the previous patchset (i.e. N and N-1 patchset releases) for a given database release.  For example, bug fixes are available for the 10g Release 2 Database patchsets:

  • The latest (N) 10gR2 patch set: 10.2.0.4
  • The previous (N-1) 10gR2 patch set: 10.2.0.3

The important change for E-Business Suite users is that database fixes are now produced for the previous (N-1) patchset for twelve months after the release of the latest (N) patchset.  This is called the grace period.  The actual calendar dates of the grace periods for database releases varies for individual operating systems.  They're listed here:

What Is the Impact on EBS Customers?

The E-Business Suite division certifies major database releases, patchsets, and Critical Patch Updates whenever they're released.  These certifications are performed on the released database code, not the beta internal drops.  These certifications take time, of course.  If our certification tests find Apps compatibilities issues with a given patchset, we need to loop back to the Server Technologies to obtain interim patches.  The patchset and interim patches are then retested with the E-Business Suite.

Let's look at two of the major database codelines.  This illustrates the impact of the new Server Technologies support policy on EBS environments.

For the 11gR1 database codeline:

  • 11.1.0.7 was released in October, 2008
  • 11.1.0.6's grace period ends in October, 2009
  • 11.1.0.7 was certified with Apps 11i in December, 2008
  • 11.1.0.7 was certified with Apps 12 in February, 2009
  • Apps 11i customers can get get patches for 11.1.0.6 for 10 months
  • Apps 12 customers get patches for 11.1.0.6 for 8 months

For the 10gR2 database codeline:

  • 10.2.0.4 was released in February, 2008
  • 10.2.0.3's grace period ends in February, 2009
  • 10.2.0.4 was certified with Apps 11i in July, 2008
  • 10.2.0.4 was certified with Apps 12 in August, 2008
  • Apps 11i customers can get patches for 10.2.0.3 for seven months
  • Apps 12 customers can get patches for 10.2.0.3 for six months

The Server Technologies division understands that you coordinate upgrades to your production EBS environments with financial reporting cycles, business events, and other major EBS updates and architectural changes.  They understand that your upgrade cycles can often exceed the database grace periods.  In circumstances like these -- i.e. outside of the grace period -- the Server Technologies division has committed to being flexible about approving requests for interim patches for the N-1 release for critical issues affecting your EBS production environments.

Another thing you should consider is that new interim patches will not be produced for older patchsets, e.g. the N-2 release or older.  In the case of the 10gR2 codeline, this means that while you can still download existing interim patches for, say, 10.2.0.2, no new interim patches will be produced for those older releases.

A prudent DBA would conclude that the safest policy is to keep your E-Business Suite environment reasonably up-to-date with database patchsets.  Even if you can't apply the latest patchset available, it's wise to try and keep up with the N-1 patchset, at minimum.

What's Involved in an EBS Database Certification?

I mentioned that EBS certifications with new database releases or patchsets take time.  That's because new database releases are a big deal around here.  We invest a lot of resources in certifying major Oracle database releases and patchsets with E-Business Suite 11i and 12.  Major new database versions such as the first 11gR1 11.1.0.6 release get a lot of attention.  New patchsets such as the 11.1.0.7 or 10.2.0.4 patchsets get a lot of attention, too.

These certifications involve my own certification teams, E-Business Suite product family teams (e.g. Financials, Order Management, Supply Chain), our Architecture teams, our Applications Technology Group (e.g. AOL/J, RapidClone, OA Framework), our Applications Platforms Group, our Applications Performance Group, Oracle Support, and of course, teams within the Server Technologies Division themselves.

This is a bit of a longwinded way of illustrating that when we announce a particular database release's certification with the E-Business Suite, it's been tested by many different E-Business Suite products and tools in a common set of environments with a reference product family and technology stack baseline.

Interim Patches Aren't Certified with EBS

The uncomfortable reality of software release cycles means that certain interim database patches will fall outside of that EBS certification process.  Our Apps certification teams don't apply or certify interim database patches unless they're also included in something like the quarterly Critical Patch Updates.

Here's a typical release and certification cycle:

Let's say that you upgraded your EBS database to 11.1.0.6 in June.  In the process of testing some of your EBS extensions, you found a generic database bug.  Oracle Support identifies an interim patch that could potentially fix that problem.

This triggers a common dilemma for some Apps sysadmins:  you can stay with the certified configuration and live with the problem, or you can add an uncertified database patch to your environment, potentially destabilizing things.

Is It Safe to Apply Interim Database Patches?

Yes, it's generally safe to apply interim database patches to your E-Business Suite environments.  Remember that an E-Business Suite environment is just like any custom application built on Oracle databases, so interim patches don't pose an unusually high risk for EBS environments in particular.  Generally speaking, if it's safe for a generic Oracle database user, it's safe for an E-Business Suite environment, too.

Also remember that interim database patches are typically small and focused on resolving a specific issue.  In other words, the solution will likely not make things any worse than they already are.  One can argue that certain interim database patches, especially those addressing performance optimization or memory management issues, may be more important to apply than some Apps product patches.

That said, there are some important things you should consider:

  1. Are your reported symptoms identical to the symptoms that the interim patch addresses?
  2. Is this a critical issue for your environment or users?
  3. Can you test your environment thoroughly to ensure that there are no other unexpected side-effects of this patch?

If you can't answer "Yes" to all of these questions, it might be worth reflecting on the wisdom of applying that interim patch.  For example, if you're experiencing a problem that only partially resembles the symptoms resolved by an interim patch, it's possible that that patch won't fix your problem.  If that's the case, it might be worthwhile hunting down a different patch that more-closely targets your problem.

Ultimately, this all boils down to testing.  If you're considering applying an interim database patch, you should remember that it's received limited testing by the Server Technologies group but likely hasn't been tested by the E-Business Suite division at all.  So, the onus is on you to verify whether the patch fixes your problem while being otherwise benign to the rest of your Apps environment.

Relax, It's Supported

The reassuring part of all of this is that interim database patches are supported with the E-Business Suite -- even if they're not explicitly certified.  In other words, if you apply an interim database patch to your certified Apps database configuration, both the Server Technologies and EBS Support divisions will do everything possible to debug issues with that patch. 

If the patch triggers other unexpected side effects, we will help you diagnose your issues, find workarounds, or request additional database or E-Business Suite fixes, as needed.

Your Thoughts?

These new Server Technologies support policies have important operational and planning implications for E-Business Suite customers.  You need to evaluate these new policies against your own practices around upgrades to new database releases, patchsets, and interim patches.

This area gets a lot of debate internally at Oracle and you are invited to participate in that dialogue.  Feel free to sound off in the comments, below.  I'll pass on notable feedback to our colleagues in the Server Technologies division, as appropriate.

Related Articles

Comments:

Hi Steven,

Thank you for bringing the Server Technology update to our attention and explaining the EBS database certification process. So far, we haven’t had a case where we are waiting for the EBS database certification and the RDBMS group can’t backport patches in N-1. The EBS database certification was relatively quick. But we did have a couple of cases where we were anxiously awaiting Portal patchset and EBS certifications…

Do the apps certification teams certify interim the AS10g CPU patches? We have to apply all CPU patches including RDBMS, AS10g midtier, infrastructure, identity management to our EBS environments as soon as they are available. Whichever Oracle division does the CPU planning and release note is doing a great job. Applying CPU patches on our project has become a completely planned event. We roll CPU patches into production three weeks after they are published. Because the CPU release note listed which product/version is the terminal one for the current CPU, we can upgrade these products/versions before the next CPU is available (an example in Note: 753340.1 table 33).

It is comforting to hear your confirmation that we will not be left without support. But in reality, it took too long and too much energy to push through a merge patch request, which we found the patch didn’t work for EBS. In a couple of cases, the certified patchset came out before a merge patch request worked out and of course, the patchset yielded a much better result. The point I am trying to convey here is if there is still a room to certify the patchset even faster, considering the resources wasted on both the part of Oracle and customers dealing with individual bug/patch, which are collectively addressed already in a patchset?

My second question is if you can publish dates for these certifications or something that helps planning in this blog? When you set dates, it may help to speed up deliveries. Also, it helps customers and Oracle support analysts to make a call on whether they should request an interim patch or wait for the certified patchset. We really hope to get the release date or timeframe on ATG RUPs because it seems to only support N, not N-1 release. We usually jump right in whenever an AGT RUP is released. It is very difficult to plan both time/resources and budget. The dates (even if you have to put a Disclaimer like Note: 742060.1 did.) would really help us planning for higher efficiencies and lower costs.

Thanks again,
Jennifer.

Posted by Jennifer Chen on February 24, 2009 at 11:18 AM PST #

Hi, Jennifer,

I appreciate your taking the time to provide such detailed feedback. I've forwarded it to our Server Technologies (ST) and CPU teams. The latter will be very pleased to hear that you're finding the regular CPU schedule to ease your planning.

I'll be covering the Fusion Middleware implications for EBS patching in a future article. The short answer is that we certify OracleAS patchsets with the E-Business Suite but we don't certify interim patches for middle-tier components. This subject warrants a more thorough discussion, so watch this blog for that article.

I also appreciate your observation about the resources wasted while chasing interim patches vs patchsets. The Server Technologies team has recently (in the past few months) reorganized the team responsible for this area, and my experience from our latest 11.1.0.7 certifications is that cycle times for EBS-related interim patches have dropped very sharply. I am optimistic that our most-recent experiences will also translate to rapid turnarounds for customer-request interim patches, too.

If that turns out not to be the case, by the way, you're welcome to forward any relevant SR numbers to me for escalation. The ST team is very committed to polishing this process so that we're all happy with it.

As for publishing release dates: I'm afraid that Oracle has strict rules that prohibit me from discussing dates at all. I've written about those rules and what I can and can't say publicly in this article:

Loose Lips Sink Ships -
http://blogs.oracle.com/stevenChan/2007/03/loose_lips_sink_ships.html

ATG RUPs follow the same N and N-1 support convention as other products. For example, for EBS 11i, ATG Support should be able to request new interim patches for both ATG RUP 6 and ATG RUP 5 since ATG RUP 6 is the latest patchset available. If you've had difficulty convincing Support of this, please forward the relevant SR number and I'll investigate.

Regards,
Steven

Posted by Steven Chan on February 25, 2009 at 06:29 AM PST #

I see a discrepancy. According to this note, 10.2.0.2 is NOT a supported DB release for e-Bus, however, according to Certify on metalink, it is, and there are no de-support notifications listed for Apps R12.
So I think Oracle needs to get it's act together and ensure that if they are going to desupport 10.2.0.2 for e-Bus R12, then they need to make a formal announcement using their established de-support mechanism, giving users sufficient time to make the upgrade.
While, this is a great document (and goes into loads of details rather than just state the facts, which is good), it's just a shame that we get different statements about supported releases, depending on which documents we read, which is very confusing for the end user.
Jason

Posted by Jason Lester on March 17, 2009 at 03:23 AM PDT #

Jason,

I agree that Oracle's communications around supported versions leaves room for improvement.

I've passed on your comments verbatim to our Server Technologies management team.

I can understand your frustration when trying to reconcile different documents from different teams. If there's no clear answer, the best bet would be to contact Oracle Support for official clarification.

Can you clarify your reference to "this note"? Following the Server Technologies' guidance, 10.2.0.2 is the "N-2" release, which is no longer supported for any customers.

E-Business Suite customers on 10.2.0.2 should plan on their upgrades to either 10.2.0.4 or even 11.1.0.7 to stay current with the ST Support policies.

Regards,
Steven

Posted by Steven Chan on March 18, 2009 at 04:10 AM PDT #

Excellent article. How do you feel about applying 7612639 - 10.2.0.4 Generic Recommended Patch Bundle #3 to a 10.2.0.4 R12 database?

Posted by Mark on April 04, 2009 at 06:31 AM PDT #

Hi, Mark,

I would generally be in favour of applying patch 76112639 and similar recommended database bundle patches to EBS environments. The great thing about these bundle patches is that the fixes are tested together.

I would definitely apply this if you've encountered database-related performance or stability issues that Oracle's Database Support team says are fixed by this bundle patch.

Regards,
Steven

Posted by Steven Chan on April 06, 2009 at 05:32 AM PDT #

Hi Steven,

I re-read the post and the highlighted sentence:

The important change for E-Business Suite users is that database fixes are now produced for the previous (N-1) patchset for twelve months after the release of the latest (N) patchset.

What this means is once 10.2.0.5 is released, patches for 10.2.0.4 are going to be released for next 12 months only, during which we should upgrade to 10.2.0.5. That sounds a lot better as 10.2.0.5 is not yet announced.

- Vikram

Posted by Vikram Das on April 13, 2009 at 02:57 AM PDT #

Hi Steven,

With the advent of database Patchset Updates, (PSUs), will these be tested / certified with EBS or they regarded as interim patches? PSUs are not mentioned in any of the EBS documentation.

I recently applied the Jan 2010 PSU for 10.2.0.4 and the (now cumulative) Jan 2010 EBS CPU to an 11.5.10.2 + RUP7 instance. Both are major improvements.

Regards,
Stuart.

Posted by Stuart smith on February 08, 2010 at 11:48 AM PST #

Hi, Stuart,

Thanks for your feedback on our latest EBS CPU and the Jan PSU. Glad to hear that these have streamlined your workflow.

For more information about PSUs, see:

Can E-Business Users Apply Database Patch Set Updates?
http://blogs.oracle.com/stevenChan/2009/08/can_ebs_users_apply_database_patch_set_updates.html

Regards,
Steven

Posted by Steven Chan on February 08, 2010 at 11:54 PM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today