On Apps Tier Patching and Support: A Primer for E-Business Suite Users
By Steven Chan - EBS-Oracle on Feb 26, 2009
The Oracle Server Technologies division issued some important updates to their support policies in the following document:
- Database, FMW, EM Grid Control, and OCS Software Error Correction Support Policy (Metalink Note 209768.1)
I discussed the implications of the database-related policy updates for E-Business Suite users in this recently-published article. It's time to discuss the implications of these policy updates for E-Business Suite application tier servers and externally-integrated Oracle Application Server instances. This article also discusses the E-Business Suite application tier certification process and the safety of applying interim Fusion Middleware patches to your Apps environments.
If you've read the previous article about database patching implications, parts of this new article will sound very familiar. But stick with us; the implications for E-Business Suite application tier servers are subtly different and it's important that EBS sysadmins get the whole picture for both tiers.
What Are the Important Changes?
The Server Technologies division periodically produces Fusion Middleware (i.e. Oracle Application Server) patchsets: integrated, cumulative, and fully-tested collections of fixes issued between product releases. These are usually produced one or twice a year.
The Server Technologies division also releases three other types of bug fixes between patch sets:
- Quarterly Critical Patch Updates (security fixes)
- Patch bundles
- An interim patch (sometimes called a "one-off") for non-Windows platforms
Server Technologies provides bug fixes for the latest patchset and the previous patchset (i.e. N and N-1 patchset releases) for a given Fusion Middleware release. The important change for E-Business Suite users is that Oracle Application Server 10g fixes are now produced for the previous (N-1) patchset for twelve months after the release of the latest (N) patchset. This is called the grace period. Unlike database patch sets, the grace periods for Fusion Middleware patchsets are currently the same for all operating system platforms.
The release dates for Fusion Middleware patchsets are listed in the following document:
- Oracle Application Server 10g Release 2 (10.1.2) Support Status and Alerts (Metalink Note 329361.1)
Another important difference from database patchsets is that Fusion Middleware patchsets are available for each Fusion Middleware product family. The product families relevant to E-Business Suite customers are:
- Oracle Application Server 10g version 10.1.2 (includes Portal, Discoverer, Forms & Reports, WebCache)
- Oracle SOA Services 10g version 10.1.3 (includes OC4J, BPEL, BAM)
- Oracle Identity Management 10g version 10.1.4 (includes Single Sign-On and Oracle Internet Directory)
The N and N-1 support policy applies to each of these individual product families or codelines. For example, at the time this article was written, the following Fusion Middleware patchsets were available and supported for each product family:
- Oracle Application Server 10g version 10.1.2 patchsets 10.1.2.3 and 10.1.2.2
- Oracle SOA Services 10g version 10.1.3 patchsets 10.1.3.4 and 10.1.3.3
- Oracle Identity Management 10g version 10.1.4 patchsets 10.1.4.3 and 10.1.4.2
What Is the Impact on EBS Customers?
The E-Business Suite division certifies major Fusion Middleware releases, patchsets, and Critical Patch Updates whenever they're released. These certifications are performed on the released Oracle Application Server 10g code, not the beta internal drops. These certifications take time, of course. If our certification tests find Apps compatibilities issues with a given patchset, we need to loop back to the Server Technologies team to obtain interim patches. The patchset and interim patches are then retested with the E-Business Suite.
Let's look at two of the major Oracle Application Server 10g product families. This illustrates the impact of the new Server Technologies support policy on EBS environments.
For the Oracle Application Server 10g version 10.1.2 codeline:
- 10.1.2.3 was released in April, 2008
- 10.1.2.2's grace period ends in April, 2009
- 10.1.2.3 was certified with Apps 12 in November, 2008
- Apps 12 customers can get patches for 10.1.2.2 for five months
For the Oracle SOA Services 10g version 10.1.3 codeline:
- 10.1.3.4 was released in July, 2008
- 10.1.3.3's grace period ends in July, 2009
- 10.1.3.4 was certified with Apps 12 in December, 2008
- Apps 12 customers can get patches for 10.1.3.3 for seven months
For the Oracle Identity Management 10g version 10.1.4 codeline:
- 10.1.4.3 was released in November, 2008
- 10.1.4.2's grace period ends in November, 2009
- 10.1.4.3 was certified with Apps 11i and 12 in February, 2009
- Apps 11i and 12 customers can get patches for 10.1.4.2 for nine months
The Server Technologies division understands that you coordinate upgrades to your production EBS environments with financial reporting cycles, business events, and other major EBS updates and architectural changes. They understand that your upgrade cycles can often exceed the Fusion Middleware grace periods. In circumstances like these -- i.e. outside of the grace period -- the Server Technologies division has committed to being flexible about approving requests for interim patches for the N-1 release for critical issues affecting your EBS production environments.
Another thing you should consider is that new interim patches will not be produced for older patchsets, e.g. the N-2 release or older. For example, in the case of the Oracle SOA Services 10g version 10.1.3 codeline, this means that while you can still download existing interim patches for, say, 10.1.3.2, no new interim patches will be produced for those older releases.
A prudent sysadmin would conclude that the safest policy is to keep your E-Business Suite environment reasonably up-to-date with Fusion Middleware patchsets. Even if you can't apply the latest patchset available, it's wise to try and keep up with the N-1 patchset, at minimum.
Teasing Apart Support Implications for Internal and External Components
Internal Components: Oracle E-Business Suite Release 12's application tier is built on both the OracleAS 10.1.2 and 10.1.3 product families. The Server Technology division's N and N-1 support policy applies to these Oracle Application Server 10g components running as part of Apps 12's internal techstack. It also applies to external Oracle Application Server 10g instances that are optionally integrated with Apps 12 for the use of Single Sign-On, Oracle Internet Directory, Portal, Discoverer, Web Cache, Web Center, and SOA Suite.
Oracle E-Business Suite Release 11i's internal middle tier is built on the older Oracle9i Application Server 220.127.116.11.2 product family. This product family is officially desupported for all users except Oracle E-Business Suite Release 11i. Apps 11i has a special support extension that covers Oracle9i Application Server 18.104.22.168.2 for the life of Apps 11i.
External Components: Oracle E-Business Suite Release 11i can optionally be integrated with external Oracle Application Server 10g instances for the use of Single Sign-On, Oracle Internet Directory, Portal, Discoverer, and WebCache.
The Server Technology division's N and N-1 support policy applies to these external Oracle Application Server 10g components integrated with Apps 11i.
What's Involved in an EBS Fusion Middleware Certification?
I mentioned that EBS certifications with new Fusion Middleware releases or patchsets take time. Each of the Fusion Middleware product lines have differing impact on the E-Business Suite, depending upon whether they're part of the E-Business Suite's internal techstack (e.g. Forms & Reports 10g embedded within the Oracle E-Business Suite Release 12 technology stack), or whether they're externally integrated (e.g. Discoverer 10g running on a standalone instance).
Depending upon the scope of the Oracle Application Server 10g patchset, these certifications can involve some or all of the following teams: my own certification teams, E-Business Suite product family teams (e.g. Financials, Order Management, Supply Chain), our Architecture teams, our Applications Technology Group (e.g. AOL/J, RapidClone, OA Framework), our Applications Platforms Group, our Applications Performance Group, Oracle Support, and of course, teams within the Server Technologies Division themselves.
This is a bit of a longwinded way of illustrating that when we announce a particular Fusion Middleware certification with the E-Business Suite, it has potentially been tested by many different E-Business Suite products and tools in a common set of environments with a reference product family and technology stack baseline.
Interim Patches Aren't Certified with EBS
The uncomfortable reality of software release cycles means that certain interim Fusion Middleware product family patches will fall outside of that EBS certification process. Our Apps certification teams don't apply or certify interim Oracle Application Server patches unless they're also included in something like the quarterly Critical Patch Updates.
Here's a typical release and certification cycle:
- We certified OC4J 10.1.3.3 with Apps 12 in January, 2008
- We certified OC4J 10.1.3.4 with Apps 12 in December, 2008
- Interim OC4J patches were released to fix specific issues found in 10.1.3.3 between January and December
- Those interim patches were not tested with the E-Business Suite.
Let's say that you upgraded your Apps 12 environment to OC4J 10.1.3.3 in April. In the process of testing some of your EBS extensions, you found a generic OC4J bug. Oracle Support identifies an interim patch that could potentially fix that problem.
This triggers a common dilemma for some Apps sysadmins: you can stay with the certified configuration and live with the problem, or you can add an uncertified database patch to your environment, potentially destabilizing things.
Is It Safe to Apply Interim Fusion Middleware Patches?
Yes, it's generally safe to apply interim Fusion Middleware patches to your E-Business Suite environments. Remember that an E-Business Suite environment is just like any custom application built on Oracle Fusion Middleware, so interim patches don't pose an unusually high risk for EBS environments in particular. Generally speaking, if it's safe for a generic Oracle Fusion Middleware user, it's safe for an E-Business Suite environment, too.
Also remember that interim Fusion Middleware patches are typically small and focused on resolving a specific issue. In other words, the solution will likely not make things any worse than they already are. One can argue that certain interim Fusion Middleware patches, especially those addressing performance optimization or memory management issues, may be more important to apply than some Apps product patches.
That said, there are some important things you should consider:
- Are your reported symptoms identical to the symptoms that the interim patch addresses?
- Is this a critical issue for your environment or users?
- Can you test your environment thoroughly to ensure that there are no other unexpected side-effects of this patch?
If you can't answer "Yes" to all of these questions, it might be worth reflecting on the wisdom of applying that interim patch. For example, if you're experiencing a problem that only partially resembles the symptoms resolved by an interim patch, it's possible that that patch won't fix your problem. If that's the case, it might be worthwhile hunting down a different patch that more-closely targets your problem.
Ultimately, this all boils down to testing. If you're considering applying an interim Fusion Middleware patch, you should remember that it's received limited testing by the Server Technologies group but likely hasn't been tested by the E-Business Suite division at all. So, the onus is on you to verify whether the patch fixes your problem while being otherwise benign to the rest of your Apps environment.
Relax, It's Supported
The reassuring part of all of this is that interim Fusion Middleware patches are supported with the E-Business Suite -- even if they're not explicitly certified. In other words, if you apply an interim Oracle Application Server 10g patch to your certified Apps configuration, both the Server Technologies and EBS Support divisions will do everything possible to debug issues with that patch.
If the patch triggers other unexpected side effects, we will help you diagnose your issues, find workarounds, or request additional database or E-Business Suite fixes, as needed.
These new Server Technologies support policies have important operational and planning implications for E-Business Suite customers. You need to evaluate these new policies against your own practices around upgrades to new Fusion Middleware releases, patchsets, and interim patches.
This area gets a lot of debate internally at Oracle and you are invited to participate in that dialogue. Feel free to sound off in the comments, below. I'll pass on notable feedback to our colleagues in the Server Technologies division, as appropriate.