Thursday Feb 11, 2016

ETCC Tool Enhanced for Finding Mandatory EBS 12.2 Patches

Contributing author: Paul Holman

The E-Business Suite Technology Codelevel Checker (ETCC) tool has been provided to help you identify missing database or middle tier patches that may need to be applied to your E-Business Suite Release 12.2 system.

The utility has now been significantly enhanced to map bug fixes to patches, generating a Patch Recommendation Summary that lists the patches (including versions and associated filenames) that are required for your system.

The following example shows part of an ETCC run, where the DB-ETCC script is used to identify missing database patches.

DB-ETCC Example

Why Does This Matter?

As the last section of the above example shows, there is a difference between bug fixes and patches. A bug fix ID for an issue does not change, but the patch ID that delivers the fix might: for example, the original patch could be superseded by another patch. So you need to not only know what bug fix you require, but which patch currently delivers it.

With previous versions of ETCC, the relevant patches (including Patch Set Updates and Exadata patches) had to be identified manually, by referring to the relevant documentation.  In this latest version, bug fix to patch mapping is done for you, avoiding the need for manual identification and the risk of misidentifying the patch needed. That is the principal enhancement in this version of ETCC.


This new version of ETCC maps missing bug fixes to corresponding patches for the latest and latest but one quarterly bundles supported by EBS 12.2. You can use this version of ETCC with older bundles, but it will only list missing bug fixes (as previous versions did), and you will see a message saying "Patch mapping not available".

Obtaining ETCC

The ETCC utility can be downloaded via Patch 17537119 from My Oracle Support.


Related Articles

Tuesday Dec 08, 2015

Updated: EBS Upgrade Performance Diagnostics White Paper

We've updated our white paper that describes methods and tools for diagnosing performance-related issues with your EBS 12.x upgrades:

This is a companion Note to:

Both Notes are written by by Jim Machin, a senior member of our Applications Performance group. The companion Note describes a framework and tools for obtaining:

  • Actual statistics: So that you can see exactly which execution plan steps are inefficient, rather than those that you think might be inefficient. You can also estimate the likely impact of a performance fix.
  • Diagnostics that are quick and easy to run, so that you can obtain diagnostics on more jobs/SQL, and obtain them faster.
  • Diagnostics that have very little impact on the performance of the Release 12 upgrade: if they can be run during the upgrade, you will get the diagnostics sooner and be able to resolve the issue more quickly.

There is an extensive range of diagnostics available, which include:

  • Identifying the top SQL in cursor cache or AWR
  • Displaying cursor reports for long-running SQL
  • Matching long-running SQL to jobs
  • Reporting on CBO statistics for all EBS tables
  • Reviewing the AD job timing report
  • Identifying long-running upgrade jobs
  • Specific diagnostics for Online Patching enablement performance

This white paper is updated regularly with new tips and recommendations based upon our Application Performance group's work with actual EBS 12.2 customers and their Service Requests.  All EBS 12.2 sysadmins should monitor this white paper regularly.

Related Articles

Monday Dec 07, 2015

Updated: Minimizing EBS 12.2 Upgrade Downtimes White Paper

We have updated our recommendations for minimizing downtimes when upgrading to EBS 12.2:

Written by Jim Machin, a senior member of our Applications Performance group, this white paper contains a wealth of detailed technical recommendations for reducing the amount of time needed for an EBS 12.2 upgrade. 

Events vs. sessions bottlenecks diagram

It covers:

  • Technical preparation and planning activities
  • Diagnostics for monitoring and troubleshooting performance issues
  • Common solutions to performance issues
  • Oracle Applications Tablespace Model (OATM) considerations
  • Methods for purging old data
  • Gathering statistics
  • Detailed recommendations for improving resource management, sizing, and workloads

This white paper is updated regularly with new tips and recommendations based upon our Application Performance group's work with actual EBS 12.2 customers and their Service Requests.  All EBS 12.2 sysadmins should monitor this white paper regularly.

Related Articles

Tuesday Aug 11, 2015

Identifying Missing App Tier and Database Tier Patches for EBS 12.2

The EBS Technology Codelevel Checker (ETCC) utility has been available for some time to assist with identifying any missing database patches needed by E-Business Suite 12.2. The utility has now been improved, most notably to include a new Middle Tier checker tool. The latest version provides two scripts rather than the previous single script. These scripts confirm that you have all the necessary technology stack patches installed on your EBS Release 12.2 system. We highly recommend using this utility to help ensure your Release 12.2 deployment is successful.

The two scripts that make up ETCC are:
  • (checkDBpatch.cmd on Microsoft Windows). This is the Database EBS Technology Codelevel Checker (DB-ETCC), which determines if all the needed bugfixes exist in the specified database ORACLE_HOME. In its previous version, this script was just known as ETCC. As well as being renamed for this release, it has been rewritten for greater ease of use.
  • (checkMTpatch.cmd on Microsoft Windows). This is the Middle Tier EBS Technology Codelevel Checker (MT-ETCC), which determines if all the needed bugfixes exist in the middle (application) tier file system. MT-ETCC is completely new with this release of ETCC, and designed to be used in conjunction with DB-ETCC.

The ETCC utility can be downloaded via Patch 17537119 from My Oracle Support.

Patch 17537119

Usage Tips

Read the README carefully.  As well as installation instructions and basic commands, the README for this patch includes a number of usage scenarios and examples.

Always use the latest version of ETCC. New bugfixes will not be checked by older versions of the utility.

Apply all missing patches.  If ETCC reports a missing bugfix, identify the patch that includes the missing fix by referring to this document:

That note provides detailed instructions on how to find the patch you require.

Related Articles

Wednesday Aug 05, 2015

Using Oracle Database In-Memory with Oracle E-Business Suite

Authors: Andy Tremayne and Deepak Bhatnagar, Oracle Development

Database In-Memory is one of a number of options that can be deployed to address Oracle E-Business Suite performance concerns and scalability requirements without the need for any application changes.

We are pleased to announce the publication of a new white paper that can help with this (including links to patches that you will need to apply):

The paper starts by providing strategic advice and guidelines that help you decide which objects to populate into Database In-Memory (DBIM), and how to size the In-Memory Column Store (IMCS). It provides a list of best practices and explains the complexities and limitations of using DISTRIBUTE/DUPLICATE with Oracle RAC.

Database In-Memory diagram

The examples in this paper take a step away from the classic headline feature of analytical reports that you might have been expecting. Instead, they show how DBIM can be applied to three novel use cases:

  • The Order Organizer Form has a huge number of queryable fields that would all need to be indexed for optimal performance. DBIM provides a 10X improvement in the end-user response time.
  • The Initialize Credit Summaries concurrent program is unusual the optimization is based on an INSERT statement. It is also interesting as the example shows why simply populating objects into the IMCS can result in a performance reduction – in this case spilling to disk. Simply populating the objects into DBIM improved the time by 1.5X; further tuning improved on this to provide an overall gain of 3.5X.
  • The Receiving Transaction Processor had two long-running queries that reduced from 7.8 hrs to 4.85 mins representing about a 100X increase in performance. The overall runtime was about 4X faster.

One of the key aspects in the paper is describing how to use Oracle E-Business Suite Application Affinity with Oracle RAC. It describes how to overcome some of the limitations of Non-Engineered Systems (commodity hardware), and explains how it can also be used to benefit Oracle Engineered Systems.

Other References

The Oracle Database In-Memory blog contains a wealth of generic information, technical details, ideas and news on Oracle Database In-Memory from the author of the Oracle White Paper on Oracle Database In-Memory.

Related Articles

Monday Dec 08, 2014

Last Stop For Oracle Applications Tablespace Migration

The Oracle Applications Tablespace Model (OATM) -- sometimes also called the Oracle E-Business Suite Tablespace Model) -- organizes the EBS database by object type rather than by products.  Prior to OATM, the EBS database had two tablespaces for each product, resulting in hundreds of tablespaces.  OATM contains only twelve locally-managed tablespaces.  This simplifies management and requires fewer operating system files.

Whether you're already using the Oracle Applications Tablespace Model depends on which version of the E-Business Suite you initially installed. If you started on an older EBS release, you have the option of migrating to the Oracle Applications Tablespace Model.

Is OATM mandatory?

No, it's optional, but we recommend that you use OATM.  It's more-efficient and easier to manage.  You can use the OATM Tablespace Migration Utility to migrate to the new model.

Can I use OATM with EBS 12.2?

The OATM Tablespace Migration Utility cannot be used after you enable Online Patching in EBS 12.2.  If you wish to migrate to OATM, you must do so before you enable Online Patching.

If you do not migrate to OATM, you must continue to manage your tablespaces separately after enabling Online Patching.


Friday Nov 21, 2014

Testing Your EBS Environment After Database Upgrades

Customers often ask, "What should I test after I upgrade my EBS database?" Your environment is, by definition, unique to you, with your own architectural configurations, setup data, and customizations.  You should always concentrate your testing efforts on the areas that may differ from our generic EBS tests.  Until now, our generic common-sense answer has always been, "Test what is important to your business."  

Our Quality Assurance group in EBS Development has just published some guidance on the things that you might consider adding to your post-upgrade test plans:

They discuss things such as memory-intensive concurrent requests, compliance with new standards for initialization parameters, custom workflows, and more.

Your feedback is welcome.  I'll make sure that the Note's authors are notified if you drop me an email or post a comment here. 

Related Articles

Wednesday Oct 22, 2014

Updated Case Study: Oracle's Own E-Business Suite 12 Environment

Oracle's own use of the E-Business Suite has always been an excellent case study. Our internal EBS environment has grown and evolved over the years. We were running EBS 11i in 2006 with a 6 Terabyte database, after consolidating over 70 EBS instances into a Global Single Instance.  It grew to a 14 Terabyte EBS 12 database by 2008.

Larry Klein, Vice President of Oracle's Enterprise IT group, gave a snapshot of the latest statistics and architecture in a fascinating session at OpenWorld 2014:

Architecture diagram of Oracle's internal EBS 12.1.3 Global Single Instance

Our EBS 12.1.3 database has grown to over 37 TB with over 91 billion rows of data:

Table row counts for Oracle's GSI EBS database

We're running our E-Business Suite database tier on an Exadata Oracle SuperCluster M6-32, the application tier on an Exalogic Elastic Cloud x3-2, ZFS for shared file system storage, and we're using Exadata X4-2 Storage Cells for storage. 

Hardware and software used in Oracle's E-Business Suite instance 2014

Related Articles

Tuesday Nov 05, 2013

Securing Flexfield Value Sets in EBS 12.2

Separation of Duties in Flexfield Value Set Security

Release 12.2 includes a new feature: flexfield value set security.

This new feature gives you additional options for ensuring that different administrators have non-overlapping responsibilities, which in turn provides checks and balances for sensitive activities.  Separation of Duties (SoD) is one of the key concepts of internal controls and is a requirement for many regulations including:

  • Sarbanes-Oxley (SOX) Act
  • Health Insurance Portability and Accountability Act (HIPAA)
  • European Union Data Protection Directive.
Its primary intent is to put barriers in place to prevent fraud or theft by an individual acting alone. Implementing Separation of Duties requires minimizing the possibility that users could modify data across application functions where the users should not normally have access.

For flexfields and report parameters in Oracle E-Business Suite, values in value sets can affect functionality such as the rollup of accounting data, job grades used at a company, and so on. Controlling access to the creation or modification of value set values can be an important piece of implementing Separation of Duties in an organization.

New Flexfield Value Set Security feature

Flexfield value set security allows system administrators to restrict users from viewing, adding or updating values in specific value sets. Value set security enables role-based separation of duties for key flexfields, descriptive flexfields, and report parameters. For example, you can set up value set security such that certain users can view or insert values for any value set used by the Accounting Flexfield but no other value sets, while other users can view and update values for value sets used for any flexfields in Oracle HRMS. You can also segregate access by Operating Unit as well as by role or responsibility.

Value set security uses a combination of data security and role-based access control in Oracle User Management. Flexfield value set security provides a level of security that is different from the previously-existing and similarly-named features in Oracle E-Business Suite:

  • Function security controls whether a user has access to a specific page or form, as well as what operations the user can do in that screen.
  • Flexfield value security controls what values a user can enter into a flexfield segment or report parameter (by responsibility) during routine data entry in many transaction screens across Oracle E-Business Suite.
  • Flexfield value set security (this feature, new in Release 12.2) controls who can view, insert, or update values for a particular value set (by flexfield, report, or value set) in the Segment Values form (FNDFFMSV).
The effect of flexfield value set security is that a user of the Segment Values form will only be able to view those value sets for which the user has been granted access. Further, the user will be able to insert or update/disable values in that value set if the user has been granted privileges to do so.  Flexfield value set security affects independent, dependent, and certain table-validated value sets for flexfields and report parameters.

Initial State of the Feature upon Upgrade

Because this is a new security feature, it is turned on by default.  When you initially install or upgrade to Release 12.2.2, no users are allowed to view, insert or update any value set values (users may even think that their values are missing or invalid because they cannot see the values).  You must explicitly set up access for specific users by enabling appropriate grants and roles for those users.

We recommend using flexfield value set security as part of a comprehensive Separation of Duties strategy. However, if you choose not to implement flexfield value set security upon upgrading to or installing Release 12.2, you can enable backwards compatibility--users can access any value sets if they have access to the Values form--after you upgrade.

The feature does not affect day-to-day transactions that use flexfields.  However, you must either set up specific grants and roles or enable backwards compatibility before users can create new values or update or disable existing values.

For more information, see:

Wednesday Oct 30, 2013

Getting Optimal Performance from Oracle E-Business Suite

Performance tuning and optimization in E-Business Suite environments can involve many different components and diagnostic tools.  Samer Barakat, Senior Architect in our Applications Performance group, held an OpenWorld 2013 session that covered:

  • Performance triage, analysis and diagnostic tools
  • Optimizing the E-Business Suite application tier, including Concurrent Manager
  • Optimizing the E-Business Suite database tier
  • Optimizing the E-Business Suite on Real Application Clusters (RAC)
  • E-Business Suite on engineered systems, including Exadata and Exalogic
  • Optimizing E-Business Suite data management, including archiving and purging 

Table showing workload management strategy for Concurrent Manager

The Applications Performance group works with the world's largest E-Business Suite customers to isolate and resolve performance bottlenecks. This team has helped tune the E-Business Suite environments of world's largest companies to handle staggering amounts of transactional volume in multi-terabyte databases.  This group also publishes our official Oracle Apps benchmarks, white papers, and performance metrics.

This is an essential set of tips and techniques that all EBS sysadmins and DBAs can use to improve the performance of their environments:

OpenWorld 2013 presentations are only available for approximately six months -- until ~March 2014.  Download this one while it's still available.

Related Articles

Wednesday Oct 23, 2013

New Whitepaper: Best Practices for Gathering EBS Database Statistics

Most Oracle Applications DBAs and E-Business Suite users understand the importance of accurate database statistics.  Missing, stale or skewed statistics can adversely affect performance. 

Oracle E-Business Suite statistics should only be gathered using FND_STATS or the Gather Statistics concurrent request. Gathering statistics with DBMS_STATS or the desupported ANALYZE command may result in suboptimal executions plans for E-Business Suite.

Our E-Business Suite Performance Team has been busy implementing and testing new features for gathering statistics using FND_STATS in Oracle E-Business Suite databases.  The new features and guidelines for when and how to gather statistics are published in the following whitepaper:

The new white paper, written by Deepak Bhatnagar, Mohammed Saleem, and Andy Tremayne, details the following options for gathering statistics using FND_STATS and the Gather Statistics concurrent request::

  • History Mode - backup existing statistics prior to gather new statistics
  • GATHER_AUTO Option - gather statistics for tables based upon % change
  • Histograms - collect statistics for histograms
  • AUTO Sampling - use the new FND_STATS feature that supports the AUTO option for using AUTO sample size
  • Extended Statistics - use the new FND_STATS feature that supports the creation of column groups and automatic statistics collection on the column groups when table statistics are gathered
  • Incremental Statistics - gather incremental statistics for partitioned tables

The new white paper also includes examples and performance test cases for the following:

  • Extended Optimizer Statistics
  • Incremental Statistics Gathering
  • Concurrent Statistics Gathering

This white paper includes details about the standalone Oracle E-Business Suite Release 11i and 12 patches that are required to take advantage of this new functionality.

Your feedback is welcome

We would be very interested in hearing about your experiences with these new options for gathering statistics.  Please feel free to post your comments here or drop us a line privately.

Related Oracle OpenWorld 2013 Session

Related My Oracle Support Notes

Non-EBS Related Blogs, White Papers and My Oracle Support Notes 

Tuesday Aug 06, 2013

You're Closer to EBS 12.1.3 Than You Think

Congratulations -- you're running EBS 12.1.1 or 12.1.2 in production.  Our standing recommendation is to apply the EBS 12.1.3 suite-wide Release Update Pack (RUP), despite the relaxed requirements in the recent changes to our Error Correction Support Policy.

If you can't apply the suite-wide 12.1.3 Release Update Pack, you should apply the individual 12.1.3 Family Packs for the EBS products that you've deployed.

Here's our best-kept secret:  this will change your environment less than you think.  If you've applied any patches to your 12.1.1 or 12.1.2 environment, you probably already have much of 12.1.3 already in production today.  

A glimpse behind the curtain

This is due to the way that our EBS source code control system works: we deliver patches based upon the latest versions of code in our source control system.  Developers sometimes refer to this as "delivering code on the tip of the codeline."

If "source control systems" are unknown territory to you, here's an example to illustrate this:

Let's imagine that you're running 12.1.1 and report an issue with a Financials screen. 

That Financials screen was first released in 12.1.1, updated in 12.1.2, and updated again in 12.1.3.

Whatever patch you apply will deliver the latest version of the 12.1.3 code and all of the related dependencies in other screens or packages

In this particular case, you might think you're on 12.1.1, but applying that patch has raised your code level for the updated files to the 12.1.3 level or even later.

Apps DBA Tip: Check which files a patch changes

Here's how to check this out yourself.  The Oracle Applications Manager contains a tool that all Apps DBAs should use:

The Patch Wizard has an "Analyze Specific Patches" function that tells you how a specific patch will affect your system.

Patch Wizard Analyse Specific Patches screenshot

The resulting "Patch Impact Analysis" report provides you with a summary of:

  • Applications patched
  • New files introduced
  • Existing files changed
  • Existing files unchanged

Screenshot of Patch Wizard Impact Analysis Report

Power users tip:  identify affected customizations

Patch Wizard Register Flagged Files screenshotIf you've flagged your customized files using the Oracle Applications Manager "Register Flagged Files" function, the Patch Wizard will also identify which of your customizations are affected by a given patch.

This allows you to reassure your developers and end-users that you have very-granular control over the impact of a given patch on critical customized functionality.

A case study: Financials and Procurement 12.1.3 vs. 12.1.1

A major steel manufacturer running EBS 12.1.1 approached us with their concerns about applying the suite-wide EBS 12.1.3 Release Update Pack to their environment.  Like many manufacturers, every hour of EBS downtime meant lost productivity so they wanted to minimize the amount of retesting triggered by 12.1.3.

They had been running 12.1.1 Financials and Procurement in production for quite some time.  They had stabilized their environment with a number of one-off patches on top of 12.1.1.  

We used the Patch Wizard's "Patch Impact Analysis" report to assess the number of files that would be affected by the 12.1.3 Family Packs for Financials and Procurement.  Here's what we found:

  • They already had applied 81% of the 12.1.3 Procurement Family Pack.  The 12.1.3 Procurement Family Pack has ~3,800 files, which means that over 3,100 files were unchanged.  Only 3 of the files that they'd customized needed review.

  • The 12.1.3 Financials Family Pack contains ~12,000 files.  They had already applied 72% of the 12.1.3 Financials Family Pack, leaving over 8,600 files unchanged.  Only one of their customized files needed review.

Do your own 12.1.3 impact analysis

Your mileage will vary.  Run the Patch Wizard comparing the 12.1.3 Family Packs to see how close you are.

I'd be interested in hearing your reports.  Feel free to post a comment here or drop me an email with your results.


Related Articles

Thursday Aug 01, 2013

New Source Database Added for EBS 12 + 11gR2 Transportable Tablespaces

The Transportable Tablespaces (TTS) process was originally certified for the migration of E-Business Suite R12 databases going from a source database of 11gR1 or 11gR2 to a target of 11gR2.

This requirement has now been expanded to include a source database of 10gR2 ( - this will potentially save time for existing 10gR2 customers as they can remove on a crucial upgrade step prior to performing the platform migration.

Transportable Tablespaces supported options

The migration process requires an updated Controlled patch delivered by the Oracle E-Business Suite Platform Engineering team, i.e. it requires a password obtainable from Oracle Support. We released the patch in this manner to gauge uptake, and help identify and monitor any customer issues due to the nature of this technology. This patch has been updated to now include supporting 10gR2 as a source database.

Does it meet your requirements?

Note that for migration across platforms of the same "endian" format, users are advised to use the Transportable Database (TDB) migration process instead for large databases. The "endian-ness" target platforms can be verified by querying the view V$DB_TRANSPORTABLE_PLATFORM using SQL*Plus (connected as sysdba) on the source platform:

SQL>select platform_name from v$db_transportable_platform;

If the intended target platform does not appear in the output, it means that it is of a different endian format from the source. Consequently. database migration will need to be performed via Transportable Tablespaces (for large databases) or export/import.

The use of Transportable Tablespaces can greatly speed up the migration of the data portion of the database. However, it does not affect metadata, which must still be migrated using export/import. We recommend that users initially perform a test migration on their database, using export/import with the 'metrics=y' parameter. This will help identify the relative amounts of data and metadata, and provide a basis for assessing likely gains in timing. In general, the larger the amount of data (compared to metadata), the greater the reduction in downtime that can be expected from using TTS as a migration process.

For smaller databases or for those that have relatively small data compared to metadata, TTS will not be as beneficial for cross endian migration and the use of export/import (datapump) for the whole database is recommended.

Where can I find more information?

Related Articles

Friday Aug 17, 2012

Top Five Errors Customers Make When Maintaining E-Business Suite 12 (Part 6)

This is the last in a series of articles that list a few of the technical myths or misunderstandings about the E-Business Suite. Previous articles have covered installations, cloning, patching, migrations, and upgrades. This article discusses general maintenance.

Each article is absolutely not definitive and I’d be interested in comments from readers who think there are other misconceptions about the day to day maintenance and upgrade of a typical E-Business Suite environment.

COMMON ERROR #1: Manually editing files that are maintained by AutoConfig.

ORACLE’S RECOMMENDATION: When you run AutoConfig, in most circumstances any manual changes to AutoConfig controlled files will be overwritten. There is a process for adding custom entries to AutoConfig maintained files; you should use this process if you need to make manual changes to files maintained by AutoConfig. Refer to Section 4 of the AutoConfig Note for information on how to customise AutoConfig.

COMMON ERROR #2: Maintaining local file systems for multiple application tier servers instead of shared file systems.

ORACLE’S RECOMMENDATION: An environment with, for example, four standalone web/forms nodes on Linux and four standalone Concurrent Processing / RAC nodes on Solaris each using a local file system is a very inefficient architecture that you will very quickly get bored of patching.  Each E-Business Suite patch will need to be applied eight times. This can be reduced to one patch session if Concurrent Processing is placed on the same node(s) as web/forms and a shared application tier file system is introduced. If you cannot achieve acceptable performance from your shared disk subsystem then you should investigate the performance issue on your disk subsystem rather than compromising with an E-Business Suite architecture that has such expensive maintenance requirements.

COMMON ERROR #3:  Sizing everything in the init.ora as large as possible.

ORACLE’S RECOMMENDATION: Database tuning is a delicate exercise. Indiscriminately increasing values in the hopes that doing so will automatically improve performance is a dangerous approach. Use the performance tools in Oracle Enterprise Manager (OEM) and other tools to establish performance bottlenecks. Do not change init.ora parameters indiscriminately; all changes should be supported by your own empirical evidence that candidate changes result in performance improvements. Thoroughly stress-test all changes before implementing in production.

COMMON ERROR #4:  Ignoring invalid objects for products you do not use.

ORACLE’S RECOMMENDATION:  All products are installed in an E-Business Suite installation in the database.  All products must be patched and maintained. Invalid objects, even in products you do not use, indicate an underlying problem in your system that should always be investigated and resolved.

COMMON ERROR #5:  Performing a complete database export/import to improve performance.

ORACLE’S RECOMMENDATION: Well possibly, but there is certainly a lot of unnecessary exporting and importing of databases in an attempt to improve performance.  This brings us nicely back to the introduction of our first article in this series. A significant amount of the data in your database is static. Tables may become fragmented but this is not automatically a performance problem. If this was the case, many databases would have performance problems.

An Oracle database is built to cope with a reasonable amount of fragmentation and will not suffer because of it. Analyse performance problems before assuming the issue will be solved by an export/import. Most performance problems are due to poor indexing of tables and inefficient SQL. Few are caused by table fragmentation.


Related Articles

Wednesday Aug 15, 2012

Five Errors Customers Make When Upgrading E-Business Suite 12 (Part 5)

This is the fifth in a series of articles about common technical myths or misunderstandings about the E-Business Suite. Other articles have covered installations, cloning, patching, migrations, and maintenance. This article discusses upgrading. 

Each article is absolutely not definitive and I’d be interested in comments from readers who think there are other misconceptions about the day to day maintenance and upgrade of a typical E-Business Suite environment.

COMMON ERROR #1: Believing that DBUA does not work for Apps databases.

ORACLE’S RECOMMENDATION: Many DBAs still like to perform database upgrades manually using Oracle-provided scripts as they feel it gives them more control of the upgrade process. For the rest of us, DBUA is an excellent tool that greatly simplifies the upgrade process and which should reduce your upgrade time. E-Business Suite database upgrade documentation now assumes (but does not mandate) that you will use DBUA and documentation will be written accordingly. Raise a Service Request if you believe you have an issue where DBUA fails and the manual process is required instead.

COMMON ERROR #2: Assuming that only one test upgrade is needed before the production upgrade.

ORACLE’S RECOMMENDATION: Oracle advise a minimum of three completely successful tests of any upgrade or maintenance downtime before your production upgrade.

COMMON ERROR #3: Using generic documentation instead of the E-Business Suite versions.

ORACLE’S RECOMMENDATION: As an example, if you need to upgrade your database do not just refer to the standard database upgrade manuals. The E-Business Suite Development Team have created documentation specifically for upgrading the E-Business Suite technology stack to supplement generic documentation. Follow these supplemental documents. They are well written, thoroughly tested and will make your upgrade a success.

COMMON ERROR #4: Skipping new products included in Release Update Packs (RUPs) because there are no plans to use that product.

ORACLE’S RECOMMENDATION: When instructed to add new products to your database as part of a RUP you must always add those products. These products will later be patched and updated by this or later RUPs and if the products are not present then the patches will fail.

COMMON ERROR #5: Manually copying files between different E-Business Suite environments.

ORACLE’S RECOMMENDATION: New or replacement files should only be introduced to your system by patching, scheduled upgrades or other documented methods. Manually copying files between systems can cause mismatches between files in the APPL_TOP and their corresponding entries in database AD tables. This could cause patching issues later.

Related Articles

Friday Jul 27, 2012

Five Errors Customers Make When Migrating E-Business Suite 12 (Part 4)

This is the fourth in a series of articles about common technical myths or misunderstandings about the E-Business Suite. Other articles have covered installations, cloning, patching, upgrades, and maintenance. This article discusses migrations.

Each article is absolutely not definitive and I’d be interested in comments from readers who think there are other misconceptions about the day to day maintenance and upgrade of a typical E-Business Suite environment.

COMMON ERROR #1: Exporting from a 9i database and importing into a fresh 10g/11g database.

ORACLE’S RECOMMENDATION: When working with 10g onwards you must use datapump. Refer to the opening section of the relevant import/export Note to establish database version compatibility for exports/imports (including datapump). When importing into a 10g or 11g database the source database must be 10g or later.

COMMON ERROR #2: Assuming that Linux x86_64 is a certified application tier platform for 11i.

ORACLE’S RECOMMENDATION: Linux x86_64 is only certified for the 11i database tier of the E-Business Suite. Linux x86_64 can be used as both an application and database tier platform with Release 12 onwards. There are no plans to certify Linux x86_64 as a platform for the 11i application tier. Linux x86_64 using any form of 32 bit emulation is also not a certified 11i application tier platform.

COMMON ERROR #3: Migrating the E-Business Suite environment to a new platform by performing a fresh install of the E-Business Suite using Rapidwiz on the new platform and then performing an export/import of the database.

ORACLE’S RECOMMENDATION: You must use the migration tools documented in the link below to migrate the application tier to the new platform. Only use Rapidwiz to create the new application tier technology stack.

COMMON ERROR #4: Migrating EBS 11i application tier servers to Exalogic servers.

ORACLE’S RECOMMENDATION: The Exadata platform is certified for use as a database tier for both 11i and R12. Both 11i and R12 require Oracle Database 11gR2. The Exalogic platform can be used as an application tier for R12 but not for 11i.

COMMON ERROR #5: Installing or upgrading your E-Business Suite technology stack components requires downloading the entire R12 installation media.

ORACLE’S RECOMMENDATION: It is possible to build a staging area that only contains the components you require. Refer to the E-Business Suite Installation Guide for instructions on how to build a partial staging area that contains only the technology stack or E-Business Suite components that you require.


Related Articles

Friday Jul 20, 2012

Five Errors Customers Make When Patching E-Business Suite 12 (Part 3)

This is the third in a series of short articles about common technical myths or misunderstandings about the E-Business Suite. Other articles have covered installations, cloning, migrations, upgrades, and maintenance. This article discusses patching. Subsequent articles will cover migrating, upgrading, and general maintenance. Each article is absolutely not definitive and I’d be interested in comments from readers who think there are other misconceptions about the day to day maintenance and upgrade of a typical E-Business Suite environment.

COMMON ERROR #1: Assuming that all patches can be applied using the hotpatch and nocheckfile parameters.

ORACLE’S RECOMMENDATION: The adpatch ‘hotpatch’ parameter should be used very sparingly and is not intended to be an easy way of applying patches to your system without shutting down the application tier services. Most patches should only be applied during scheduled downtimes when all users are logged off the system, application tier services have been shut down and Maintenance Mode enabled. From the adpatch documentation:

“If an emergency patch can be applied without shutting down services, the customized instructions generated by PAA will explicitly state so. If the customized instructions do not explicitly state this, assume that you need to shut down services and enable Maintenance mode before applying the patch.”

Similarly, the ‘nocheckfile’ parameter should not be used on all patches. Nocheckfile forces the patch to run all jobs in a patch regardless of whether the jobs have been run before. It degrades patch performance and should only be used in specific circumstances where it is genuinely needed. It should not be used as the default option for applying all patches. There are very few cases in which you would want to run with nocheckfile.

COMMON ERROR #2: Assuming that patches can be applied “manually” or backed out of “manually”.

ORACLE’S RECOMMENDATION: There is no manual alternative to using adpatch to apply a patch. It is not supported to selectively choose only certain files from a patch and manually copy or execute these files in your E-Business Suite environment. Similarly, there is no simple method of reversing a patch.

COMMON ERROR #3: Increasing the number of patch workers in an attempt to make patches run faster.

ORACLE’S RECOMMENDATION: Do not assume more workers automatically improves patch performance. When you first test a patch, accept the default number of workers. Use this as the performance benchmark. If you think a different number of workers will improve patch performance then try this once you have your default benchmark, but do not exceed the default number of workers without good justification for doing so. Once a patch is started, the number of workers cannot be changed mid-patch.

COMMON ERROR #4: Ignoring or skipping patch workers that fail on products that are not used or needed.

ORACLE’S RECOMMENDATION: You should aim to have zero errors when applying a patch. All E-Business Suite products are installed in the E-Business Suite database whether you are licenced to use them or not. Patches will update all products in the database whether they are licenced or not. If you find yourself having to skip jobs or ignore errors during patching then this indicates an underlying problem in your system that should always be investigated and addressed. Review the patch prerequisites and ensure all steps have been completed successfully.

COMMON ERROR #5: Using adpatch default log file names during patching.

ORACLE’S RECOMMENDATION: Identify each new patch log file with a unique patch log file name. If you always choose the default log file name in adpatch, your adpatch.log quickly becomes enormous. This makes it very difficult to identify individual patch information in the log file and makes it difficult to keep an accurate record of your patch history. Use a simple naming convention, e.g. choose a unique adpatch log name for each patch (<patchnumber>.log. Archive and then delete your main patch log and worker files after each patch so each patch has new log and worker files.


Related Articles

Monday Jul 16, 2012

Five Errors Customers Make When Cloning E-Business Suite 12 (Part 2)

This is the second in a series of articles about common technical myths or misunderstandings about the E-Business Suite. Other articles have covered installations, patching, migrations, upgrades, and maintenance. This article discusses cloning. Subsequent articles will cover patching, migrating, upgrading, and general maintenance.

Each article is absolutely not definitive and I’d be interested in comments from readers who think there are other misconceptions about the day-to-day maintenance and upgrade of a typical E-Business Suite environment.

: Assuming that use of the Rapid Clone utility is optional when cloning your Apps environment.

ORACLE’S RECOMMENDATION: Rapid Clone greatly simplifies the process of cloning your E-Business Suite environment. Simply copying your E-Business Suite environment to new host machines, editing the context files to reflect the new environment and then running AutoConfig is not a supported cloning method. Devising your own cloning methodology that bypasses Rapid Clone is a dangerous strategy. If Rapid Clone does not work for you, then raise a Service Request and ask for help in fixing the issue. Do not create an alternative cloning method that you hope will sidestep whatever issue you are seeing.

: Assuming that the preclone scripts only need to be run once on an environment.

ORACLE’S RECOMMENDATION: You must run the preclone scripts immediately before each clone, prior to shutting down the application and database tiers in order to copy the files for cloning (or RMAN backup). It’s also good practice to remove or rename the database tier $ORACLE_HOME/appsutil/clone and application tier $COMMON_TOP/clone before running the preclone script. Always check the cloning documentation for the latest patches before each clone.

: Mistakenly using Rapid Clone to migrate from 32 bit Linux to 64 bit Linux.

ORACLE’S RECOMMENDATION: Although these platforms are binary compatible, cloning your E-Business Suite from 32 bit Linux to 64 bit Linux will only produce a 32 bit E-Business Suite environment on your 64 bit operating system. In order to fully exploit your 64 bit operating system you should follow specific migration documentation for your E-Business Suite environment. The same applies to other certified operating systems available as both 32 and 64 bit.

:  Mistakingly forgetting to update the central inventory when overwriting a previous clone with a new clone in the same location.

ORACLE’S RECOMMENDATION: If you regularly repeat a clone to the same location on a target node, you must ensure the central inventory entries for the previous clone are removed before configuring the new clone. As with a repeated installation, when deleting an E-Business Suite environment prior to cloning to the same location, in addition to shutting down and deleting files, you must also remove its entries from the central inventory. The central inventory is a file based repository of all Oracle homes installed on a node. This central inventory may exist outside the E-Business Suite file system and is not always removed when you remove the E-Business Suite file system. If the central inventory contains entries for previously deleted E-Business Suite environments then subsequent new clones may fail.

COMMON ERROR #5: Assuming that the clone is not a completely accurate copy.

ORACLE’S RECOMMENDATION: A system created using Rapid Clone is the most genuine and supported copy you can create. It has an identical database, application tier and technology stack as the source environment it was created from. The only significant difference between a source and target clone environment is the data specific to the new hardware (node names, directory locations etc.). It is supported to clone your production environment, apply patches to the clone and then clone the patched system back to become your new production environment. Assuming the cloning documentation is followed correctly, an E-Business Suite environment does not degrade through being repeatedly cloned.


Related Articles

Friday Jul 13, 2012

Five Errors Customers Make When Installing E-Business Suite 12 (Part 1)

One customer recently asked if we could supply documentation and scripts to remove all the E-Business Suite objects from their database leaving them with an empty database. After a few questions, it became apparent they were concerned about performance and a third party consulting organisation had advised the best solution to performance problems was to export and then import the whole database.

Whilst it’s true an export/import might solve performance problems, there are many other things you can do before you conclude the best solution is an export/import; it’s certainly not the default solution. That’s a pretty extreme example but there are quite a few myths and misunderstandings which seem to crop up regularly in Service Requests.

This is the first in a series of short articles that lists a few of the technical myths or misunderstandings about Oracle E-Business Suite. Other articles have covered cloning, patching, migrations, upgrades, and maintenance. This first article discusses installations.

Each article is absolutely not definitive and I’d be particularly interested in comments from readers who think there are other misconceptions about the day to day maintenance and upgrade of a typical E-Business Suite environment.

COMMON ERROR #1: In a global implementation, putting your database tier host in your global data centre and putting local application tier hosts in each country.

ORACLE’S RECOMMENDATION: You must put all host nodes of your E-Business Suite environment in one data centre with the fastest possible network connection between nodes. This applies regardless of the location of your users.

COMMON ERROR #2: Placing concurrent processing services and the database tier on the same node to improve performance.

ORACLE’S RECOMMENDATION: There is no significant performance benefit to placing concurrent processing and the database tier on the same node. Your system will be more scalable if your database tier hardware is dedicated solely to serving the needs of the database. Patching, cloning and E-Business Suite administration will also be much easier as you will have at least one less APPL_TOP and application tier technology stack to maintain. Placing concurrent processing on the database tier node to distribute load might suggest that your application tier hardware is already undersized. You should place concurrent processing on the same nodes as your web/forms services or create a dedicated concurrent processing tier in exceptionally busy environments.

COMMON ERROR #3: Installing only the default database character set although you know that you will need a different character set later.

ORACLE’S RECOMMENDATION: Failing to consider national language support (NLS) and database character set requirements when installing a new environment is a critical mistake. You should normally have a very good idea of language requirements before you start your installation. Implement the correct languages and database character set when you perform your installation. Characters sets can be changed later and languages can also be added but if you think you are going to need them then implement them early in your project. Realising that you have forgotten the purchasing team in Denmark the day before you go live will not make you a popular DBA.

You cannot specify a different character set when upgrading to R12. The R12 upgrade process cannot do a character set conversion.

COMMON ERROR #4: Separating Oracle database and application tier ownership by different operating system users/groups.

ORACLE’S RECOMMENDATION: In simple single node installations the whole E-Business Suite file system (including the database) can be owned by a single user/group. This can simplify installation and maintenance. Oracle documentation and Oracle Support engineers sometimes refer to the ‘applmgr’ user and the ‘oracle’ user. These are generic terms used loosely to describe the operating system owner of the application and database tier file systems respectively – they are not mandatory user names. It is also not mandatory to perform E-Business Suite installations as the root user.

COMMON ERROR #5: Before retrying a failed installation, forgetting to remove its entries from the central inventory.

ORACLE’S RECOMMENDATION: When deleting an E-Business Suite environment, in addition to shutting down and deleting files, you must also remove its entries from the central inventory. The central inventory is a file based repository of all Oracle homes installed on a node. This central inventory may exist outside the E-Business Suite file system and is not always removed when you remove the E-Business Suite file system. If the central inventory contains entries for previously deleted E-Business Suite environments then subsequent new installations may fail.


Related Articles

Wednesday Mar 21, 2012

ATG Live Webcast March 22 Reminder: Network, WAN, and PC Performance Tuning (Performance Series Part 3 of 3)

A quick reminder about tomorrow's webcast -- Thursday, March 22:  Andy Tremayne, Senior Architect, Applications Performance, and co-author of Oracle Applications Performance Tuning Handbook from Oracle Press, and Uday Moogala, Senior Principal Engineer, Applications Performance, will discuss network performance for E-Business Suite. Andy and Uday will cover tuning the client and tuning the network. They will share real-life examples of network performance, and show you tools and techniques that you can use to estimate or simulate performance on your own network.

The agenda for the Performance Tuning - Part 3 of 3 webcast includes the following topics:
  • Tuning the Client
  • Tuning the Network
Date:               Thursday, March 22, 2012
Time:              8:00 AM - 9:00 AM Pacific Standard Time
Presenters:  Andy Tremayne, Senior Architect, Applications Performance
                        Uday Moogala, Senior Principal Engineer, Applications Performance

Webcast Registration Link (Preregistration is optional but encouraged)

To hear the audio feed:
   Domestic Participant Dial-In Number:           877-697-8128
    International Participant Dial-In Number:      706-634-9568
    Additional International Dial-In Numbers Link:
    Dial-In Passcode:                                              99341

To see the presentation:
    The Direct Access Web Conference details are:
    Website URL:
    Meeting Number:  591264961

If you miss the webcast, or you have missed any webcast, don't worry -- we'll post links to the recording as soon as it's available from Oracle University.  You can monitor this blog for pointers to the replay. And, you can find our archive of our past webcasts and training here.

Monday Mar 12, 2012

ATG Live Webcast March 22: Network, WAN, and PC Performance Tuning (Part 3 of 3)

Some E-Business Suite customers have end-users who connect over a wide-area network (WAN) from remote locations to a central data center.  These remote locations may have very low bandwidth connections.  It is possible to optimise and tune the E-Business Suite to work well even with very narrow pipes. 

Andy Tremayne, co-author of Oracle Applications Performance Tuning Handbook from Oracle Press, has worked with E-Business Suite worldwide on these types of EBS tuning projects.  Andy is presenting the next installment of our ATG Live Webcast series on Mar. 22, 2012.
Network, WAN, and PC Performance Tuning (Part 3 of 3)

Please join Andy Tremayne, Senior Architect, Applications Performance, as he discusses network performance for E-Business Suite. Andy will cover tuning the client and tuning the network. He will share real-life examples of network performance, and he will show you tools and techniques that you can use to estimate or simulate performance on your own network.

Screenshot of Network Test page within E-Business Suite 12.1

The agenda for the Network, WAN, and PC Performance Tuning webcast includes the following topics:

  • Tuning the Client
  • Tuning the Network

Date:               Thursday, March 22, 2012
Time:              8:00 AM - 9:00 AM Pacific Standard Time
Presenter:    Andy Tremayne, Senior Architect, Applications Performance

Webcast Registration Link (Preregistration is optional but encouraged)

To hear the audio feed:
    Domestic Participant Dial-In Number:           877-697-8128
    International Participant Dial-In Number:      706-634-9568
    Additional International Dial-In Numbers Link:
    Dial-In Passcode:                                              99341

To see the presentation:
    The Direct Access Web Conference details are:
    Website URL:
    Meeting Number:  591264961

If you miss the webcast, or you have missed any webcast, don't worry -- we'll post links to the recording as soon as it's available from Oracle University.  You can monitor this blog for pointers to the replay. And, you can find our archive of our past webcasts and training here.

If you have any questions or comments, feel free to email Bill Sawyer (Senior Manager, Applications Technology Curriculum) at BilldotSawyer-AT-Oracle-DOT-com. 

Tuesday Feb 07, 2012

Best Practices for Combining EBS Upgrades with Platform Migrations

Odds are that you're planning an EBS upgrade soon.  EBS 11i Extended Support runs out in 2013, and EBS 12.0 Premier Support ended in January 2012. You might need to combine your EBS upgrade with a operating system upgrade or hardware migration for your servers, too.

Projects that combine EBS upgrades with migrations pose some interesting questions.  What tools do you use if you're...:

  • ... upgrading to a newer version of your current operating system?
  • ... migrating to a different platform with the same endian format?
  • ... migrating to a different platform with a different endian format?

Diagram showing small endian and big endian platforms

Evaluating the different options

I have published a new whitepaper that provides a comprehensive evaluation of the different mechanisms available in upgrading the Oracle E-Business Suite while considering platform migration. This document is meant as an overview to supplement existing detailed documentation that outlines specific processes to perform the migration:

This whitepaper discusses the use of Transportable Database, Recovery Manager (rman), Export/Import using Datapump, Transportable Tablespaces (TTS), Rapid Install, and Release Update Packs.

Application server vs. database server upgrades

Platform migrations may considered separately for the application and database tiers, or for both tiers together. In either case, it is critical to understand that the migrations are separate processes that can be thought of logically as such. This is especially crucial in light of potentially reducing downtimes by breaking them out into separate discrete events which may have different considerations for users and administrators.

Upgrade your database server before upgrading EBS

We recommend that you migrate your database to a new version and new server platform first.  This allows you to perform this in a separate earlier downtime.

For example, you might wish to migrate your EBS database tier running 11gR2 to Oracle Linux 5 on newer and faster hardware prior to an R12 upgrade. Performing this migration first would leave the your environment in a certified configuration that you can continue using until you wish to perform your EBS R12 upgrade.

Regardless of whether this migration is done in a separate earlier downtime or as part of a single downtime, the newer hardware would allow the EBS upgrade to go faster.

Best practices for combined upgrades

Here are our recommendations when performing EBS upgrades and platform migrations in a single project:

  1. Consider the database and application tier migrations separately and plan to perform the database migration first.

  2. Choose the right migration process for the database while considering the target platform, database size and process complexity.

  3. Migrate and upgrade your EBS application tier from 11i to R12 by laying down the new application tier on the target platform as part of the upgrade.

For more details, see:

Related Articles

Thursday Sep 01, 2011

Do You Need to Bounce E-Business Suite Application Servers Regularly?

A customer recently reported an issue where they needed to bounce their E-Business Suite application tier (mid-tier) servers once a week to resolve stability issues.  This shouldn't be necessary.  We recommend against bouncing EBS application servers regularly.  We've done a lot of work with the E-Business Suite to ensure that regular bounces are not required with the latest releases.

Bouncing your system can reduce performance

It seems counter-intuitive, but bouncing your EBS application tier servers can actually reduce your system's performance rather than improve it.  Oracle E-Business Suite is explicitly designed to minimize round trips between the application tier and the database.  We make extensive use of caches:

  • JDBC cache buffers
  • Java object cache
  • MSD cache for OA Framework pages
  • ... and others
Every time you bounce your mid-tier, you clear these caches.  The longer your mid-tier remains up and running, the better-populated these caches will be, and the better your overall system performance.

Treat the symptoms but miss the real problem

If you need to bounce your EBS application tier server regularly, there's a deeper problem with your environment. 

Here are some issues that can be deceptively-masked by bouncing your EBS mid-tier:

  1. Improperly-tuned heap space (e.g. insufficient heap space, insufficient space for perm generation)
  2. Memory leaks (usually the result of poorly-designed customizations)
  3. Memory hemorrhage (usually the result of infinite loops or poorly-designed custom queries)
  4. Heap space fragmentation (usually the result of running older Java releases)
  5. JDBC connection leaks (resolved in and all R12 releases)

If you're experiencing problems that seem to go away when you bounce your mid-tier, that's a red flag.  You should log a formal Service Request with Oracle Support so that we can help you identify the root cause.


Related Articles

Wednesday Aug 31, 2011

Using Single Client Access Name (SCAN) Listeners with Oracle E-Business Suite

[July 30, 2012 update: Corrected Note reference to 823586.1]

[Sept. 2, 2011 update:  Our Maximum Availability Architecture architects have pointed out two additional considerations that are documented in our existing documentation:  1)  Although SCAN listener is supported with EBS 12, if scan_name resolves in DNS to IP1 and IP2, the client-side 10.1.2 network code does not use round robin on the underlyng IPs if the first IP fails.  An Autoconfig solution for this is being tracked through bug 10427234. The only existing workaround is to create custom TNS aliases configured with multiple SCAN IP addresses (Note 823587.1); 2) EBS does support use SCAN functionality, it cannot make complete use of the SCAN since the EBS clients are pre-11.2.  For more details, see this Exadata whitepaper.]

Oracle database 11g logoSingle Client Access Name (SCAN) is a new feature of Oracle Real Applications Clusters (RAC) 11g Release that provides a single name for the clients to access Oracle Database running in a cluster.  The benefit of the SCAN Listener is that the client's connection data does not need to changed if you add or delete a node from a cluster.

The Single Client Access Name is configured during the installation of Oracle Grid Infrastructure. Once configured, application tier connection descriptors just specify the SCAN name rather than all the [virtual] hosts in the cluster.

Without the Single Client Access Name, the descriptor for a two-node cluster would be


With the Single Client Access Name, just the SCAN name needs to be specified:


The benefit of the Single Client Access Name becomes apparent as the number of nodes in the cluster increases.

Which E-Business Suite releases can be used with SCAN Listeners?

You can configure E-Business Suite Release 11i and 12 environments to take advantage of Single Client Access Name functionality.  The use of SCAN Listener is optional.

EBS 11i

AutoConfig in E-Business Suite 11i does not  support automated handling of SCAN Listeners.  If you would like to use SCAN Listeners with EBS 11i environments, a manual setup process is described in:

EBS 12

AutoConfig in Oracle E-Business Suite Release 12 supports automated handling of SCAN Listeners.  If you would like to use SCAN Listeners with EBS 12 environments, see:


    Related Articles

    Tuesday Aug 30, 2011

    WebCast Replay Available: Tuning All Layers of EBS (Part 1)

    I am pleased to release the replay and presentation for the latest ATG Live Webcast:

    Lester Gutierrez, Senior Architect, and Deepak Bhatnagar, Senior Manager, from the E-Business Suite Application Performance team, lead Tuning All Layers of E-Business Suite (Part 1 of 3). This webcast provides an overview of how Oracle E-Business Suite system administrators, DBAs, developers, and implementers can improve E-Business Suite performance by following a performance tuning framework. Part 1 focuses on the performance triage approach, tuning applications modules, upgrade performance best practices, and tuning the database tier. This ATG Live Webcast is an expansion of the performance sessions at conferences that are perennial favourites with hardcore Apps DBAs. (August 2011)

    Stay tuned for Part 2 after OpenWorld

    There will not be an ATG Live Webcast on the last Thursday in September. The normally scheduled event would be just days prior to Oracle Open World 2011. ATG Live Webcasts will resume on October 27th, 2011 with Tuning All Layers of E-Business Suite (Part 2 of 3), with Lester Gutierrez, Senior Architect, and Deepak Bhatnagar, Senior Manager, continuing on in the series of performance tuning presentations. Look for the announcement of the October 2011 webcast in late September.

    Finding other recorded ATG webcasts

    The catalog of ATG Live Webcast replays, presentations, and all ATG training materials is available in this blog's Webcasts and Training section.

    Thursday Jul 28, 2011

    EBS Support Information Center + Patching & Maintenance Advisor Available on My Oracle Support

    The Knowledge Base in My Oracle Support is a sprawling database.  It can be very difficult to find the right needle in that haystack.  My Oracle Support colleagues have recently published two new resources on My Oracle Support that highlight some documents that Apps DBAs might find useful.

    Screenshot of EBS Support Information Center Noe 1322667.1

    1. EBS Support Information Center (Note 1322667.1) contains pointers to:

    • EBS Diagnostics
    • EBS Upgrade related resources
    • EBS Electronic Technical Reference Manual (e-TRM)
    • EBS Release Content Documents (RCD)
    • Oracle Support webcast series
    • Oracle Support newsletters

    2. Patching & Maintenance Advisor: E-Business Suite 11i and R12 (Note 313.1) contains pointers to:

    • EBS 11i and 12 Patching Frequently Asked Questions (FAQ)
    • Patching best practices
    • Methods for reducing patching downtimes
    • Patch Wizard

    Related Articles

    Tuesday Jul 26, 2011

    What's the Best Way to Patch an E-Business Suite Environment?

    One of our senior Oracle Support engineers recently asked, "What's the best strategy for applying patches to E-Business Suite environments?"  Applying updates and patches is such a major part of an Apps DBA's responsibilities that I assumed that this was already covered in our existing documentation. In looking through our systems administration guides, I was surprised to find that this isn't discussed anywhere.

    Here's a general best practices patching strategy for all EBS environments.  In order of priority, you should:

    1. Apply the latest EBS Release Update Pack.
      For example: 12.1.3 for Release 12.1, 12.0.6 for Release 12.0, for Release 11i.

    2. Apply the latest EBS Family Packs and all patches on the Recommended Patch List.
      This includes ATG Release Update Packs and AutoConfig updates.

    3. Upgrade all technology stack components to the latest certified levels.
      For example, as of today, the latest certified levels for EBS 12 are Database, Forms, OC4J, Oracle Internet Directory  Check the Certifications database on My Oracle Support or this blog's one page summary of EBS certifications for a snapshot of the latest certified patchsets for technology stack components.

    4. Apply the latest Critical Patch Updates (CPU)
      You should always apply the latest security patches.  Always.  These are released quarterly.

    5. <Optional> Apply the latest Database Patch Set Updates (PSU) and the required EBS prerequisites
      It's safe to apply Database PSUs to your EBS environment.  Some customers like the convenience of PSUs.  Some don't.  It's up to you.

    6. Apply specific one-off and interim patches...
      ... but only if they're really required for your environment.  This applies to both EBS and technology stack component patches.  You're generally better-off waiting for patches to be rolled into the bigger release vehicles listed in #1 to #4, above.  Those consolidated release vehicles are tested more comprehensively with all Apps modules and configurations than one-off or interim patches. 

    Pay attention to the ranking

    The order of priority is important.  Most customers have difficulties in keeping up with the first four sets of updates.  If you do keep current with those first four categories of patches, the amount of incremental work associated with the remaining two categories is minimal and can generally be avoided entirely.

    Why bother patching?

    Because it's cheaper and safer in the long run.  I'm always surprised by customers who pay more attention to maintaining their car than a mission-critical enterprise resource planning system.

    The E-Business Suite is a large, complex system.  The further you fall behind in maintaining your system:

    • The more work it requires to isolate and resolve issues.
    • The greater the likelihood of encountering an avoidable known issue.
    • The greater the risk of upgrading from an untested configuration to a new version.
    • The greater the effort required to upgrade to new versions.
    • The greater the risk that you will not be able to get patches for your existing configuration.

    Use tools to automate your updates

    The savvy DBA takes advantage of as many tools as possible to automate these updates.  If you are still applying patches manually to every EBS environment, you're wasting a lot of time.  Check out the Application Change Management Pack for Oracle E-Business Suite (part of the Application Management Suite).  This Enterprise Manager plug-in automates the process of checking for required patches, staging those patches, then applying those patches to multiple EBS environments. 

    Yes, it requires additional licencing, but I think that most organisations can recoup those costs very, very quickly.  Patching manually is time-consuming, error-prone, and tricky to keep organized when you have multiple development, staging, QA, and preproduction environments.  In my eyes, automation is really the only solution.

    Formal documentation updates are coming

    Future versions of our formal EBS documentation will include this advice.  Our Oracle Support team is also working on a set of Life Cycle Advisor documents for the E-Business Suite.  They tell me that this will cover everything a DBA should know if you've never applied patches to EBS before.  I'll profile those new Advisor guides as soon as they're published.

    Related Articles

    Tuesday May 17, 2011

    Database Migration using 11gR2 Transportable Tablespaces Now Certified for EBS 12

    Database migration across platforms of different "endian" (byte ordering) formats using the Transportable Tablespaces (TTS) process is now certified for Oracle E-Business Suite Release R12 (12.0.4 and higher, 12.1.1 and higher) with Oracle Database 11g Release 2.

    Small endian to big endian platform mappings

    Obtaining the Required EBS Patch

    This migration process requires a patch delivered by the E-Business Suite Platform Engineering team that is 'Controlled', i.e. it requires a password obtainable from Development through Oracle Support. We released the required patch in this manner to potentially monitor customer issues (due to the nature of this technology), to gauge suitability and to monitor uptake. Customers may be required to fill out a questionnaire as part of an evaluation and approval process.

    Does it meet your requirements?

    Note that for migration across platforms of the same "endian" format, users are advised to use the Transportable Database (TDB) migration process instead for large databases. The "endian-ness" target platforms can be verified by querying the view V$DB_TRANSPORTABLE_PLATFORM using sqlplus (connected as sysdba) on the source platform:

    SQL> select platform_name from v$db_transportable_platform;
    If the intended target platform does not appear in the output, it means that it is of a different endian format from the source and Transportable Tablespaces (for large databases) or export/import should be used for database migration.

    The use of Transportable Tablespaces can greatly speed up the migration of the data portion of the database - it does not affect metadata which must still be migrated using export/import. We recommend that users initially perform a test migration with export/import on their database with the 'metrics=y' parameter to find out the relative size of data vs metadata in their database and to have a basis to compare any gains in timing. Generally speaking, the larger the relative size of data (as compared to metadata), the more likely it would be that XTTS is suitable to reduce downtime.


    Related Articles

    Monday Jan 24, 2011

    In-Depth: Thoughts on Testing from a Battle-Scarred Support Engineer

    [Editor's Note: Weighing in at over 3,100 words, Nick Quarmby's treatise below is more of a whitepaper-in-disguise than our traditionally-short blog articles.  This article is worth your time, though.  In my experience, shortchanging testing simply translates to longer upgrade times downstream.  With EBS 11i now in Extended Support, everyone still on that release should already have their R12 planning underway.  Nick's insights and tips are worth careful consideration by everyone, but particularly so for if you're planning your R12 upgrade.]

    "The more I practice, the luckier I seem to get."

    "If you think safety is expensive, just try having an accident."

    The two quotes above I think pretty accurately sum up the importance of preparation and planning. The first has been attributed to various professional golfers over the years and suggests luck is never accidental. If things work, they work for a reason and that reason is usually because of good preparation, not good luck.

    The second quote has been attributed to various airline industry executives. Here the message is that however much time, and inevitably money, that you spend making sure your planes do not crash, it will cost you considerably more if one of them does.

    The real message is of course that planning and preparation pay off in the long term and, in the context of your Oracle Applications upgrade, you can conclude that successful upgrades are the result of good testing, not good fortune.

    This article is just a few thoughts from a battle-scarred Support Engineer on why good testing is crucial to the success of your system and why it is important for you to thoroughly test any patches, upgrades or migrations before carrying out the work on your production environment. The great majority of the work that you do in any project will be testing. Underestimate it at your peril. Testing is not simply a necessary evil which has to be endured before the inevitable, white-knuckle ride of the production upgrade. Testing is not where you should be cutting corners. Testing is where you should be going into every conceivable corner you can find and looking for something that might, without warning, jump out of the shadows and derail your production upgrade.

    There are links at the end of the document to various tools you have access to which may help your testing but this article is not intended to go into the specifics of testing but simply to discuss how testing should be approached and hopefully raise awareness of the consequences of not doing enough testing.


    A significant Oracle Applications upgrade is a project that will be measured in months, not weeks. The most important part of that long process will be the work that is done before the production upgrade.

    A requirement to upgrade a system or perform maintenance is usually triggered via a business or technical requirement. Once a need to upgrade is identified, the work always seems to be considered urgent - these days everything is urgent. Early on in the project you need to clearly identify what is needed to achieve a successful upgrade and present this to the people driving the project. You may sometimes feel you are presented with unreasonable timescales to complete a project and commercial pressures beyond your control have placed you in this position. This is sometimes an inevitability but it is for these reasons that, as early as possible in a project, you emphasise to the people driving your business that systems do not change over the course of one frenetic weekend. Well, actually, they do, but what happens during that weekend is simply the tip of the iceberg.

    Behind the scenes you must do a lot of work to prepare for that tiny weekend window at the end of your long project plan. It's sometimes inevitable that tasks tend to concertina towards the end of a project and, whilst some people work better under pressure with a visible and looming deadline, many others do not. Use your time wisely unless you look forward to sleepless nights.

    Testing should be performed on a complete cloned copy of your production environment, and, where possible, on an identical hardware environment. If migrating to new hardware, your upgrade testing should include the new hardware so you are confident you can configure the software on the new hardware and that it performs correctly in the new environment.

    You should never consider applying untested patches on your production environment.  However urgent a patch may be, it is never so urgent you should risk the stability of the whole production environment by applying it untested. Oracle Applications patches (patches applied using adpatch) are not reversible without using database rollback/flashback features and it would also require a detailed analysis of the patch and its log files in conjunction with your specific system. You should never assume you can manually reverse even the simplest of Oracle Applications patches. Oracle Support cannot provide this service for you remotely via a Service Request (SR).

    Do not rely on the assurances of your suppliers and assume everything will work on the day. You need to prove to yourself that your suppliers will deliver what you need and when you need it. Contact them early in the project - they will be grateful for the opportunity to be part of your plans. If things go wrong, having a supplier to blame for things not working may absolve you of some responsibility but you will still have to present this failure within your organisation and that will reflect on you, no matter how much you consider that you were not responsible for that failure.

    Always follow the documentation.   Context sensitive documents exist for upgrading all components of Oracle Applications and should be followed. If you cannot find an appropriate document for a process in your upgrade, contact Oracle Support who should be able to advise you of the right Note or manual to follow. Do not assume generic documentation will be applicable to your Oracle Applications environment.

    Our published documentation is designed to achieve a successful upgrade. It is not tested by trying to perform the upgrade steps in a different order or by deliberately omitting steps to see how the upgrade will turn out. You should perform all steps in the upgrade documentation unless you have a clear reason not to, or clear guidance from Oracle Support that a step can be omitted.

    Consider phasing your upgrade.  If a major upgrade involves both a database and Applications upgrade you may be able to perform this in separate, shorter downtimes. 10gR2 and 11gR2 are certified on both 11i and R12 so you could perform a database upgrade during your first downtime, return the system to your users and perform the Applications upgrade in the second downtime at a later date. This will increase the testing you have to do and will also require additional user acceptance testing but you may find this method works for you.

    Testing is not just to establish that the technical upgrade is a success. Testing should also include ensuring that the system you build performs acceptably under load and remains stable. The links at the end of this article refer to products you can use to measure performance and load on your environment. 

    The truth is the hardest work you will do in your whole upgrade project will be the testing and preparation. It is during testing that you will find where you have to reduce what takes two weeks into something that you'll be lucky to get 48 hours to perform. It is during testing that you will have to find imaginative ways to solve the issues you encounter.

    Issues Encountered During Testing

    Some people see testing as something that is only for the overly cautious - those people who are too scared to confront real-time events when they arise. We may live in an increasingly risk-averse society but the truth is that smart people are always testing. Only fools rush in. A macho or cavalier attitude to testing rarely produces a reliable or stable environment.

    Your testing will reveal issues that have to be solved.  These truly are opportunities and not problems. This is why you are testing in the first place - so that you do not see these errors when you come to the production upgrade. If you encounter issues during testing then you must find a solution to these issues or find a workaround that does not compromise the rest of the upgrade. Skipping a documented step because you don't think it's significant can easily cause problems further down the line. 

    Perform at least one test upgrade before you make a commitment to how soon you can perform your production upgrade. A well run project should allow the key people involved time to scope out their work before committing to any deadlines. If you do not know the scale of the job, you cannot accurately plan for the production upgrade. As part of the whole project, you should plan to perform a minimum of three successful test upgrades.

    During testing you will see software acting inconsistently under apparent laboratory conditions. Some people seem to accept software is a somewhat capricious product that can behave unpredictably and that it's acceptable to see it acting differently from one day to the next so long as it doesn't actually do the wrong thing. This is not how commercial business software is designed or expected to behave. You should be able to find a reason for that inconsistent behaviour. If you cannot, then you have to make a call on how significant you believe that inconsistency is, but be assured, there is a reason for this apparently erratic behaviour. Software - even fiendishly complicated software - does not have a mind of its own.

    Use Service Requests (SR) with Oracle Support to validate your upgrade plans and to help you if you encounter any technical issues during your testing. An upgrade where the first we hear from you is when you raise a Severity one SR during the production upgrade is already a failing upgrade. We want to know about any issues you encounter during testing when these can be handled as Severity two and three type SRs where you can work closely with a single engineer, usually in your own time zone, for the duration of the SR.

    Be flexible in your testing. Oracle may release a new maintenance pack or updates to patches or the technology stack in the middle of your testing. Consider whether you should now integrate these later releases into your planned upgrade. This will require additional testing but you may not get another downtime window for some time so it's important you upgrade to the latest software whenever possible.

    The Production Upgrade

    Some DBAs approach a production upgrade with about as much relish as they approach invasive dental work. They know it will be painful, there may be some sleepless nights, and it will cost much more than expected. This is a worrying but increasingly common scenario and it's a real concern to think that some customers feel that the production phase of an upgrade is not only the most stressful part of their project but also the stage most likely to produce an unpredictable outcome.

    It would be wrong to dismiss the above concerns as groundless but it's important to realise that with good planning and testing, the production upgrade should not exactly be a formality, but it should certainly be something for which you are fully prepared for every scenario you can think of, and probably a few you have not thought of.

    If you approach your production upgrade with dread, unsure of what might go wrong, and wondering if it will succeed then this is almost certainly due to a lack of effective preparation and testing. You may feel under pressure to deliver a system but this is nothing compared to the pressure you will feel if you deliver nothing at all. If you really go into your production upgrade being unsure of its success then you should not be contemplating performing it. Production downtime is not something you will be offered regularly and it should not be lost on speculative or poorly tested work.

    The Apollo astronauts on their way to the Moon, had a "go/no go" decision at each critical point in their mission. Your production upgrade should run on similar lines. At each significant stage in the production upgrade you should decide whether you are on course to reach your objective. You should not be afraid to abandon a production upgrade which has gone wrong and instead focus on returning the old system to your users as quickly as possible. This should be the worst scenario you can contemplate but your project plan should include a contingency for it. Your users may not thank you for this but having the old system available on Monday morning is always more acceptable than having no system at all to offer them. You can always come back to fight another day. At no time should you be in a position that you have nothing to offer your users but a broken, part-upgraded environment.

    Your production upgrade is not the place where you should be trying any of the following:-

    • Fixing errors that you have not seen during testing
    • Trying out a different upgrade path that you did not think of during testing
    • Applying patches that you did not require during testing

    The common message in the above is that you should not be trying anything in your production upgrade that you have not done during testing. This should be pretty much a golden rule of your production upgrade. One day you may have to break that rule but that day will be an exceptional one.

    Using standby database functionality from Data Guard is very useful when upgrading. You can configure up to nine standby databases off your production database. If, for example, you have a standby database, you can decouple this at the start of the upgrade. This is then immediately available as a pre-upgrade backup should your production upgrade fail. If the production upgrade succeeds then you just bring this standby database back online after the upgrade and it will automatically resynchronise with the new production database through Data Guard. This may be a little more complicated if your upgrade includes a database upgrade but standby databases can still have an important part to play in any maintenance exercise.

    It's important to approach a production upgrade with a positive mental attitude. A belief that what you are going to do will succeed means you will approach any problems with a belief that you can solve them. If you expect your upgrade to fail then problems come like body blows and you struggle to pick yourself up and fight back. You will only have that positive mental attitude if the weeks and months you have spent prior to the production upgrade have been used constructively giving you the confidence you need to make your production upgrade a success.

    When Things Go Wrong

    Fans of The Hitchhiker's Guide to the Galaxy will know that printed on the cover of this book is the phrase "DON'T PANIC" (always in upper case). This is always good advice in difficult circumstances. A calm head is always best when things don't seem to be going to plan.

    Always have a contingency plan.  Expect things to go wrong and congratulate yourself when they don't, but do not assume that the most fastidious of testing will guarantee a success. Having a contingency or backup plan will support you. You'll be reassured that you were smart enough to plan for every eventuality.

    Do not expect to rely on others in a crisis.  This is your upgrade and your responsibility. Your colleague at the next desk may be sympathetic to your predicament but that does not mean they do not have their own issues to deal with. Nobody knows your upgrade as intimately as you do. If your production upgrade fails and you need help, consider how much time you may have to spend familiairising a third party with your upgrade and your environment before they can offer constructive help as to how you might overcome your current problem. They may be able to help you but if they cannot, then the onus is back on you to find a solution to your problem. Good preparation before you started your upgrade will be invaluable here.

    Do not assume that if a component of the upgrade fails then you can resort to the supplier and they will be responsible for the failure. That may give you a solution in the long term but it may not fix your immediate short term problem. Having a new solution after the event may deflect some of the recriminations as to why the upgrade failed but that will not make the failure go away.


    We want your upgrade to succeed.  Oracle Support is available to help during testing and also during your production upgrade. Make sure you use us to ensure everything we offer is available to you throughout your project. Engage your Service Delivery Manager (SDM) and ensure they know about your upgrade. They can make Oracle Support aware of your plans.

    The media is full of stories about IT upgrades that have failed. Outside of promotional literature, you rarely read a headline that says "IT Project Goes Exactly to Plan." That does not mean that successes do not exist -- it just means that they do not make the news.

    There is rarely much glory in doing your work quietly, efficiently and delivering what is promised, on time and within budget. But if you look around your organisation you will notice that some of the most successful people in the business are the ones who do exactly that. They're not lucky. They planned it that way.

    Related Articles

    Thursday Dec 02, 2010

    Maintaining Your EBS Environment for Maximum Performance

    I think that all E-Business Suite DBAs know that they need to apply a number of interoperability patches when upgrading from one database release to another.  These are documented in our Interoperability Upgrade Notes for a given EBS + database combination.  For example, if you're upgrading Oracle E-Business Suite Release 12 to the latest 11gR2 database, you'd apply the patches listed here:
    However, I don't think it's as widely-known that there are two other classes of patches that you might need to apply.  These are documented in the following two Notes:
    1. Oracle E-Business Suite Recommended Performance Patches (Note 244040.1)
    2. Database Patch Set Update Overlay Patches Required for Use with PSUs and Oracle E-Business Suite (Note 1147107.1)

    Performance benchmark graph for EBS 12 1 2 cpu_load.png

    What are E-Business Suite "recommended performance patches"?

    Oracle E-Business Suite Recommended Performance Patches (Note 244040.1) is written and maintained by our Applications Performance team.  I've covered this team's responsibilities previously, but to recap: this team is dedicated to optimizing the performance of the E-Business Suite.  They perform all of our E-Business Suite benchmarks and high-volume scalability testing, and they work intensively with our largest E-Business Suite customers in fine-tuning E-Business Suite configurations. 

    They've summarized all of their hard-won expertise about performance-related patches in a single document.  A major part of this document covers database patches, but that's not all.  It also covers performance-related patches for:
    • Client tools
    • Applications Technology products and components
    • Financials products
    • Manufacturing and Supply Chain products
    • Human Resources products
    • Sales & Marketing products
    This should be your first stop after upgrading your database or if you're trying to nail down a troublesome database performance-related issue.  I consider this mandatory reading for all E-Business Suite DBAs.

    Do you want to apply database Patch Set Updates?

    Our Server Technologies division began releasing quarterly database Patch Set Updates (PSU) in July 2009.   Patch Set Updates include:
    • Field-tested fixes for critical technical issues that may affect a large number of customers
    • Critical Patch Update fixes
    Database Patch Set Updates are optional for E-Business Suite users.  If you choose to use them, database Patch Set Updates may safely be applied to Oracle E-Business Suite Release 11i and 12 environments.  Patch Set Updates are fully compatible and supported for use with the E-Business Suite.  Depending upon the database version and Oracle E-Business Suite release, one or more overlay patches may be required to address conflicts with a specific Oracle Database Patch Set Update.

    Database Patch Set Update Overlay Patches Required for Use with PSUs and Oracle E-Business Suite (Note 1147107.1) summarizes the latest overlay patches required when applying Database Patch Set Updates to Oracle E-Business Suite environments running on the 10gR2, 11gR1, and 11gR2 databases.  If you use PSUs with your EBS database, you should review this Note on a regular basis.

    Related Articles



    « July 2016