Debugging Series: Fusion Applications Incidents
By Richard Bingham on Mar 26, 2013
When the incident is created the end user sees only a generic error dialog that gives them the incident number and server details. This protects them from confusing error messages and gives them something they can use to follow up with their helpdesk.
Figure 1 – What the end user sees when an incident is created (click here to enlarge).
Administrators (and Oracle's own Cloud team) manage problem incidents using Oracle Enterprise Manager. Each Enterprise Manager target displays any incidents associated with it, plus the Support Workbench feature allows full management including; file download, compare to other metrics, match against known issues, and incident packaging and sending to MyOracleSupport as a Service Request.
Figure 2 – Incidents associated with Fusion Procurement, with various information and actions available for each (click here to enlarge).
What Exactly Is An Incident?
As mentioned each incident contains a snapshot of system data to help identify and resolve the error. There is an internal framework process that captures this data, known as the Diagnostic Data Extractor (DDE), and some embedded logic that tells it what to collect for each problem type. In addition, as illustrated by the Additional Diagnostics tab in Figure 4, it is possible to explicitly execute the DDE process to get more diagnostic data from a seeded catalog, should it be required.
Fusion Applications J2EE apps incidents contain many files already proven as useful in troubleshooting, with a summary readme.txt detailing out the problem and components involved. Some of the basic middleware diagnostics include:
- Logs – including a file with the last 5 minutes of logs for that WLS Server, another based on all lines for the ECID for which the incident was created, and a third of all lines in the QuickTrace log buffer at that point in time.
- DMS Metric Data – including a dump of all metric values for that server and for that ECID.
- Thread Dump – standard thread dump for all JVM threads at the time of incident creation.
- Click History – for ADD-originating incidents a log of activity passed through the ADF component.
- JRockit Flight Recording – a low level JVM activity trace for the period leading up to incident creation.
- WLS Diagnostic Image – WebLogics’ own system snapshot with detailed component information.
For incidents created by C and PL/SQL code the content is obviously different, and as these are less frequent they contain fewer items, mostly centered around the associated logs and trace files.
The incident content is being expanded all the time, as common troubleshooting requirements are determined and the appropriate DDE collections are built. An example is the plan to include a SQL History file that details out the recent related database statements issued.
Figure 3 – The content of an incident, as shown in Enterprise Manager (click here to enlarge).
Where are Incidents Stored?
All DFw Incident files are stored in a pre-configured filesystem location, known as the Advanced Diagnostic Repository, or ADR. Each component that creates incidents maintains its own ADR directory structure, including the Database, SOA, WebLogic, and each deployed Fusion J2EE Application. The default ADR directories
- Fusion J2EE Apps
- WebLogic Server
- SOA Server
SQL> SELECT * FROM V$DIAG_INFO;
Enterprise Manager has all of the ADR directories pre-registered, therefore it is much easier to use features like Support Workbench, as it already shows files and knows which incidents belong to which component.
When and Why Should Incidents Be Used?
It obviously breaks UX guidelines to show exceptions and call stacks to business users, and implementing the Diagnostic Framework to speed problem resolution time was a decision made at executive level early in the Fusion Applications design. Uptake and adherence is expected, with many bugs already logged where it was originally missing.
Customers may also use and implement incidents in the standard application, and it is recommended for use in all extensions and customizations.
How To Create Incidents
Without going into too much coding depth, there are really two main ways in which to create an incident in Fusion Applications, and exact details and guidelines can be found in the Fusion Applications Developers Guide, under Chapter 78 – Getting Started with Logging and Diagnostics.
Just for completeness it is also possible for customers to create an ad-hoc incident in Enterprise Manager, using the Create User Defined Problem feature.
1) Explicitly using the AppsLogger API
This can be done through either Java code or the SOA Server.
The first is done by calling the AppsLogger.createIncident() method, passing in the appropriate variables for incident creation.
For SOA incidents can be created by using either the LOG_INCIDENT_VAR Sensor Variable with a seeded message, or making the Mediator component call the AppsLoggerService logIncident operation.
For more details on both of these see the references below.
The explicit methods are generally most useful in in catch-all sections of error handlers, where a system problem might cause serious process failures. This is a flexible option for use where seeded messages are inappropriate.
2) Implicitly using Seeded Messages
If you call a seeded message from the database using one of the standard methods it includes a check of for message attributes, and if they’re set appropriately, it will spawn incident creation. Since seeded messages are available for use from Java, PL/SQL, C, or SOA programs, then this is a broader method for incident creation.
For incident creation messages must be defined in FND_MESSAGES_B table with all of the following values:
- MESSAGE_CATEGORY is not NULL
- MESSAGE_SEVERITY = HIGH
- TYPE = ERROR
- LOGGABLE_ALERTABLE = Y
This method is recommended for use with catch-all error messages shown to end users or put into logs, so that administrators and support have diagnostics without having to run additional testcases.
It is possible to edit and add Seeded Messages for use in extensions and customizations using the Function Setup Manager, under Applications Core Setup -> Messages -> Manage Messages. The following screenshot shows a message, with the incident-sensitive fields on the right hand side, namely Message Type, Category, Severity, and Logging Enabled. For this message whenever it is used an incident will also get created, with a problem key of PO-2055005.
Figure 4 – A seeded message that will create an incident when used (click here to enlarge).
The following documents give exact details on using the APIs and tools for creating and using incidents to help you diagnose problems.
- Fusion Applications Administrators Guide - overview in Chapter 15 and troubleshooting in Chapter 17
- Fusion Middleware Administrators Guide – Chapter 13 : Diagnosing Problems
- Oracle Support notes on using Incident Content, including the low level JVM diagnostics: Note 109.1: Troubleshooting Index Note
- Oracle Enterprise Manager Administrators Guide – Chapter 3 : Using Incident Management