Debugging Series: Troubleshooting EDN Events for SOA

This SOA internal component provides the queuing and messaging platform, most notably for events subscribed to by BPM and BPEL processes. Fusion Applications contains more than 400 BPEL processes, and they are launched from EDN by activities such as ESS job execution or the ADFbc layer run during a page submit. Understanding EDN can therefore help with both development and troubleshooting extension and customization.

System Faults vs Business Faults

As the EDN component is part of the SOA Infrastructure (SOA-INFRA) deployed as a J2EE Application on a WebLogic Server, generic problems are viewed and managed just like any other deployed application. The image below shows status and performance for the Java Server running the SOA-INFRA application.


Figure 1 – The WLS Server running the SOA-INFRA application.

Any BPEL processes that have failed due to a problem with a technology component like EDN may be recovered using actions available in the EM FA Control console. Although this depends on the error handling for each BPEL process,failure during event processing is  a common break in execution where retries can be made. All BPEL processes will contain a generic throw activity as the default catch-all fault handler, therefore once the component is restarted review the composite instance in Enterprise Manager to see what is available.

Where the failure is based on a business problem, specific to that BPEL composite instance, then the option to retry is often supported. In fact there are commonly two options to do this, in Enterprise Manager or in Fusion Applications. The recommendation is to use the latter of these when it exists, since it will ensure that all the required and dependent application data is in a proper state for continued execution. The figure below shows the Distributed Order Orchestration page with the Recover Process button available for the failed shipment flow, and similar exists for several Financials and HCM processes.


Figure 2 – Option to retry a failed BPEL process online.

Design-Time Recommendations

If you are extending existing BPEL processes, or adding your own composites, there are some recommendations to consider in terms of handling failures and their relation to using EDN.

SOA supports something called Fault Policies which are predetermined fault conditions and their respective actions. These are assigned to one or more of your composites via something known as a Fault Binding.Using a set of standardized fault policies is recommended so that failures are easy to recognize and support, and these should be at least used for catch-all handling. It is also possible to extend the standard policies provided (XML definition), although this requires redeployment of each  composite that uses them.

In addition here are some other generic recommendations in terms of ensuring failures in your custom BPEL processes are easy to support. We’ll look at these in more detail in other BPEL-specific blog posts also.

  • Use a meaningful name for your custom faults (e.g. “MissingContactEmail”).
  • Make sure your data cannot be corrupted by a recovery (retry) from your custom composite. Run tests to verify code re-execution doesn’t cause any problems.
  • Use asynchronous design and a switch condition after a mid-process receive activity to check the response status for any errors passed back.
  • Use the BPEL Sensor Variables to make sure standard Fusion Applications Logging is used. These variable names are as follows and then populated with values they automatically call the AppsLoggerService to write to the application diagnostic log. See the developers guide for more details.
    • LOG_FINEST_VAR
    • LOG_FINER_VAR
    • LOG_FINE_VAR
    • LOG_CONFIG_VAR
  • Always terminate your process upon failure, after sending the appropriate notifications.

Run-Time Troubleshooting and Debugging

First it is possible to monitor EDN usage in Enterprise Manager using the Business Events item in the SOA_INFRA menu, as shown below.


From here you can see the details of all events, the subscribing composites, and there is a tab also listing any EDN-related faults for this SOA Infrastructure.


Figure 3 and 4 – EDN monitoring using Enterprise Manager

Logging Data

In addition to reviewing the status and message data in the Enterprise Manager screens, there are a few lower level areas that can provide fine-grained information for use in troubleshooting.

The first is to simply run a quick SQL query to review the current waiting backlog in the EDN advanced queue tables in the database. There are basically two tables to query, given below, and specific event items move from first to the second table.

  • EDN_EVENT_QUEUE_TABLE – contains all events.
  • EDN_OAOO_DELIVERY_TABLE – holds the once-and-only-once events (most for Fusion Applications).

In addition there are useful database Views that can be used for the same types of SQL queries, such as AQ$EDN_EVENT_QUEUE_TABLE and others all starting with AQ$…

These views and tables exist for each SOA-INFRA application (i.e. one per product family), therefore it is important to make sure you’re looking at the right one in the right database schema. The schema names are fairly self-evident, such as HCM_FUSION_SOAINFRA, CRM_FUSION_SOAINFRA, SCM_FUSION_SOAINFRA and so on.

As per standard advanced queuing, once processed the event message row in the table is cleared, so its just the waiting backlog that is shown.

In addition, it is worth checking the different Event Types to help distinguish to rows you are looking at are not just internal ones, plus the MSG_STATE column has the following cryptic values:

  • 0 meaning “ready”
  • 1 meaning “wait”
  • 2 meaning “processed”
  • 3 meaning “expired”
  • 8 meaning “deferred”
  • 10 meaning “buffered expired”

The EDN messages in queue tables do have a timeout and max retries properties. This is setup in the SOA configuration files and can be accessed online using System MBean browser in Enterprise Manager. The screenshot below shows these configurations, including the NumberOfRetries value.


Figure 5 – System MBean Browser showing the EDN Configuration MBeans.

In addition, all EDN event logs go into the EDN_LOG_MESSAGES table. This is again provisioned once for each {family}FUSION_SOAINFRA schema. As well as simply querying the rows in this table, the same content can be shown to administrators using a webpage, using the following link:

  • http://[domainURL:port]/soa-infra/events/edn-db-log

Finally, in order to get more detailed output from the EDN components it is recommended to configure the related Java Loggers. These are accessible in the Configure Log Messages screen in Enterprise Manager, and are listed under the following parent node. They support TRACE levels 1, 16, and 32 for debug logging.

  • oracle.integration.platform.blocks.event

For more details on configuring and using the loggers see the references section below.

Forthcoming Features

Finally the following items illustrate our continued commitment to enhance the EDN processing system, and help offer developers and administrators even more useful features in troubleshooting.

  • Auto-retry will become available for all Fusion Applications SOA composites from Fusion Applications Release 8 onwards.
  • It will be soon possible to re-route problematic human workflow tasks to another participant who can take alternative actions.
  • Should a composite fail the capabilities of automated remedial actions (Fault Policies) will be extended to include a call out to a pre-registered WebService.

References

The following links provide more in-depth reading on some of the topics covered in this post.

  • The excellent Oracle A-Team blog post also on debugging EDN. This has very similar content to the logging section above, however with a little more depth in some areas. It also contains some good examples of applied use.
  • Fusion Applications Administrators Guide section on Troubleshooting SOA. This has many common issues and solutions, such as twelve things to check if an EDN Event Subscription is not fired.
  • Fusion Applications Administrators Section on SOA incidents and on logging.
  • Fusion Applications Developers Guide section on debugging ADF and SOA.

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Follow us on twitter Fusion Applications Extensibility, Customizations and Integration forum Fusion Applications Dev Relations YouTube Channel
This blog offers news, tips and information for developers building extensions, customizations and integrations for Oracle Fusion Applications.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
4
5
6
8
11
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today