Introduction
In a particular customer's production instance (11.5.9, post ATG RUP 5), the WF administrator email was being bombarded with thousands of emails like these:==================================================
An Error occurred in the following Workflow.
Item Type = XITEMIFS
Item Key = 982893-5825-38602875
User Key =
Error Name = ECX_PROGRAM_EXIT_EMBD
Error Message = 6012: XML Gateway process aborted : 'ORA-29532: Java
call terminated by uncaught Java exception: java.lang.OutOfMemoryError -
ecx_utils.INITIALIZE: '. For further details, see the log corresponding to
FND-Logging AFLOG MODULE Name: ''.
Error Stack =
Wf_Engine.GetItemAttrEvent(XITEMIFS, 982893-5825-38602875,
ECX_EVENT_MESSAGE)
ECX_STANDARD.sendDirectCover(XXINV, SYNC_ITEM)
ECX_STANDARD.Send(XITEMIFS, 982893-5825-38602875, 113178, RUN)
Wf_Engine_Util.Function_Call(ECX_STANDARD.SEND, XITEMIFS,
982893-5825-38602875, 113178, RUN)
Activity Id = 113178
Activity Label = XXINV_RECEIVE_SYNC_EVENT:DOCUMENT_SEND_XMLGATEWAY
Result Code = #EXCEPTION
Notification Id =
Assigned User =
Workflow monitor
Action History
=========================
An Error occurred in the following Workflow.
Item Type = XITEMIFS
Item Key = 986029-5704-38881957
User Key =
Error Name = ECX_PROGRAM_EXIT_EMBD
Error Message = 6012: XML Gateway process aborted : 'ORA-29532: Java call terminated by uncaught Java exception: java.lang.NullPointerException - ecx_utils.INITIALIZE: '. For further details, see the log corresponding to FND-Logging AFLOG MODULE Name: ''.
Error Stack =
Wf_Engine.GetItemAttrEvent(XITEMIFS, 986029-5704-38881957, ECX_EVENT_MESSAGE)
ECX_STANDARD.sendDirectCover(XXINV, SYNC_ITEM)
ECX_STANDARD.Send(XITEMIFS, 986029-5704-38881957, 113178, RUN)
Wf_Engine_Util.Function_Call(ECX_STANDARD.SEND, XITEMIFS, 986029-5704-38881957, 113178, RUN)
Activity Id = 113178
Activity Label = XXINV_RECEIVE_SYNC_EVENT:DOCUMENT_SEND_XMLGATEWAY
Result Code = #EXCEPTION
Notification Id =
Assigned User =
==================================================
Another variation...
An Error occurred in the following Workflow.Item Type = XITEMIFS
Item Key = 982893-5825-38602875
User Key =
Error Name = ECX_PROGRAM_EXIT_EMBD
Error Message = 6012: XML Gateway process aborted : 'ORA-29532: Java call
terminated by uncaught Java exception: java.lang.OutOfMemoryError -
ecx_utils.INITIALIZE: '. For further details, see the log corresponding to
FND-Logging AFLOG MODULE Name: ''.
Error Stack =
Wf_Engine.GetItemAttrEvent(XITEMIFS, 982893-5825-38602875,
ECX_EVENT_MESSAGE)
ECX_STANDARD.sendDirectCover(XXINV, SYNC_ITEM)
ECX_STANDARD.Send(XITEMIFS, 982893-5825-38602875, 113178, RUN)
Wf_Engine_Util.Function_Call(ECX_STANDARD.SEND, XITEMIFS,
982893-5825-38602875, 113178, RUN)
Activity Id = 113178
Activity Label = XXINV_RECEIVE_SYNC_EVENT:DOCUMENT_SEND_XMLGATEWAY
Result Code = #EXCEPTION
Notification Id =
Assigned User =
Workflow monitor
Action History
==================================================
Initial investigation...
Initial investigation revealed that maybe it was beneficial to apply patch 5194357 - REHOST XML PARSER FOR PL/SQL v1.0.2+ for Oracle EBS 11i (as per Metalink note 356141.1). Patch 5356822 (PATCH FOR XMLGATEWAY MEMORY ISSUES) was also mentioned, but it was not clear if it would solve the problem.After studying bugs, another patch come up: 5209280 (APPST: ORA-29532: JAVA CALL TERMINATED BY JAVA EXCEPTION:JAVA.LANG.OUTOFMEMORYER). The product mentioned in the patch information is CORE, which can easily be confused with CORE RDBMS product e.g. 9i, 8i etc. The CORE was actually referring to the CORE component of XDK xmlparser product. Apparently, the XDK XML parser was broken in version 2.0.2.9 and fixed in 8.1.7.2.99.
Later on, we came to understand that patch 5194357 was a APPS rehosting of core patch 5209280.
rehosting ?
"rehosting" is an interesting patch terminology. Essentially, it means that a patch for a core XML sub-component was re-certified to be applied in an Oracle Applications 11i environment. For example, While running patch.sh supplied in patch 5209280, it failed to copy files to $ORACLE_HOME/xdk/plsql/parser/bin . Also since load.sql which was not present in $ORACLE_HOME/xdk/plsql/parser/bin, as mentioned in the README, it could not be executed. It was clear that patch 5209280 made a lot of assumptions about the environment where it was to be executed.
Thats where patch 5194357 (REHOST XML PARSER FOR PL/SQL v1.0.2+ for Oracle EBS 11i) was useful; it was fully compliant to be run with adpatch utility. It was the same fix, but delivered to be applied in an 11i environment.
Workaround
The workaround was to bounce the Workflow Agent Listener Service GSM container from Oracle Applications Manager GUI.Solution
After applying patch 5194357 (REHOST XML PARSER FOR PL/SQL v1.0.2+ for Oracle EBS 11i) and doing heavy regression testing (marking status of SENT or ERRORed notifications in WF_NOTIFICATIONS as MAIL), the memory stamina of the Workflow Agent Listener Service GSM container increased considerably.Essentially, there was a memory leak introduced in the previous XML parser used by the pl/sql sessions of Workflow Agent Listener Service.
