Proactive insights, news and tips from Oracle Fusion Middleware Support. Learn Oracle from Oracle.

Diagnostics Enhancements in SOA Suite 11g PS6 (

What's new with Diagnostics in PS6?

Interval Sampling

Purpose: To collect Diagnostic Dumps at specified intervals for as long as they need to be collected.

Activation: By default there are samplings configured for the Diagnostic Dumps 'jvm.threads' and 'jvm.histogram' but they will not begin collecting samples
until an Incident is generated on the server. New samples can be started and managed through

NOTE: To enable dump sampling on a healthy server that has not yet generated a DFW Incident you must activate dump sampling for a healthy server. Steps:

  1. Login in the EM Console

  2. Right click on the domain name in the left menu and select 'System MBean Browser'

  3. Navigate to the MBean: '‘Application Defined MBeans’ -> ‘oracle.dfw’ -> ‘Domain: <domain>’ -> ‘oracle.dfw.jmx.DiagnosticsConfigMbean’

  4. Select the 'DiagnosticsConfig' mbean

  5. Change the value of the 'DumpSamplingIdleWhenHealthy' attribute to 'false'

  6. Click 'Apply'. No restart is necessary for the change to take effect

Collection: The samples are loaded into memory and maintained there until the sampling session is stopped. To collect the current sample archive requires
only the execution of a simple WLST command.

Example Commands:

Start and connect to a running server with WLST:

  1. Start WLST with '/oracle_common/common/bin/wlst.sh'

  2. Connect to the running server with 'connect('<user>','<pwd>','t3://<host>:<port>')'

List the currently running dump samples:
wls:/soasingle/serverConfig> listDumpSamples()

(default output)

Dump Name:jvm.threads
Application Name:
Sampling Interval:5
Rotation Count:10
Dump Implicitly:true
Append Samples:true
Dump Arguments:timing=true, context=true

Dump Name:jvm.classhistogram
Application Name:
Sampling Interval:1800
Rotation Count:5
Dump Implicitly:false
Append Samples:true
Dump Arguments:

Collect the current memory buffer for the 'JVMThreadDump' sample:
wls:/soasingle/serverConfig> getSamplingArchives(sampleName='JVMThreadDump',outputFile='<file>')

wrote 194751 bytes to <file>

The archive is written to the specified location as a zip. In the case of JVMThreadDumps the archive contains a .dmp file containing the actual thread dumps
and a text file listing the sampling parameters and the dumps that were taken.

Start a new sampling session:
wls:/soasingle/serverConfig> addDumpSample(sampleName='ThreadsSample',diagnosticDumpName='jvm.threads',samplingInterval=20,rotationCount=20,
toAppend=true,args={'context' : 'true'})

ThreadsSample is added

You can then confirm the sampling session with the 'listDumpSamples()' command.

Kill a sampling session:
wls:/soasingle/serverConfig> removeDumpSample(sampleName='ThreadsSample')

Removed ThreadsSample

You can confirm that the sampling session is gone by again running the 'listDumpSamples()' command.

back to top

Contextual SOA Thread Dump Translation

In PS6 we now have the ability to generate thread dumps from DFW that will tell us which composite and component a particular active SOA thread is running.
This is best illustrated through an example.

Execute the jvm.threads Diagnostic Dump from WLST against an active SOA Server:

NOTE: You MUST pass the 'context' argument in order to have the SOA context information included in the dump.
wls:/soasingle/serverConfig> executeDump(name='jvm.threads',outputFile='<file>',args={'context' : 'true'})

(no output in the command window)

Go to the location where you wrote the file and open it. The file will look like a typical thread dump but scroll past the stack traces and you'll find some CPU statistics for
each thread and below that the SOA context information. Here we list a partial stack for an active SOA thread followed by the corresponding context entry:
"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'"
idx=0x100 tid=27191 prio=5 alive, in native, daemon

    at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)

    at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)

    at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

    at java/net/SocketInputStream.read(SocketInputStream.java:129)

    at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:730)

    at oracle/net/ns/Packet.receive(Packet.java:302)

    at oracle/net/ns/DataPacket.receive(DataPacket.java:108)

    at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:317)

    at oracle/net/ns/NetInputStream.read(NetInputStream.java:262)



idECIDRIDContext Values

The entries of primary interest are highlighted in red.

  1. Thread ID: This is the ID of the thread from the thread dump entry. We list them all but only provide the context information for SOA threads

  2. composite_name: The name of the composite that is running in that thread

  3. component_name: The name of the component inside the composite that is running in the thread

  4. activity_name: The name of the BPEL activity that is currently running

  5. composite_instance_id: The instance id of the composite that can be looked up in EM

There are obviously some other entries but these are the one that we think will be of most benefit.

NOTE: In the initial release you may see instances where Mediator is listed as the running component when it ought to be BPEL. In these instances the
'activity_name' should still be accurate and list the running BPEL activity

back to top

BPEL Diagnostic Dumps

There are 3 new Diagnostic Dumps for BPEL in PS6:

  • bpel.dispatcher: Dumps the current state of the BPEL thread pools including Engine, Invoke and System.

  • bpel.adt: Dumps the average time that messages for asynchronous BPEL processes wait in the DLV_MESSAGE table before being consumed by an Engine thread

  • bpel.apt: Dumps the average processing time for all BPEL components in every deployed composite


List the available SOA Diagnostic Dumps after starting WLST with '/oracle_common/common/bin/wlst.sh' and connecting to a running SOA server.
wls:/soasingle/serverConfig> listDumps(appName='soa-infra')
















(To list the dumps that are part of the system default)

wls:/soasingle/serverConfig> listDumps()












Execute bpel.dispatcher:
wls:/soasingle/serverConfig> executeDump(name='bpel.dispatcher',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location. Sample

The report first lists the BPEL thread pools along with their configured sizes. It then breaks them down by statistics such as number of active threads, processed messages,
errored messages, etc. Keep in mind that synchronous requests are handled by the default WLS Work Manager and therefore will not be reflected here. These pools handle
callbacks (Invoke) and asynchronous requests (Engine).

Execute bpel.adt:
wls:/soasingle/serverConfig> executeDump(name='bpel.adt',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location. No sample available at the moment report looks identical to bpel.apt, listing the asynchronous BPEL
instances and their avg delay in processing messages from the DLV_MESSAGE table.

Execute bpel.apt:
wls:/soasingle/serverConfig> executeDump(name='bpel.apt',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location. Sample

The report lists the deployed BPEL components and their avg processing times in seconds.

back to top

Adapater Diagnostic Dumps

In PS6 there are 3 new Adapter Diagnostic Dumps:

  • soa.adapter.connpool: Dumps the current adapter connection pool statistics

  • soa.adapter.ra: Dumps the adapter Connection Factory configuration parameters

  • soa.adapter.stats: Dumps the current DMS statistics for all deployed adapter instances

Execute soa.adapter.connpool:
wls:/soasingle/serverConfig> executeDump(name='soa.adapter.connpool',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location. Sample

The report groups the connection pool statistics by the adapter instance so you'll see a header including the instance name followed by the list of statistics.

Execute soa.adapter.ra:
wls:/soasingle/serverConfig> executeDump(name='soa.adapter.ra',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location. Sample

The report groups the Connection Factory configurations by deployed adapter so again you'll see the header with the Adapter type followed by the configuration parameters.

Execute soa.adapter.stats:
wls:/soasingle/serverConfig> executeDump(name='soa.adapter.stats',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location. Sample

The report lists statistics such as min time, max time and avg time along with number of processed / error instances. They are grouped by adapter instance.

back to top

Service Bus Diagnostic Dumps

In PS6 there are 3 new Service Bus Diagnostic Dumps:

  • OSB.derived-cache: Dumps the current statistics and contents of all Service Bus derived caches

  • OSB.jms-async-table: Dumps the current contents of the JMS async table which stores message correlation IDs, timestamps, etc

  • OSB.mq-async-table: Dumps the current contents of the MQ async table which stores similar information to the JMS async table

Execute OSB.derived-cache:
wls:/soasingle/serverConfig> executeDump(name='OSB.derived-cache',appName='OSB',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location and the specific caches dumped are:

  • ArchiveClassLoader: Cache of dependency-aware archive class-loaders

  • ArchiveSummary: Cache of archive summaries

  • CodecFactory: Cache of codec factories

  • EffectiveWSDL: Cache of the EffectiveWSDL objects that are derived from the service or wsdl resources of Proxy or Business services

  • Flow_Info: Cache of flow info objects

  • LightweightEffectiveWSDL: Cache of the EffectiveWSDL objects that are derived from the service or wsdl resources of Proxy or Business services

  • MflExecutor: Cache of the MFL executors

  • RouterRuntime: Cache of the compiled router runtimes for proxy services

  • RuntimeEffectiveWSDL: Cache of the Session valid EffectiveWSDL objects that are derived from the service or wsdl resources of Proxy or Business services

  • RuntimeEffectiveWSPolicy: Cache of WS Policies for Proxy or Business services

  • SchemaTypeSystem: Type system cache for MFL, XS and WSDLs

  • ServiceAlertsStatisticInfo: Cache of service alert statistics for Proxy or Business services

  • ServiceInfo: Cache of compile service info for Proxy or Business services and WSDLs

  • Wsdl_Info: Cache of WSDL Info objects

  • WsPolicyMetadata>: Cache of compiled WS Policy Metadata

  • XMLSchema_Info: XML Schema info cache for xml schema objects

  • XqueryExecutor: Cache of Xquery executors

  • XsltExecutor: Cache of XSLT executors

  • alsb.transports.ejb.bindingtype: Cache of EJB Binding Info for EJB Business Services

  • alsb.transports.jejb.business.bindingtype: Cache of JEJB Binding Info

  • alsb.transports.jejb.proxy.bindingtype: Cache of JEJB Binding Info

Execute OSB.jms-async-table:
wls:/soasingle/serverConfig> executeDump(name='OSB.jms-async-table',appName='OSB',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location.

Execute OSB.mq-async-table:
wls:/soasingle/serverConfig> executeDump(name='OSB.mq-async-table',appName='OSB',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location.

back to top

There is one new Diagnostic Dump for Mediator which dumps the content of the resequencer table to help in troubleshooting resequencer issues. To execute:
wls:/soasingle/serverConfig> executeDump(name='mediator.resequencer',appName='soa-infra',outputFile='<file>')

(There is no console output)

The output is a text file in the specified location.

Troubleshooting Example with the new Features

Most of the new Diagnostic Dumps report performance related information so we will use a performance example. Our composite called 'DiagProject' is failing to process
messages at the expected rate. What might be the problem?

We can start at a number of places but here is a list of Diagnostic Dumps to collect up front:
wls:/soasingle/serverConfig> executeDump(name='bpel.dispatcher',appName='soa-infra',outputFile='<file>')

wls:/soasingle/serverConfig> executeDump(name='bpel.apt',appName='soa-infra',outputFile='<file>')

wls:/soasingle/serverConfig> executeDump(name='soa.adapter.connpool',appName='soa-infra',outputFile='<file>')

wls:/soasingle/serverConfig> executeDump(name='soa.adapter.stats',appName='soa-infra',outputFile='<file>')

wls:/soasingle/serverConfig> executeDump(name='jvm.threads',outputFile='<file>',args={'context' : 'true'})

So what do we have?

  • bpel.dispatcher shows that all of the BPEL thread pools are fine and

    running without a backlog. This means that the bottleneck is not a lack of available BPEL threads

  • bpel.apt clearly shows that average execution time of the bpel component is

    over 5 seconds which is much longer than expected

  • soa.adapter.connpool shows that only one adapter, AQ, is running and

    its connection pool is healthy

  • soa.adapter.stats again shows that the only adapter running is AQ and its

    performance is quite good with an average processing time of 421ms

  • jvm.threads may actually be the first thing you want to look at and when viewing

    the context information at the bottom it's clear that everything is running the BPELProcess1 component, indicating that it is taking by far the most

    time to complete. Looking a little closer at the 'activity_name' we can see that many of them are running an embedded activity which is indeed where the

    delay is (Thread.sleep()).

So clearly the bottleneck is within the BPEL process itself and on further review you find that someone left a 5 second Wait activity in the process. Contrived to be sure but
hopefully still somewhat helpful.

back to top

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.