Friday Mar 28, 2014

One Queue to Rule them All

Using a Single Queue for Multiple Message Types with SOA Suite

Problem Statement

You use a single JMS queue for sending multiple message types /  service requests.  You use a single JMS queue for receiving multiple message types / service requests.  You have multiple SOA JMS Adapter interfaces for reading and writing these queues.  In a composite it is random which interface gets a message from the JMS queue.  It is not a problem having multiple adapter instances writing to a single queue, the problem is only with having multiple readers because each reader gets the first message on the queue.

Background

The JMS Adapter is unaware of who receives the messages.  Each adapter instance just takes the message from the queue and delivers it to its own configured interface, one interface per adapter instance.  The SOA infrastructure is then responsible for routing that message, usually via a database table and an in memory notification message, to a component within a composite.  Each message will create a new composite but the BPEL engine and Mediator engine will attempt to match callback messages to the appropriate Mediator or BPEL instance.
Note that message type, including XML document type, has nothing to do with the preceding statements.

The net result is that if you have a sequence of two receives from the same queue using different adapters then the messages will be split equally between the two adapters, meaning that half the time the wrong adapter will receive the message.  This blog entry looks at how to resolve this issue.

Note that the same problem occurs whenever you have more than 1 adapter listening to the same queue, whether they are in the same composite or different composites.  The solution in this blog entry is also relevant to this use case.

Solutions

In order to correctly deliver the messages to the correct interface we need to identify the interface they should be delivered to.  This can be done by using JMS properties.  For example the JMSType property can be used to identify the type of the message.  A message selector can be added to the JMS inbound adapter that will cause the adapter to filter out messages intended for other interfaces.  For example if we need to call three services that are implemented in a single application:
  • Service 1 receives messages on the single outbound queue from SOA, it send responses back on the single inbound queue.
  • Similarly Service 2 and Service 3 also receive messages on the single outbound queue from SOA, they send responses back on the single inbound queue.
First we need to ensure the messages are delivered to the correct adapter instance.  This is achieved as follows:
  • aThe inbound JMS adapter is configured with a JMS message selector.  The message selector might be "JMSType='Service1'" for responses from Service 1.  Similarly the selector would be "JMSType='Service2'" for the adapter waiting on a response from Service 2.  The message selector ensures that each adapter instance will retrieve the first message from the queue that matches its selector.
  • The sending service needs to set the JMS property (JMSType in our example) that is used in the message selector.
Now our messages are being delivered to the correct interface we need to make sure that they get delivered to the correct Mediator or BPEL instance.  We do this with correlation.  There are several correlation options:
  1. We can do manual correlation with a correlation set, identifying parts of the outbound message that uniquely identify our instance and matching them with parts of the inbound message to make the correlation.
  2. We can use a Request-Reply JMS adapter which by default expects the response to contain a JMSCorrelationID equal to the outgoing JMSMessageID.  Although no configuration is required for this on the SOA client side, the service needs to copy the incoming JMSMessageID to the outgoing JMSCorrelationID.

Special Case - Request-Reply Synchronous JMS Adapter

When using a synchronous Request-Reply JMS adapter we can omit to specify the message selector because the Request-Reply JMS adapter will immediately do a listen with a message selector for the correlation ID rather than processing the incoming message asynchronously.
The synchronous request-reply will block the BPEL process thread and hold open the BPEL transaction until a response is received, so this should only be used when you expect the request to be completed in a few seconds.

The JCA Connection Factory used must point to a non-XA JMS Connection Factory and must have the isTransacted property set to “false”.  See the documentation for more details.

Sample

I developed a JDeveloper SOA project that demonstrates using a single queue for multiple incoming adapters.  The overall process flow is shown in the picture below.  The BPEL process on the left receives messages from the jms/TestQueue2 and sends messages to the jms/Test Queue1.  A Mediator is used to simulate multiple services and also provide a web interface to initiate the process.  The correct adapter is identified by using JMS message properties and a selector.

 

The flow above shows that the process is initiated from EM using a web service binding on mediator.  The mediator, acting as a client, posts the request to the inbound queue with a JMSType property set to "Initiate".

Model Client BPEL Service
Inbound Request Client receives web service request and posts the request to the inbound queue with JMSType='Initiate' The JMS adapter with a message selector "JMSType='Initiate'" receives the message and causes a composite to be created.  The composite in turn causes the BPEL process to start executing.
The BPEL process then sends a request to Service 1 on the outbound queue.
Key Points
  • Initiate message can be used to initate a correlation set if necessary
  • Selector required to distinguish initiate messages from other messages on the queue
Service 1 receives the request and sends a response on the inbound queue with JMSType='Service1' and JMSCorrelationID= incoming JMS Message ID.
Separate Request and Reply Adapters   The JMS adapter with a message selector "JMSType='Service1'" receives the message and causes a composite to be created.  The composite uses a correlation set to in turn deliver the message to BPEL which correlates it with the existing BPEL process.
The BPEL process then sends a request to Service 2 on the outbound queue.
Key Points
  • Separate request & reply adapters require a correlation set to ensure that reply goes to correct BPEL process instance
  • Selector required to distinguish service 1 response messages from other messages on the queue
Service 2 receives the request and sends a response on the inbound queue with JMSType='Service2' and JMSCorrelationID= incoming JMS Message ID.
Asynchronous Request-Reply Adapter   The JMS adapter with a message selector "JMSType='Service2'" receives the message and causes a composite to be created.  The composite in turn delivers the message to the existing BPEL process using native JMS correlation.
Key Point
  • Asynchronous request-reply adapter does not require a correlation set, JMS adapter auto-correlates using CorrelationID to ensure that reply goes to correct BPEL process instance
  • Selector still required to distinguish service 2 response messages from other messages on the queue
The BPEL process then sends a request to Service 3 on the outbound queue using a synchronous request-reply.
Service 3 receives the request and sends a response on the inbound queue with JMSType='Service2' and JMSCorrelationID= incoming JMS Message ID.
Synchronous Request-Reply Adapter   The synchronous JMS adapter receives the response without a message selector and correlates it to the BPEL process using native JMS correlation and sends the overall response to the outbound queue.
Key Points
  • Synchronous request-reply adapter does not require a correlation set, JMS adapter auto-correlates using CorrelationID to ensure that reply goes to correct BPEL process instance
  • Selector also not required to distinguish service 3 response messages from other messages on the queue because the synchronous adapter is doing a selection on the expected CorrelationID
 
Outbound Response Client receives the response on an outbound queue.      

Summary

When using a single JMS queue for multiple purposes bear in mind the following:

  • If multiple receives use the same queue then you need to have a message selector.  The corollary to this is that the message sender must add a JMS property to the message that can be used in the message selector.
  • When using a request-reply JMS adapter then there is no need for a correlation set, correlation is done in the adapter by matching the outbound JMS message ID to the inbound JMS correlation ID.  The corollary to this is that the message sender must copy the JMS request message ID to the JMS response correlation ID.
  • When using a synchronous request-reply JMS adapter then there is no need for the message selector because the message selection is done based on the JMS correlation ID.
  • Synchronous request-reply adapter requires a non-XA connection factory to be used so that the request part of the interaction can be committed separately to the receive part of the interaction.
  • Synchronous request-reply JMS adapter should only be used when the reply is expected to take just a few seconds.  If the reply is expected to take longer then the asynchronous request-reply JMS adapter should be used.

Deploying the Sample

The sample is available to download here and makes use of the following JMS resources:

JNDI Resource; Notes
jms/TestQueue Queue Outbound queue from the BPEL process
jms/TestQueue2 Queue Inbound queue to the BPEL process
eis/wls/TestQueue JMS Adapter Connector Factory This can point to an XA or non-XA JMS Connection Factory such as weblogic.jms.XAConnectionFactory
eis/wls/TestQueue None-XA JMS Adapter Connector Factory This must point to a non-XA JMS Connection Factory such as weblogic.jms.ConnectionFactory and must have isTransacted set to “false”

To run the sample then just use the test facility in the EM console or the soa-infra application.

Thursday Mar 20, 2014

Packt Publishing Buy One Get One Free Offer

Packt Publishing celebrates their 2000th title with a Buy One Get One Free Offer

Great time to get those Packt books you’ve been thinking of buying, like the SOA Suite 11g Developers Guide or the SOA Suite 11g Developers Cookbook.

Tuesday Aug 13, 2013

Oracle SOA Suite 11g Performance Tuning Cookbook

Just received this to review.

It’s a Java World

The first chapter identifies tools and methods to identify performance bottlenecks, generally covering low level JVM and database issues.  Useful material but not really SOA specific and the authors I think missed the opportunity to share the knowledge they obviously have of how to relate these low level JVM measurements into SOA causes.

Chapter 2 uses the EMC Hyperic tool to monitor SOA Suite and so this chapter may be of limited use to many readers.  Many but not all of the recipes could have been accomplished using the FMW Control that ships and is included in the license of SOA Suite.  One of the recipes uses DMS, which is the built in FMW monitoring system built by Oracle before the acquisition of BEA.  Again this seems to be more about Hyperic than SOA Suite.

Chapter 3 covers performance testing using Apache JMeter.  Like the previous chapters there is very little specific to SOA Suite, indeed in my experience many SOA Suite implementations do not have a Web Service to initiate composites, instead relying on adapters.

Chapter 4 covers JVM memory management, this is another good general Java section but has little SOA specifics in it.

Chapter 5 is yet more Java tuning, in this case generic garbage collection tuning.  Like the earlier chapters, good material but not very SOA specific.  I can’t help feeling that the authors could have made more connections with SOA Suite specifics in their recipes.

Chapter 6 is called platform tuning, but it could have been titled miscellaneous tuning.  This includes a number of Linux optimizations, WebLogic optimizations and JVM optimizations.  I am not sure that yet another explanation of how to create a boot.properties file was needed.

Chapter 7 homes in on JMS & JDBC tuning in WebLogic.

SOA at Last

Chapter 8 finally turns to SOA specifics, unfortunately the description of what dispatcher invoke threads do is misleading, they only control the number of threads retrieving messages from the request queue, synchronous web service calls do not use the request queue and hence do not use these threads.  Several of the recipes in this chapter do more than alter the performance characteristics, they also alter the semantics of the BPEL engine (such as “Changing a BPEL process to be transient”) and I wish there was more discussion of the impacts of these in the chapter.  I didn’t see any reference to the impact on recoverability of processes when turning on in-memory message delivery.  That said the recipes do cover a lot of useful optimizations, and if used judiciously will cause a boost in performance.

Chapter 9 covers optimizing the Mediator, primarily tweaking Mediator threading.  THe descriptions of the impacts of changes in this chapter are very good, and give some helpful indications on whether they will apply to your environment.

Chapter 10 touches very lightly on Rules and Human Workflow, this chapter would have benefited from more recipes.  The two recipes for Rules do offer very valuable advice.  The two workflow recipes seem less valuable.

Chapter 11 takes into the area where the greatest performance optimizations are to be found, the SOA composite itself.  7 generally useful recipes are provided, and I would have liked to see more in this chapter, perhaps at the expense of some of the java tuning in the first half of the book.  I have to say that I do not agree with the “Designing BPEL processes to reduce persistence” recipe, there are better more maintainable and faster ways to deal with this.  The other recipes provide valuable ideas that may help performance of your composites.

Chapter 12 promises “High Performance Configuration”.  Three of the recipes on creating a cluster, configuring an HTTP plug in and setting up distributed queues are covered better in the Oracle documentation, particularly the Enterprise Deployment Guide.  There are however some good suggestions in the recipe about deploying on virtualized environments, I wish they had spoken more about this.  The use of JMS bridges recipe is also a very valuable one that people should be aware of.

The Good, the Bad, and the Ugly

A lot of the recipes are really just trivial variations on other recipes, for example they have one recipe on “Increasing the JVM heap size” and another on “Setting Xmx and Xms to the same value”.

Although the book spends a lot of time on Java tuning, that of itself is reasonable as a lot fo SOA performance tuning is tweaking JVM and WLS parameters.  I would have found it more useful if the dots were connected to relate the Java/WLS tuning sections to specific SOA use cases.

As the authors say when talking about adapter tuning “The preceding sets of recipes are the basics … available in Oracle SOA Suite. There are many other properties that can be tuned, but their effectiveness is situational, so we have chosen to focus on the ones that we feel give improvement for the most projects.”.  They have made a good start, and maybe in a 12c version of the book they can provide more SOA specific information in their Java tuning sections.

Add the book to your library, you are almost certain to find useful ideas in here, but make sure you understand the implications of the changes you are making, the authors do not always spell out the impact on the semantics of your composites.

A sample chapter is available on the Packt Web Site.

Friday Jul 13, 2012

Deploying Fusion Order Demo on 11.1.1.6

How to Deploy Fusion Order Demo on SOA Suite 11.1.1.6

We need to build a demo for a customer, why not use Fusion Order Demo (FOD) and modify it to do some extra things.  Great idea, let me install it on one of my Linux servers I said…

Turns out there are a few gotchas, so here is how I installed it on a Linux server with JDeveloper on my Windows desktop.

Task 1: Install Oracle JDeveloper Studio

I already had JDeveloper 11.1.1.6 with SOA extensions installed so this was easy.

Task 2: Install the Fusion Order Demo Application

First thing to do is to obtain the latest version of the demo from OTN, I obtained the R1 PS5 release.

Gotcha #1 – my winzip wouldn’t unzip the file, I had to use 7-Zip.

Task 3: Install Oracle SOA Suite

On the domain modify the setDomainEnv script by adding “-Djps.app.credential.overwrite.allowed=true” to JAVA_PROPERTIES and restarting the Admin Server.

Also set the JAVA_HOME variable and add Ant to the path.

I created a domain with separate SOA and BAM servers and also set up the Node Manager to make it easier to stop and start components.

Taking a Look at the WebLogic Fusion Order Demo Application

Note that when opening the composite you will get warnings because the components are not yet deployed to MDS.

Deploying Fusion Order Demo

Task 1: Create a Connection to an Oracle WebLogic Server

If some tests complete when you test the connection to the WebLogic domain but other tests fail, for example the JSR-88 tests, then you may need to go into the console and under each servers Configuration->General->Advanced setting, set the “External Listen Address” to be the name that JDeveloper uses to access the managed server.

Task 2: Create a Connection to the Oracle BAM Server

I can’t understand why customers wouldn’t want to use BAM.  Monitor Express makes it a matter of a few clicks to provide real time process status information to the business.

Oh yes!  I remember now, several customers IT staff have told me they don’t want the business seeing this data because they will hassle the IT department if something goes wrong, and BAM lets them see it going wrong in real time…

Task 3: Install the Schema for the Fusion Order Demo Application

When editing the Infrastructure->MasterBuildScript->Resources build.properties make sure that you set jdeveloper.home to the jdeveloper directory underneath the directory that you installed JDeveloper into.  I installed JDeveloper Studio into “C:\JDev11gPS5” so my jdeveloper.home is “C:\JDev11gPS5\jdeveloper”.

Gotcha #2 – the ant script throws an error partway through but does not report it at the end, so check carefully for the following error:

oracle.jbo.JboException: JBO-29000: Unexpected exception caught: java.lang.NoClassDefFoundError, msg=oracle/jdbc/OracleClob

This occurs because the build.xml does not include the ojdbc6dms library which have CLOB support so in JDeveloper add it to the path “oracle.jdbc.path” in Infrastructure->/DatabaseSchema/Resources build.xml file.

<path id="oracle.jdbc.path">
  <fileset dir="${jdeveloper.home}/../wlserver_10.3/server/lib">
    <include name="ojdbc6.jar"/>
  </fileset>
  <fileset dir="${jdeveloper.home}/../oracle_common/modules/oracle.jdbc_11.1.1">
    <include name="ojdbc6dms.jar"/>
  </fileset>
</path>

Rerun the ant script from Infrastructure/Ant with a target of “buildAll” and it should now complete without errors.

Task 4: Set the Configuration Property for the Store Front Module

Nothing to watch out for here.

Task 5: Edit the Database Connection

Nothing to watch out for here.

Task 6: Deploy the Store Front Module

There is an additional step when deploying, you will be asked for an MDS repository to use.  Best to use the MDS-SOA repository and put the content in its own partition.

Gotcha #3 –when prompted select the mds-soa MDS repository and choose the soa.  Note that this is an MDS partition, not a SOA partition.

Note that when you deploy the StoreFrontServiceSDO_Services application it will populate the local WebLogic LDAP with the demo users.  If this step fails it will be because you forgot to set the “-Djps.app.credential.overwrite.allowed=true” parameter and restart the Admin Server.

Task 7: Deploy the WebLogic Fusion Order Demo Application

Set up the environment for BAM.

When editing the WebLogicFusionOrderDemo->bin->Resources build.properties make sure that you set oracle.home to the jdeveloper directory underneath the directory that you installed JDeveloper into.  I installed JDeveloper Studio into “C:\JDev11gPS5” so my oracle.home is “C:\JDev11gPS5\jdeveloper”.

Gotcha #5a – Make sure that you create directories on the server for the FileAdapter to use as a file directory and a separate control directory and make sure you set the corresponding properties in the build.properties file:

  • orderbooking.file.adapter.dir
  • orderbooking.file.adapter.control.dir

Gotcha #5b – Also make sure you set the following properties in the build.properties file:

  • soa.domain.name
  • managed.server.host
  • managed.server.rmi.port
  • soa.db.username
  • soa.db.password
  • soa.db.connectstring

Note that the soa.server.oracle.home property must be set to the ORACLE_SOA_HOME (usually Oracle_SOA1 under the MW_HOME) on the server.

Gotcha #6 – I found that unless I went into the console to each servers Configuration->Protocols->IIOP->Advanced setting, and set the “Default IIOP Username” and “Default IIOP Password” to be the weblogic user the deployment failed.

Gotcha #7 – when deploying BAM objects in seedBAMServerObjects activity I got an exception “java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64” which is caused because the BAM installation under JDeveloper does not have all the required libraries.  To fix this copy the commons-codec-1.3.jar file from the server machine ORACLE_SOA_HOME/bam/modules/oracle.bam.third.party_11.1.1 to the JDev machine ORACLE_JDEV_HOME/bam/modules/oracle.bam.third.party_11.1.1.

Gotcha #8 – when deploying BAM objects in seedBAMServerObjects activity I got an error “BAM-02440: ICommand is unable to connect to the Oracle BAM server because user credentials (username/password) have not been specified.”.  The quick way to fix this is to change to the directory where the import script was created on the JDeveloper machine (ORACLE_JDEV_HOME\bam\dataObjects\load) and run the load script after setting the JAVA_HOME

..\..\bin\icommand -CMDFILE ImportFODBamObjects.xml

I am sure if I spent more time in the ant scripts I could have found what was wrong with the script for deploying this.

Running Fusion Order Demo

You are now ready to place an order through the frontend app at http://soahost:soaport/StoreFrontModule/faces/home.jspx.  The BAM dashboard is available for you to monitor the progress of your order and EM is all set to let you monitor the health of the processes.  Enjoy studying a relatively complex example that demonstrates many best practices such as use of MDS.

Friday Mar 16, 2012

Memory Efficient Windows SOA Server

Installing a Memory Efficient SOA Suite 11.1.1.6 on Windows Server

Well 11.1.1.6 is now available for download so I thought I would build a Windows Server environment to run it.  I will minimize the memory footprint of the installation by putting all functionality into the Admin Server of the SOA Suite domain.

Required Software

  • 64-bit JDK
  • SOA Suite
    • If you want 64-bit then choose “Generic” rather than “Microsoft Windows 32bit JVM” or “Linux 32bit JVM”
    • This has links to all the required software.
    • If you choose “Generic” then the Repository Creation Utility link does not show, you still need this so change the platform to “Microsoft Windows 32bit JVM” or “Linux 32bit JVM” to get the software.
    • Similarly if you need a database then you need to change the platform to get the link to XE for Windows or Linux.

If possible I recommend installing a 64-bit JDK as this allows you to assign more memory to individual JVMs.

Windows XE will work, but it is better if you can use a full Oracle database because of the limitations on XE that sometimes cause it to run out of space with large or multiple SOA deployments.

Installation Steps

The following flow chart outlines the steps required in installing and configuring SOA Suite.

The steps in the diagram are explained below.

64-bit?

Is a 64-bit installation required?  The Windows & Linux installers will install 32-bit versions of the Sun JDK and JRockit.  A separate JDK must be installed for 64-bit.

Install 64-bit JDK

The 64-bit JDK can be either Hotspot or JRockit.  You can choose either JDK 1.7 or 1.6.

Install WebLogic

If you are using 64-bit then install WebLogic using “java –jar wls1036_generic.jar”.  Make sure you include Coherence in the installation, the easiest way to do this is to accept the “Typical” installation.

SOA Suite Required?

If you are not installing SOA Suite then you can jump straight ahead and create a WebLogic domain.

Install SOA Suite

Run the SOA Suite installer and point it at the existing Middleware Home created for WebLogic.  Note to run the SOA installer on Windows the user must have admin privileges.  I also found that on Windows Server 2008R2 I had to start the installer from a command prompt with administrative privileges, granting it privileges when it ran caused it to ignore the jreLoc parameter.

Database Available?

Do you have access to a database into which you can install the SOA schema.  SOA Suite requires access to an Oracle database (it is supported on other databases but I would always use an oracle database).

Install Database

I use an 11gR2 Oracle database to avoid XE limitations.  Make sure that you set the database character set to be unicode (AL32UTF8).  I also disabled the new security settings because they get in the way for a developer database.  Don’t forget to check that number of processes is at least 150 and number of sessions is not set, or is set to at least 200 (in the DB init parameters).

Run RCU

The SOA Suite database schemas are created by running the Repository Creation Utility.  Install the “SOA and BPM Infrastructure” component to support SOA Suite.  If you keep the schema prefix as “DEV” then the config wizard is easier to complete.

Run Config Wizard

The Config wizard creates the domain which hosts the WebLogic server instances.  To get a minimum footprint SOA installation choose the “Oracle Enterprise Manager” and “Oracle SOA Suite for developers” products.  All other required products will be automatically selected.

The “for developers” installs target the appropriate components at the AdminServer rather than creating a separate managed server to house them.  This reduces the number of JVMs required to run the system and hence the amount of memory required.  This is not suitable for anything other than a developer environment as it mixes the admin and runtime functions together in a single server.  It also takes a long time to load all the required modules, making start up a slow process.

If it exists I would recommend running the config wizard found in the “oracle_common/common/bin” directory under the middleware home.  This should have access to all the templates, including SOA.

If you also want to run BAM in the same JVM as everything else then you need to “Select Optional Configuration” for “Managed Servers, Clusters and Machines”.

To target BAM at the AdminServer delete the “bam_server1” managed server that is created by default.  This will result in BAM being targeted at the AdminServer.

Installation Issues

I had a few problems when I came to test everything in my mega-JVM.

  • Following applications were not targeted and so I needed to target them at the AdminServer:
    • b2bui
    • composer
    • Healthcare UI
    • FMW Welcome Page Application (11.1.0.0.0)

How Memory Efficient is It?

On a Windows 2008R2 Server running under VirtualBox I was able to bring up both the 11gR2 database and SOA/BPM/BAM in 3G memory.  I allocated a minimum 512M to the PermGen and a minimum of 1.5G for the heap.  The setting from setSOADomainEnv are shown below:

set DEFAULT_MEM_ARGS=-Xms1536m -Xmx2048m
set PORT_MEM_ARGS=-Xms1536m -Xmx2048m

set DEFAULT_MEM_ARGS=%DEFAULT_MEM_ARGS% -XX:PermSize=512m -XX:MaxPermSize=768m
set PORT_MEM_ARGS=%PORT_MEM_ARGS% -XX:PermSize=512m -XX:MaxPermSize=768m

I arrived at these numbers by monitoring JVM memory usage in JConsole.

Task Manager showed total system memory usage at 2.9G – just below the 3G I allocated to the VM.

Performance is not stellar but it runs and I could run JDeveloper alongside it on my 8G laptop, so in that sense it was a result!

Tuesday Mar 13, 2012

Packt Oracle Discount Month

Packt Publishing are celebrating the publication of their 50th Oracle book by offering an exclusive discount throughout March for all Oracle books. Here is the link which explains it in detail: Packt's Oracle Campaign, https://www.packtpub.com/news/hit-the-oracle-packtpot. If you haven’t bought mine and Matt’s book the Oracle SOA Suite 11g R1 Developer's Guide book then now is a great time to get it from Packt.  Packt have recently published a number of new Oracle SOA Suite books, including an Oracle Service Bus Bus 11g Development Cookbook by those smart guys in the Netherlands and Switzerland, and an Oracle BAM 11gR1 Handbook by my friend Pete Wang.

Thursday Sep 29, 2011

Mapping the Java World Part I

How to Customise Java/XML Mapping in SOA Suite
Part I Setting Up EclipseLink MOXy

The Challenge

java-xml-duke-mediumDuring a recent POC, the customer asked us to integrate with several of their backend EJBs, which hosted legacy code talking to backend systems.  The EJB interfaces could not be changed, and had to be called a certain way to be successful.  The customer was looking to phase out another set of "integration EJBs", written over the last few years that orchestrated their core backend EJBs.  SOA Suite will allow them to drastically reduce their development time, and provide much better visibility into the processes as they run.  Given their past experiences (writing their integration EJBs was painful and long winded), one of the key requirements was to support their legacy EJBs without writing code in the SOA environment.  At first glance this seemed impossible, because we couldn't change anything on the EJB side.  In addition many of the parameters to the interfaces contained either incomplete annotations or methods that didn't follow the JavaBeans spec.  This was not previously a problem for them because all their EJBs ran in the same JVM and were making local Java calls

We decided to use a powerful yet obscure feature of SOA Suite to do the mapping.  Chapter 49.7 of the SOA Suite Developer's guide mentions this marriage between EclipseLink MOXy and SOA Suite, but it does more than advertised, and works outside of the Spring Framework components.  We decided to use this functionality to "fix" some of the things we had mapping issues with in the customer code.  Additionally, we used the framework to do other helpful tasks, such as changing namespaces, fixing arrays, and removing unnecessary mappings.

In this article we'll cover the theory behind the use of this functionality, basic setup and usage, and several examples to get you started.

Background

When we use an EJB Reference or a Spring component in SOA Suite we usually want to wire it to a non-Java resource.  When we do this JDeveloper uses JAXB to create an XML representation of the parameters and return values of the methods in the Java interface we are using.  In this article we will show how to override those mappings.  Overriding the default generation of mappings allows us to specify target namespaces, rationalize the structure of the data and remove unneeded properties in the Java classes.  Some things we may want to customize include:

  • Specifying concrete implementations for abstract classes and interfaces in the interface
    • This allows us to map to Java objects in the interface which cannot be instantiated directly.  For example often we have lists of abstract classes or interfaces, by  specifying the possible concrete implementations of these classes we can generate an XML schema that includes additional properties available only through the concrete classes.
  • Hiding unwanted properties
    • This allows us to remove properties that are not needed for our implementation, or not needed because they are convenience properties such as the length of an array or collection which can easily be derived from the underlying array or collection.
  • Providing wrappers for arrays and collections
    • The default mapping for an array or collection is to provide a list of repeating elements.  We can modify the mapping to provide a wrapper element that represents the whole array or collection, with the repeating elements appearing a level down inside this.
  • Changing WSDL namespaces
    • It is often necessary to change the namespaces in a generated WSDL to match a corporate standard or to avoid conflicts with other components that are being used.

Approach

EclipseLinkMoxySOA Suite allows us to describe in XML how we want a Java interface to be mapped from Java objects into XML.  The file that does this is called an “Extended Mapping” (EXM) file.  When generating a WSDL and its associated XML Schema from a Java interface SOA Suite looks for an EXM file corresponding to the Java Interface being generated from.  Without this file the mapping will be the “default” generation, which simply attempts to take each field and method in the Java code and map it to an XML type in the resulting WSDL.  The EXM file is used to describe or clarify the mappings to XML and uses EclipseLink MOXy to provide an XML version of Java annotations.  This means that we can apply the equivalent of Java annotations to Java classes referenced from the interface, giving us complete control over how the XML is generated.  This is illustrated in the diagram which shows how the WSDL interface mapping depends on the JavaInterface of the EJB reference or Spring component being wired (obviously), but is modified by the EXM file which in turn may embed or reference an XML version of JAXB annotations (using EclipseLink MOXy).

The mapping will automatically take advantage of any class annotations in the Java classes being mapped, but the XML descriptions can override or add to these annotations, allowing us fine grained control over our XML interface.  This allows for changes to be made without touching the underlying Java code.

Setup

Using the mapper out of the box is fairly simple.   Suppose you set up an EJB Reference or Spring component inside your composite.  You'd like to call this from a Mediator or BPEL Process which expect to operate on a WSDL.  Simply drag a wire from the BPEL process or Mediator to the EJB or Spring component and you should see a WSDL generated for you, which contains the equivalent to the Java components business interface and all required types.  This happens for you, as the tool goes through the classes in your target.

But what if you get a "Schema Generation Error" or if the generated WSDL isn't correct.  As discussed earlier there may be a  number of changes we need or want to make to the mapping.   In order to use an "Extended Mapping", or EXM file, we need to do the following:

  1. We need to register the EclipseLink MOXy schema with JDeveloper.  Under JDeveloper Tools->Preferences->XML Schemas we click Add… to register the schema as an XML extension.

    The schema is found in a jar file located at <JDEV_HOME>/modules/org.eclipse.persistence_1.1.0.0_2-1.jar! and the schema is inside this jhar at /xsd/eclipselink_oxm_2_1.xsd so the location we register is jar:file:/<JDEV_HOME>/modules/org.eclipse.persistence_1.1.0.0_2-1.jar!/xsd/eclipselink_oxm_2_1.xsd where <JDEV_HOME> is the location where you installed JDeveloper.

    NOTE: This will also work as OXM instead of XML.  But if you use an .oxm extension then in each project you use it you must add a rule to copy .oxm files from the source to the output directory when compiling.
  2. Change the order of the source paths in your SOA Project to have SCA-INF/src first.  This is done using the Project Properties…->Project Source Paths dialog.  All files related to the mapping will go here.  For example, <Project>/SCA-INF/src/com/customer/EXM_Mapping_EJB.exm, where com.customer is the associated java package.
  3. We will now use a wizard to generate a base mapping file.
    1. Launch the wizard New XML Document from XML Schema (File->New->All Technologies->General->XML Document from XML Schema).
    2. Specify a file with the name of the Java Interface and an .exm extension in a directory corresponding to the Java package of the interface under SCA-INF/src.  For example, if your EJB adapter defined soa.cookbook.QuoteInterface as the remote interface, then the directory should be <Project>/SCA-INF/src/soa/cookbook and so the full file path would be <Project>/SCA-INF/src/soa/cookbook/QuoteInterface.exm.  By using the exm extension we are able to Use Registered Schema which will automatically map to the correct schema so that future steps in the wizard will understand what we are doing.
    3. The weblogic-wsee-databinding schema should already be selected, select a root element of java-wsdl-mapping and generate to a Depth of 3.  This will give us a basic file to start working with.
  4. As recommended by Oracle, separate out the mappings per package, by using the toplink-oxm-file element.  This will allow you, per package, to define re-usable mapping files outside of the EXM file.  Since the EXM file and the embedded mappings have different XML root elements, defining them separately allows JDeveloper to provide validation and completion, a sample include is shown below:

    <?xml version="1.0" encoding="UTF-8" ?>
    <java-wsdl-mapping xmlns="
    http://xmlns.oracle.com/weblogic/weblogic-wsee-databinding">
      <xml-schema-mapping>
        <toplink-oxm-file file-path="./mappings.xml" java-package="soa.cookbook"/>
      </xml-schema-mapping>
    </java-wsdl-mapping>

  5. Create an "OXM Mapping" file to store custom mappings.  As mentioned, these files are per package, separate from the EXM files, and re-usable.  We can use the "New XML Document from Schema" to create these as well.  In this case, they will have an XML or OXM extension, use the persistence registered schema (http://www.eclipse.org/eclipselink/xsds/persistence/oxm), and be stored relative to the EXM file.  That is, they can go in the same directory, or in other directories, as long as you refer to them by relative path from the EXM file.

    <?xml version="1.0" encoding="UTF-8" ?>
    <xml-bindings xmlns="
    http://www.eclipse.org/eclipselink/xsds/persistence/oxm">
      <!-- Set target Namespace via namespace attribute -->
      <xml-schema namespace=
    http://cookbook.soa.mapping/javatypes
                  element-form-default="QUALIFIED"/>
    </xml-bindings>

  6. In the newly created OXM Mapping file, we can use completion and validation to ensure we follow the EclipseLink MOXy documentation.  For example, to declare that a field is to be transient and not show up in the WSDL mapping, do this:

    <?xml version="1.0" encoding="UTF-8" ?>
    <xml-bindings xmlns="
    http://www.eclipse.org/eclipselink/xsds/persistence/oxm">
      <!-- Set target Namespace via namespace attribute -->
      <xml-schema namespace=
    http://cookbook.soa.mapping/javatypes
                  element-form-default="QUALIFIED"/>
      <java-types>
        <java-type name="soa.cookbook.QuoteRequest">
          <java-attributes>
            <!-- Can remove mappings by making them transient via xml-transient element -->
       
        <xml-transient java-attribute="product"/>
          </java-attributes>
        </java-type>
      </java-types>
    </xml-bindings>

  7. Once complete, delete any existing wires to the Java components and rewire.  You should notice the dialog box change to indicate that an extended mapping file was used.

    Note that an “extended mapping file” was used.
  8. As an iterative process, changing a mapping is quite easy.  Simply delete the wire, and JDeveloper will offer to delete the WSDL file, which you should do.  Make updates to the EXM and OXM Mapping files as required, and rewire.  Often this occurs several times during development, as there are multiple reasons to make changes.  We will cover some of these in our next entry.

Summary

The extended mapping file puts the SOA composite developer in control of his own destiny when it comes to mapping Java into XML.  It frees him from the tyranny of Java developer specified annotations embedded in Java source files and allows the SOA developer to customize the mapping for his own needs.  In this blog entry we have shown how to set up and use extended mapping in SOA Suite composites.  In the next entry we will show some of the power of this mapping.

Patches

To use this effectively you need to download the following patch from MetaLink:

  • 12984003 - SOA SUITE 11.1.1.5 - EJB ADAPTER NONBLOCKINGINVOKE FAILS WITH COMPLEX OBJECTS (Patch)
    • Note that this patch includes a number of fixes that includes a performance fix for the Java/XML mapping in SOA Suite.

Sample Code

There is a sample application uploaded as EXMdemo.zip.  Unzip the provided file and open the EXMMappingApplication.jws in JDeveloper.  The application consists of two projects:

  • EXMEJB
    • This project contains an EJB and needs to be deployed before the other project.  This EJB provides an EJB wrapper around a POJO used in the Spring component in the EXMMapping project.
  • EXMMapping
    • This project contains a SOA composite which has a Spring component that is called from a Mediator, there is also an EJB reference.  The Mediator calls the Spring component or the EJB based on an input value.  Deploy this project after deploying the EJB project.

Key files to examine are listed below:

  • QuoteInterface.java in package soa.cookbook
    • This is the same interface implemented by the Spring component and the EJB.
    • It takes a QuoteRequest object as an input parameter and returns a QuoteResponse object.
  • Quote.java_diagram in package soa.cookbook
    • A UML class diagram showing the structure of the QuoteRequest and QuoteResponse objects.
  • EXM_Mapping_EJB.exm in package soa.cookbook
    • EXM mapping file for EJB.
    • This is used to generate EXM_Mapping_EJB.wsdl file.
  • QuoteInterface.exm in package soa.cookbook
    • EXM mapping file for Spring component.
    • This is used to generate QuoteInterface.wsdl file.
  • mappings.xml in package soa.cookbook
    • Contains mappings for QuoteRequest and QuoteResponse objects.
    • Used by both EXM files (they both include it, showing how we can re-use mappings).
    • We will cover the contents of this file in the next installment of this series.

Sample request message

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body xmlns:ns1="
http://cookbook.soa.mapping/types">
    <ns1:quoteRequest>
      <ns1:products>
        <ns1:product>Product Number 1</ns1:product>
        <ns1:product>Product Number 2</ns1:product>
        <ns1:product>Product Number 3</ns1:product>
      </ns1:products>
      <ns1:requiredDate>2011-09-30T18:00:00.000-06:00</ns1:requiredDate>
      <!-- provider should be “EJB” or “Spring” to select the appropriate target -->
      <ns1:provider>EJB</ns1:provider>
    </ns1:quoteRequest>
  </soap:Body>
</soap:Envelope>

Co-Author

This blog article was co-written with my colleague Andrew Gregory.  If this becomes a habit I will have to change the title of my blog!

Acknowledgements

Blaise Doughan, the Eclipse MOXy lead was extremely patient and helpful as we worked our way through the different mappings.  We also had a lot of support from David Twelves, Chen Shih-Chang, Brian Volpi and Gigi Lee.  Finally thanks to Simone Geib and Naomi Klamen for helping to co-ordinate the different people involved in researching this article.

Thursday Sep 22, 2011

Coping with Failure

Handling Endpoint Failure in OSB

HardwareFailureRecently I was working on a POC and we had demonstrated stellar performance with OSB fronting a BPEL composite calling back end EJBs.  The final test was a failover test which tested killing an OSB and bringing it back online and then killing a SOA(BPEL) server and bringing it back online and finally killing a backend EJB server and bringing it back online.  All was going well until the BPEL failover test when for some reason OSB refused to mark the BPEL server as down.  Turns out we had forgotten to set a very important setting and so this entry outlines how to handle endpoint failure in OSB.

Step 1 – Add Multiple End Points to Business Service

The first thing to do is create multiple end points for the business service, pointing to all available backends.  This is required for HTTP/SOAP bindings.  In theory if using a T3 protocol then a single cluster address is sufficient and load balancing will be taken care of by T3 smart proxies.  In this scenario though we will focus on HTTP/SOAP endpoints.

Navigate to the Business Service->Configuration Details->Transport Configuration and add all your endpoint URIs.  Make sure that Retry Count is greater than 0 if you don’t want to pass failures back to the client.  In the example below I have set up links to three back end webs service instances.  Go to Last and Save the changes.

MultiOSBEndpoint

Step 2 – Enable Offlining & Recovery of Endpoint URIs

When a back end service instance fails we want to take it offline, meaning we want to remove it from the pool of instances to which OSB will route requests.  We do this by navigating to the Business Service->Operational Settings and selecting the Enable check box for Offline Endpoint URIs in the General Configuration section.  This causes OSB to stop routing requests to a backend that returns errors (if the transport setting Retry Application Errors is set) or fails to respond at all.

Offlining the service is good because we won’t send any more requests to a broken endpoint, but we also want to add the endpoint again when it becomes available.  We do this by setting the Enable with Retry Interval in General Configuration to some non-zero value, such as 30 seconds.  Then every 30 seconds OSB will add the failed service endpoint back into the list of endpoints.  If the endpoint is still not ready to accept requests then it will error again and be removed again from the list.  In the example below I have set up a 30 second retry interval.  Remember to hit update and then commit all the session changes.

OfflineOSBEndpoint

Considerations on Retry Count

A couple of things to be aware of on retry count.

If you set retry count to greater than zero then endpoint failures will be transparent to OSB clients, other than the additional delay they experience.  However if the request is mutative (changes the backend) then there is no guarantee that the request might not have been executed but the endpoint failed before turning the result, in which case you submit the mutative operation twice.  If your back end service can’t cope with this then don’t set retries.

If your back-end service can’t cope with retries then you can still get the benefit of transparent retries for non-mutative operations by creating two business services, one with retry enabled that handles non-mutative requests, and the other with retry set to zero that handles mutative requests.

Considerations on Retry Interval for Offline Endpoints

If you set the retry interval to too small a value then it is very likely that your failed endpoint will not have recovered and so you will waste time on a request failing to contact that endpoint before failing over to a new endpoint, this will increase the client response time.  Work out what would be a typical unplanned outage time for a node (such as caused by a JVM failure and subsequent restart) and set the retry interval to be say half of this as a comprise between causing additional client response time delays and adding the endpoint back into the mix as soon as possible.

Conclusion

Always remember to set the Operational Setting to Enable Offlining and then you won’t be surprised in a fail over test!

About

Musings on Fusion Middleware and SOA Picture of Antony Antony works with customers across the US and Canada in implementing SOA and other Fusion Middleware solutions. Antony is the co-author of the SOA Suite 11g Developers Cookbook, the SOA Suite 11g Developers Guide and the SOA Suite Developers Guide.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today