Primary Key Education

Bizarre Behaviour

I had an interesting instant message last night while I was preparing a demonstration for today and watching Mr Bean with my youngest children.  One of my friends and colleague, Sanjeev, couldn't understand why he was submitting a query through the database adapter and getting 3 entries back (as expected) that were all identical (not expected).
The table being queried looked a little like this

SystemID	Date	Message
22222	24-MAY-2007	Disk at 90% Utilisation
22222	29-MAY-2007	Disk at 98% Utilisation
22222	2-JUNE-2007	Error Writing to Disk
33333	1-JUNE-2007	Out of Squirrel Memory

Sanjeev had used the DB adapter to query the table.  Effectively he was running a query like:
Running this in BPEL with an input of 22222 gave him a document like this:
Notice all three records are the same.
Running this query from SQL Plus gave him the following
In the database it seems that all three records are different.
As you can imagine this was pretty baffling.

A Solution

I told him to rerun the wizard, it is re-entrant so previous settings are saved for you, and select all the columns as the primary key in the mapping portion of the wizard.  He did this and voila, all behaved as expected.

The Reason Why

So what was going on?
The table had no primary key constraint on it (sounds of outrage from Mr Codd) and hence effectively had a primary key consisting of all the columns in the table.  When Sanjeev created the database mapping the wizard asked him for a primary key and so he chose the ItemID column.
Under the covers the database adapter uses TopLink for persistence.  TopLink requires a primary key to be used so that it can identify tuples uniquely and manage the object mapping of tuples avoiding duplicates.  To be more efficient with large data sets TopLink only load the primary key fields on a query and then loads the objects as needed based on primary key.
Of course in this case the 'primary key' was the same for all tuples returned and so toplink thought aha! I can save myself some time here and only load the object once and repeat it three times because it is the same object.
Hence changing the mapping to tell the database adapter and hence TopLink that the primary key was really all the columns in the database allowed TopLink to realise that the three entries returned by the query were really three distinct entries and load them as such.

Many Morals

This is a story of many morals.

1. All tables should really have an explicit primary key constraint, it makes the intent clearer.
2. Don't accept default settings without giving some thought to what you are doing.
3. It often helps to understand what is going on under the covers.
4. Getting an Instant Message from a friend during Mr Bean can lead to a whole set of intersting avenues.
   

Organisational Hierarchies

Just had an interesting question from my friend Dave, the only kind of question I get from Dave.  He was using Human Workflow (HW) in BPEL and asked how to set up the management hiearchy.  Basically HW has an identity service which can handle  all identity queries and is mapped onto an identity provider such as a jazn file based provider, or Oracle Internet Directory or Active Directory.  It is also possible to seperate out the authentication and/or authorization providers so they are distinct from the main identity provider.  Well the jazn service has a property that point to a file user-properties.xml that contains all kinds of extra information about the users described in jazn-data.xml.  This type of splitting of stores can be done by using seperate services as part of the identity provider.  Pretty useful, particularly when your main authorization store has no other useful information such as manager information which is critical to HW.  Adding a secondary store is very useful in this scenario.
For more information look at the appropriate section in the manual.

Testing Notification Services

Just in the middle of developing a proof of concept for a customer that is heavily human workflow focussed and so has a need for lots of email notifications to fly around.  The challenge is how to test this without the need for a heavy EMail setup.

Requirements

Here are what I see as the requirements

• Lightweight mail server, my machine is already running database XE, SOA Suite with BAM, JDeveloper and browser so CPU and memory are at a premium.
  
• Easy to install client and server
  
• Lightweight mail client for the same reasons as above.
  
• Easy to administer mail server, particularly creating new accounts
• Easy to install multiple accounts into mail client.
  
• Ability to switch mail client between different mail accounts easily.  Should also be easy to see which mail account you are using.

These requirements are dictated by a desire to have different users participate in the human workflow and receive their own EMail notifications.

Lightweight Mail Client

And the winner of the lightweight mail client category is .... Microsoft Outlook Express.  It comes pre-installed with Windows so installation is non-existant.  It is not a memory hog, much smaller than Thunderbird (my personal preference for day to day use) or Outlook.  Finally it has an ability to switch users with the File->Switch Identity ... option.  This lets me switch between users when testing and demoing without having to shutdown and startup a mail client.

Lightweight Mail Server

The winner of the lightweight mail server is MailEnable.  This is easy to install and get running and administration is trivial.  Adding a new mailbox takes about 2 minutes at most.  It provides standard POP 3 services swo setting up an additional account in Outlook Express is just a matter of providing the e mail address and setting the POP3 and SMTP host names.  Adding a new account in MailEnable and then providing an additional identity in Outlook Express again only takes 2 minutes.
Best of all the base edition is free for persoanl and commercial use as explained on the web site.

Running with Human Workflow

Once I had installed MailEnable and set up OutLook Express to use it all I needed to do was configure the EMAil notification service as described in the documentation and I had a lightweight EMail notification service hooked into BPEL Human Workflow.  Great for development and handy for demos.

Instrumenting Human Workflow

or BAM Enabling Oracle BPEL Workflow Services

I was just working on a proof of concept for a customer with my colleague Matt Wright when we had a need to monitor, through BAM, the progress of tasks in the human workflow service that ships with Oracle BPEL.  So we dove into JDeveloper and looked at the workflow editor to try and find where we could generate events for BAM to consume.  Shock, horror!  There don't seem to be any!
At this point Matt started thinking about how we could move some of the auto routing out of the human workflow and make it explicit in the surrounding BPEL process.

A Blind Alley

While Matt considered how he might rework his workflows I had a look at a button called 'Configure CallBacks ...' in the advanced section of the workflow editor.  This is described on the screen as 'Specify callback class on task status.'. 
Sounds interesting I thought, so I dived in and discovered that I could
use this button to register a Java callback class with the human workflow
task.  I could control when I wanted the callback to occur, the
following being the most interesting

• On Assignment
• On Update
• On Completion

I was well down this route when a far superior solution presented itself.

Expanding Workflow

At this point one of the product team (thanks Bhagat I will always be grateful) came to our rescue and pointed out the 'Allow task and routing customization in BPEL callbacks' check box on the advanced section of the workflow editor.  This  causes the workflow task to callback into the initiating BPEL process on useful events such as

• On Assignment
• On Update
• On Completion

This seemed to be exactly what we wanted so we clicked it!


So now we had the callbacks being sent but we still had to enable the BPEL process to properly receive and process them.  To do this I went into the diagram view of the BPEL editor and clicked on the human task in my process.  I then chose the advanced tab and clicked the 'Allow task and routing customizations in BPEL callbacks' check box to cause the BPEL editor to generate the code to handle the callbacks from the workflow task.

This caused the BPEL to be expanded to include event notification handling for the callbacks as shown below:

BPEL flow before callbacks	BPEL flow after callbacks

With this in place I could now instrument the callbacks by adding BAM sensors to get events into BAM whenever a task was assigned, completed or updated, allowing me to track in real time how the process was developing.  Hopefully a future release of workflow services will let us embed the BAM sensors direct into the workflow.

In Conclusion

In this entry we have looked at how we can use the callback feature of human workflow to generate events into a BPEL process so that we can then trivially instrument those events using BAM  This extends the range of things we can monitor with BAM into the internal routing of tasks within the BPEL workflow service.  To enable this for an existing BPEL process with human workflow we do the following

• Enable callbacks from human workflow by checking the "Allow task and routing customization in BPEL callbacks" box in the "Advanced Settings" section of the workflow editor.  This configures the Human Workflow service to send the callbacks.
  
• Modify the BPEL generation to include handling code for the callbacks by checking the "Allow task and routing customization in BPEL callbacks" in the "Advanced" tab of the human task dialog on the BPEL diagram editor.  This must be done if human workflow is configured to send the callbacks otherwise you will get errors in the worklist application.
  

That just leaves the hard part, deciding which data we want to capture and monitor in BAM.

Extending XPath

Of late I have run come across the need to extend the built in functionality of XPath.  Occasionally you come across transformations or calculations that can't be done in XPath.  For example I have found it tricky to calculate the time interval between two dates.  This is very useful when you have a deadline to meet and want to set a time interval for a given time period from now.  The problem is you only know the time interval from the deadline, not to the dealine.  So lets look at how we would extend XPath functionality to provide some useful additional functions.

Adding a Duration Calculator

The functionality we want is very simple.  Given two XML datetimes we want to be able to calculate the XML time duration between them.  In Java this is very simple.  The following Java method does exactly this.
    public String calculateDifference(Date earlier, Date later) {
        Duration duration = null;
        try {
            long diff = later.getTime() - earlier.getTime();
            duration = DatatypeFactory.newInstance().newDuration(diff);
        } catch (DatatypeConfigurationException e) {
            return e.toString();
        }
        return duration.toString();
    }
So how do we wrap this code up to make it available to BPEL via XPath.  Obviously we could turn it into a web service but it would be more convenient to have this available as an XPath function, avoiding the need for an extra partner link.
To create an XPath function version of this code we follow the instructions in the manual.  Clemens has also blogged about this, but I plan on covering a bit more detail.

Wrapping the Code

To be made available as an XPath function we need to first implement the IXPathFunction interface.  This has a single method that it requires us to implement:
The IXPathContext interface provides access to the environment, principally BPEL variables, and personally I don't feel it should be used much if we are creating true functions.  The List interface is a standard java.util.List that provides access to the parameters, if any.  Here is the implementation of the method.
    public Object call(IXPathContext iXPathContext,
                       List list) throws XPathFunctionException {
        try {
            // Declarations
            DatatypeFactory factory = DatatypeFactory.newInstance();
            XMLGregorianCalendar start = null;
            XMLGregorianCalendar end = null;
            // Verify got two parameters
            if (list.size() != 2)
                throw new XPathFunctionException("calculateDuration(startdate, enddate) needs 2 arguments");
            // Get value of input parameters
            Object p1 = list.get(0);
            Object p2 = list.get(1);
            String sp1 = getParamValue(p1);
            String sp2 = getParamValue(p2);
            // Convert to Date objects and pass in to function
            start = factory.newXMLGregorianCalendar(sp1);
            end = factory.newXMLGregorianCalendar(sp2);
            return calculateDifference(start.toGregorianCalendar().getTime(),
                                       end.toGregorianCalendar().getTime());
        } catch (DatatypeConfigurationException e) {
            throw new XPathFunctionException("Failed to initialise DatatypeFactory",
                                             e);
        }
    }
This implementation verifies that it received two parameters and then extracted the value of those parameters using the following method.
    private String getParamValue(Object param) {
        String value;
        if (param instanceof Element) {
            Node n = (Node)param;
            value = n.getFirstChild().getNodeValue();
        } else if (param instanceof Text) {
            Node n = (Node)param;
            value = n.getNodeValue();
        } else {
            value = String.valueOf(param);
        }
        return value;
    }
This method takes an Element or a String literal and their value.  It assumes that the element has a String value associated with it.
Now that we have implemented the Java code for the XPath function we can deploy it by dropping the classes into the $ORACLE_HOME/bpel/system/classes directory of the BPEL PM server.

Telling BPEL about the New Function

Dr. Strangelove: Of course, the whole point of a Doomsday Machine is lost, if you *keep* it a *secret*! Why didn't you tell the world, EH?

So how do we tell the world, or at least the BPEL PM about our new function?  We add it to the $ORACLE_HOME/bpel/system/config/xpath-functions.xml file.  The appropriate entry is shown below.
    <function id="getTimeDifference" arity="2">
        <classname>ajr.TimeDifference</classname>
        <comment>
            <![CDATA[The signature of this function is <i>ajr:getTimeDifference(time1, time2)</i>.
      The arguments to the function:
      <ol type="1">
          <li>time1 - String or element containing the datetime of the start of the interval</li>
          <li>time2 - String or element containing the datetime of the end of the interval</li>
      </ol>
      Returns: an XML duration formatted string]]>
        </comment>
        <property id="namespace-uri">
            <value>http://ajr/utilities/xpath</value>
            <comment>Namespace URI for this function</comment>
        </property>
        <property id="namespace-prefix">
            <value>ajr</value>
            <comment>Namespace prefix for this function</comment>
        </property>
    </function>
This maps the XPath function ajr:getTimeDifference onto the Java class ajr.TimeDifference.  In addition it tells BPEL that at least 2 parameters (arity="2") are expected.
After these changes and deploying the classes I bounced the container running the BPEL PM to get it to pick up the changes.
We have now implemented the new XPath function and told BPEL about it, so it is time to test it.

Testing the Duration Calculator

To test it I created a BPEL process that accepted as input a duration and then added it to the current time to give a future date.  I then calculated the duration between the two dates using the getTimeDifference XPath funciton that we just registered.  Within the BPEL process I performed 4 copy operations

1. Copy xp20:current-dateTime() to start element.  This gives a starting date.
2. Copy the input duration to duration element.  Note that this should be in the format P[xxY][xxM][xxD][T[xxH][mmM][ssS]. For example 'PT10M' indicates 10 minutes whilst 'P1DT2H5S' represents 1 day, 2 hours and 5 seconds.
3. Add the duration element to the start element using xp20:add-dayTimeDuration-to-dateTime and copy the result to the startPlusDuration element.
4. Calculate the difference in time between the start element and the startPlusDuration element using the ajr:getTimeDifference function and copy the result to the calculatedDuration element.
   

The result of a test run is given below:
Note that the calculated duration has a slightly different textual value but the same semantic value as the original duration, indicating that the funciton is working as expected.

Gotchas Along the Way

The biggest gotcha I have come across is getting the right types passed into XPath functions.  Basically you have three different types that can be passed across

• Element types
• Node Lists
• Strings

If we get the value of an element then it will return null and we actually need to get the value of the first child node which should be a text node if the element has a value.  The XPath function ora:getNodeValue does this for us or it can be done in the XPath functions Java as shown in the getParamValue method.
Node lists are as the name implies lists of elements and we need to iterate over them within our XPath function using Java code to get each individual item.
Finally Strings are exactly what you thouught you where passing in the first place!

Downloads

I have uploaded the XPath code in a JDeveloper project called {manilaSuite.gems.includeGem (64)}.  In addition to the ajr:getTimeDifference function it also has a function called ajr:inspectParameters that will return an XML document listing the structure of all the parameters passed to it.  Useful for exploring all the possible ways in which an XPath extension function can receive data.  To try it out just create a BPEL process with an anyType output element and assign the output of the function to that element.  It is easier to view the return result in the visual flow.
The sample BPEL project to test the time difference function is uploaded as {manilaSuite.gems.includeGem (65)}.

What's the Time Mr BPEL?

Seems that often in BPEL you have a process that has some sort of time dependency.  Examples are :

• Response must be given in 5 days.
• Product must be shipped to arrive by 12 October 2007.
  
• How old will a child born on 19th January 1995 be on 1 September 2007?
  

Lets quickly review how we deal with these.

XML Date Formats

XML dateTime takes the form 'YYYY-MM-DDTHH:MM:SS+Z:ZZ' So 7:30PM 6 July 2007 in UK is '2007-07-06-T19:30:00+01:00', the '+01:00' indicates that this is in a timezone one hour ahead of UTC or GMT in old money.  This is because of British Summer Time moving the clock one hour forward.  We could also represent this time as the following

• 2007-07-06-T20:30:00+02:00 - Paris summer time
  
• 2007-07-06-T11:30:00-07:00 - San Francisco summer time

However the following is not necessarily identical

• 2007-07-06-T19:30:00 - indeterminate time zone

Note all the fields other than the timezone must be present in a dateTime type.
In addition to dateTimes we can also specify duration types that take the form 'PyyYmmMddDThhHmmMssS'.  Apart from the initial P and the T is hours, minutes  or seconds are being specified all fields are optional.  This gives us time periods such as

• P7D - a week
• P1M - a month
• P1Y1DT1H1S - One year, one day, one hour and one second
• PT5M - five minutes

Note that unlike real life, 5 minutes does not mean wait until I am ready!

Now!

The time now can be obtained by using XPath 2.0 functions.  In January of this year XPath 2.0 finally became a W3C recommendation (W3C standard).  Oracle BPEL PM already implements many of the XPath2.0 functions (usually prefixed with  xp20).  The following functions (documented here) can be used in assign statements and xslt transforms to get the current date and/or time.

• xp20:current-dateTime() - returns a dateTime element that has the current date and time as formatted above.
• xp20:current-date() - returns a date element that has the current date as formatted above without the time portion.
• xp20:current-time() - returns a time element that has the current time formatted as above without the date portion.
  

So now we know what time it is!

Period of time later/earlier

So how do we calculate a period of time a given time period earlier or later than we actually have.  To do this in XPath 2.0 we could just use the addition (+) and subtraction (-) operators but they also have corresponding XPath functions that are available to use as shown below:

• xp20:add-dayTimeDuration-to-dateTime(dateTime as string, duration as string)
• xp20:subtract-dayTimeDuration-from-dateTime(dateTime as string, duration as string)

When using in an assign statement it will be necessary to use the 'ora:getNodeValue()' function to extract the underlying value from an dateTime or duration element.  So to get a day 10 days in the future from today we could use the expression

• xp20:add-dayTimeDuration-to-dateTime(xp20:current-dateTime(), 'P10D')

So now we can calculate dates relative to another date.

Calculating Differences

In XPath 1.0 this was tricky!  In XPath 2.0 it is built in functionality.  See this post by Ramkumar Menon for details of how to calculate a difference using XPath 2.0 in a stylesheet.  Make sure when doing this you set the versin of the stylesheet to be '2.0' or it won't work.  I haven't got this function to work in an assign statement yet, hence my custom XPath function that does work in an assign statement.

• ajr:getTimeDifference(earlier as string, later as string)

Answering the questions

So lets look at those questions again :

• Response must be given in 5 days.
• Response must be given in a time period 'P5D' or
• Response must be given by xp20:add-dayTimeDuration-to-dateTime(xp20:current-dateTime(), 'P5D')
• Product must be shipped to arrive by 12 October 2007.
  Assuming shipping takes 2 days
• Ship by xp20:subtract-dayTimeDuration-from-dateTime('2007-10-12T00:00:00', 'P2D')
• How old will a child born on 19th January 1995 be on 1 September 2007?
• Use a stylesheet set to version 2.0 to calculate (xsd:date('2007-09-01')-xsd:date('1995-01-19')) or
• use custom xpath function ajr:getTimeDifference('2007-09-01', '1995-01-19')
  

So now we can go and set timers or notification alerts for these dates and times.  So date processing isn't really as scary as it first seems.  So must go, Its dinnertime!

PS If someone can tell me how to get the xpath 2.0 operators working in assign statements I would be a happy man.

Changing SOA Suite HTTP Port

I just had a call from a colleague asking how to change the HTTP port of a developer install of SOA Suite.  He had a default installation on port 8888 and needed to change it to port 80 so that it would go through a firewall.  So lets look at the steps needed to perform this change.

Process Outline

Basically this process has two steps

1. Change the port number of the web server
2. Change any references to that port number elsewhere in the install

Sounds simple enough so lets see how it plays out.  I have to confess that this was easier in 10.1.2 but when the Application Server Control moved to JMX there was a limit on how much functionality could be migrated over in the available timescale, and opmn port changing was one of things that didn't make the release date.

Change the HTTP Listening Port

First we need to change the listener to listen on port 80 instead of the default port 8888 (or whatever port was free on your install).  the OC4J web listening port is set dynamically at run time by OPMN from an available port in a given range.  The range for the HTTP port is always set to be a single port number and this is the port number we must change.

• Stop the SOA Suite running
• In directory $ORACLE_HOME/opmn/bin execute command 'opmnctl stopall'
• Edit the file $ORACLE_HOME/opmn/conf/opmn.xml
• Find the entry '<port id="default-web-site" range="8888" protocol=HTTP/>'
• Change the range to port 80
• Save the file
• Start the SOA Suite running
• In directory $ORACLE_HOME/opmn/bin execute command 'opmnctl startall'

Verify that SOA Suite is now running on port 80.  Note that IE has a nasty habit of caching a page even if you do a Control Refresh, and so it may appear that your SOA Suite is running on two ports.  In Firefox hitting Control Refresh causes the old port address to come back with a server not found error.  Score one for Firefox!

Modify BPEL to use new Port Number

BPEL needs to know the port number it is listening on in order to construct SOAP endpoint addresses for use in WSDL and callbacks.  We need to tell BPEL that the port number has changed.

• Go to BPEL Admin 'http://localhost/BPELAdmin'
• Log in as 'oc4jadmin'
• Change the following entries to have port 80
• soapServerUrl
• soapCallbackUrl
• clusterName
• Click 'apply'
  

I don't think you really need to change clusterName but it will confuse the heck out of you if you don't!  I would bounce SOA Suite again to make sure the change get picked up.

Reload BPEL Processes

At this point I would reload any deployed processes, making sure that I revalidated them before deploying because any partner links referring to existing processes on this server will now be pointing to the wrong port!

Change Build Parameters

If you plan on using ant to deploy applications then update it to the new port number.

• Edit $ORACLE_HOME/bpel/utilities/ant-orabpel.properties
• Change the http.port property to the correct value
  

Now ant will deploy to the correct port on the server.
If you plan on deploying from the samples directory then remember to go through and change all partner link port numbers in the bpel.xml files.

Voila! the Job is Done!

That's all folks - simple really if you know where to look!
For other ways of doing this look in the documentation for 10.1.3 here.

A Tale of Two Installs

A regular question I get asked is what is the difference between the different SOA Suite installs and which is best for me.  I thought I would address that question in this entry.  The short answer is that one is intended for developers and the other is intended for production deployments.  However that over simplifies things.

Initial Choice

When starting the SOA installer we are faced with a choice between Basic Install and Advanced Install.  The basic install provides us with very few options of how we want things to be.  On the other hand it gives us most of what we want in a useful configuration.

Basic Install

The basic install option will install and configure the following components :

• OPMN - Oracle Process Manager and Notification Server
  This is responsible for starting and stopping other processes, monitoring processes for failure and if necessary restarting them.  It also tracks the current topology of the processes.
  
• OC4J - Oracle Containers for J2EE
  This is the Java container that is used to run the rest of the SOA Suite functionality.  In the basic install there is a single OC4J instance called home.  Remember that an OC4J instance is effectively a configuration unit that provides a J2EE container with a set of applications and may execute on multiple JVMs.  This instance hosts the following SOA Suite components
• BPEL Process Manager
• ESB - Enterprise Service Bus
  
• OWSM - Oracle Web Service Manager
  
• Rules
  
• AS Control -Application Server Control
• JSSO - Java Single Sign On
• Oracle Lite - embedded database for use as a repository.  Note that this is on Windows only, if installing on Unix/Linux then you will be asked to specify an external database in a similar way to the advanced install.
  

OC4J will listen directly for HTTP requests from browsers using its built in HTTP listener.

Advanced Install

The basic install option allows you little choice in how things are set up.  The advanced install gives more options, allowing you to install

• SOA Suite with a web server
• just a web server
• just a J2EE server
• or a web server and J2EE server without SOA Suite

If installing SOA Suite it also requires an external database to act as the repository.  When installing SOA Suite it creates two OC4J instances called home and oc4j_soa.  The home container will normally run the application server control and Java SSO components.  The oc4j_soa instance will have the same components as in the basic install but normally the AS control and the Java SSO will not be active in this instance.

The advanced install SOA Suite option also provides the Oracle HTTP Server (OHS) to act as the web server.

Choices, Choices

So which option should I install and why?  Lets look at different scenarios for each install.

Simple Developer Install

If I need a development environment then the Basic Install offers the smallest footprint and the fewest moving parts.  Everything needed is installed as part of the one click install process.  Oracle Lite can be used for simple data storage but it is missing many of the advanced features of a a full Oracle Database and is only single user.  Don't get hung up on the single user nature of Oracle Lite though, because in this case the single user is the OC4J home container!  Any serious database work will still need an external database.  I have not tried to repoint the basic install at a full Oracle database,  I would like to try this some time, basically it would mean installing the repository into a new database (see Marc Keldermans blog for a succinct summary of how to do this) and then modifying the data sources in OC4J to point to the new database repository.

Advanced Developer Install

If I need access to an Oracle database then I can either perform a Basic Install and use an additional Oracle database or I can do an Advanced Install and have the repository in the the same database as the rest of my application is using.

Production

For production use it is absolutely essential to use a database other than Oracle Lite.  The only fully supported way to achieve this is to use the Advanced Install option.  If the installation is going to be accessed by the web from outside the firewall it is also imperative to use the Advanced Install option as all Oracles penetration testing is done with this configuration rather than against the embedded HTTP Server in OC4J.  If necessary the footprint of the Advanced Install can be reduced by not running the OC4J home instance.

Summary

The different SOA installs have somewhat different architectures but clear intended usage.  A Basic Install is really only intended for development use whilst an Advanced Install can be used for development, testing or production.  Understanding how the architecture is put together will hopefully help you make intelligent informed choices about the configurations you require.

Building a SOA Development Environment

Just had to build a new SOA development environment and so thought I would share with you the steps I went through to create it.

Planning

I wanted a SOA Suite 10.1.3.3 install with an Oracle XE database for use as a meta-data repository and for use as an application database.  I also wanted to install JDeveloper into the environment so that I had everything I needed.  I was also keen to be able to produce dashboards with BAM and model my processes using BPA Suite.
I acquired the components I needed by downloading them from OTN.

• JDeveloper 10.1.3.3 (Studio Install)
• SOA Suite 10.1.3.1 (CD 1)
• BAM 10.1.3.3
• SOA Suite 10.1.3.3 patchset
  
• Oracle XE
• BPA Suite 10.1.3 (CD 1)

Operating System

In my case I was going to be running on Windows Server 2003 hosted in a VMWare environment.  I installed W2K3, making sure that I added a loopback adapter with a fixed IP address and that I enabled IIS for later use by BAM.

Setting Up Database

First step was to install Oracle XE and configure it for use by SOA Suite.  Installation was painless and I then configured it for use by SOA Suite as detailed in another blog entry "Making DB XE Work with SOA Suite".

Setting Up JDeveloper

Next up I installed JDeveloper.  I installed this next so that I could use the JDK it provides later when running the repca utility.  I created a shortcut on the desktop and also put a link onto the task bar to make it easier to find.

Setting up SOA Suite

I first went to the SOA Suite installation media and ran the irca.bat utility found in 'Disk1\install\soa_schemas\irca'.  This creates the SOA schemas in the XE database, assuming of course that you tell it where the database lives.  To run this you need a JVM, hence why I installed JDeveloper before SOA Suite.
I then ran the SOA Suite 10.1.3.1 installer and selected the advanced option and chose J2EE and SOA Suite (see previous entry "A Tale of Two Installs" for explanation of what this installs).  I chose to make this an "Administrative Instance", meaning that it installed App Server Control so I could manage the instance.  I chose not to use the Oracle HTTP Listener to cut down on the memory footprint.
After installing SOA Suite I then patched it to the 10.1.3.3 level.  This required me to run the run the patch set installer and then run the update repository script upgrade_10131_10133_oracle.sql found in '$ORACLE_HOME\bpel\system\database\scripts'.  I also updated PHP in Apache to use PHP 5 rather than 4.
A restart of the SOA Suite and everything was now working.

Adding Business Activity Monitoring

Before installing BAM I made sure that IIS was working on my machine and also checked in IIS Manager that both Active Server Pages and ASP.Net were enabled.  I then ran the BAM installer and installed BAM. This requires two steps, one to install the Oracle client libraries and another to install BAM itself.  Note that if you only plan to use BAM with BPEL or via webs services there is no need to select the Enterprise Link option.

System Startup Choices

Under Windows all the SOA Suite services and the XE database appear as services.  Apart from the TNS listener I set all my Oracle services to be manual startup so that I could control what I was actually using.  I could then startup first the database and then either just SOA Suite and BAM or both together.  All components can be started from their 'Program Files' menu.

Installing BPA Suite

Almost there now.  I installed BPA Suite, pointing it at the XE database.  There was a minor problem which was that I had accepted the default western european character set (WE8MSWIN1252) for the database and BPA Suite was demanding that I use a multi-byte character set (AL32UTF8).  I fixed this problem by looking at the location of the character set test script in $ORACLE_HOME\testsql.txt (it is called check_charset.sql).  I then went to the temporary directory specified and changed the check_charset.sql file so that the line
became
Note that this is not supported and will cause problems if you want to use characters not supported by WE8MSWIN1252.  However if you are in Western Europe then it is a quick hack to avoid having to re-install XE.
If like me you want to publish BPEL models from BPA Suite into JDeveloper then you need to install the BPA Suite extensions into JDeveloper.  To do this

• Select Help->Check for Updates
• Check the "Install from Local File" option and browser for the pcbpel_bundle/pcbpel_bundle.zip file on the BPA Suite installation media.
• Click Next
• Click Finish
• Say Yes to restart
• Say yes to all overwrite warnings or click the don't tell me again box!
• Say no to import settings
  

Now BPA Suite is installed and integrated with JDeveloper.

Final Configuration

I find it very handy to have links to all the SOA Suite consoles directly available from the favorites menu, so it is probably worth adding them there.
You will also want to go through and set up connections in JDeveloper for

• Application Server
• Integration Server
• BAM Server
  
• BPA Server - choose local server for the connection.  You can only set this up after creating a "database" in BPA Suite.
  

Note that the relevant components must be running to test the connections (or to set it up at all in the BPA case)

Voila! The job is done

Seems like a lot to do until you realise that you have installed a huge amount of function that should fit all your development needs.  In addition to SOA Suite and BPA Suite and Database XE and JDeveloper you also have Application Server and WebCenter installed!  So I'm off to write something with all that technology, enjoy!

A Quick Response

Returning an Immediate Response from an Asynch Process

I just got asked by a colleague how to return an immediate response from a BPEL process created using the template "Asynchronous BPEL Process".  Often we want to return some data to the caller of a process to indicate that the request has been received and is being processed.  To do this we need to add an immediate response to the caller in addition to calling them back later when the process is completed.  For example consider a process to make a booking for some resource.  The process might be lengthy and the caller may need some mechanism for canceling the process before it completes.  In this case returning an immediate response could give the caller a handle allowing him to call back into the specific process instance to cancel it.
So lets look at how we alter the template to allow for this immediate response.

Create the Project

First we create our process using File->New Project in JDeveloper and choose a BPEL PRocess Project and select the "Asynchronous BPEL Process" template.

Adding a Response Element

Once we have created our project we then need to add an immediate response element to the XSD.

Adding a New Message Type

Having created a new immediate response element to the XSD we then need to use that element within a new message type in the WSDL.

Adding an Output to the Initiate Operation

Within the port type for the process we need to modify the "initiate" operation to return an immediate response of the message type we just created.

Adding a New Variable to the Process

We need to add a new variable to the process to hold the immediate return value.

Add a Reply to the Process

We can add the reply operation to the process and wire it up the client partner link.

Complete Process

Finally we need to complete the process.  I will often return the process instance ID as a return value.  This can then be used a correlation token if the client needs to call into the process again before it completes.

Once the process is finished being written it can be deployed and executed.

Remind Me Why Again

So adding an immediate response lets us confirm to the caller that we have received their request and also gives them a token to call back into the process later.  A complete example is uploaded here.  As you can see it is very easy to do and personally I feel all "asynchronous" processes should give some immediate response.

Feeding Time

How to Feed Oracle BAM from IBM MQ Series

I'm working with a customer who has IBM MQ installed and wants to intercept messages from MQ and deliver them to Oracle BAM.  Thought you might be interested in how I set things up in my environment to try this out.

Setting Up MQ Series

To begin with I installed MQ Series, I installed Server as I was running on a single machine but it should work fine with client as well.  I also installed the 6.0.2 update for MQ and the Pub/Sub plugin for Eclipse.  After a default installation of MQ Series I needed to set up publish subscribe.  This was achieved as follows:

• I used the MQ Explorer to start the Message Broker.  By default it is not started.  To start it I went to 'Queue Managers/QMgr_Name/Advanced/Services', selected 'Show System Objects' and started the 'System.Broker'.
• I created the JMS queues needed by the JMS pub/sub interface by running the command 'runmqsc QM_w2k3 < MQJMS_PSQ.mqsc' in the $MQ_HOME/java/bin directory.
  

I then needed to create a JNDI tree with links to appropriate MQ objects.  I started by creating a directory for my JNDI entries (I used C:IBMJNDI) and then I configured the JMS Admin tool by editing $MQ_HOME/Java/bin/jmsadmin.config to have the following entries:
This uses a file based JNDI providor rooted at C:IBMJNDI.
It is sometimes handy to use the JMS Admin tool but the new JMS support in Eclipse makes it easier to create entries in JMS using that tool rather than JMS Admin.

• To set up MQ Explorer for JMS navigate to 'JMS Administered Objects' and select 'Add Initial Context ...'.  Then I selected the File Based provider and chose the directory I wanted to use.

I then used the Eclipse JMS Admin tool to create bindings for the Topic Connection Factory and the Topic I was going to use from BAM.

• Create the TopicConnectionFactory entry by navigating to 'JMS Administered Objects/location/Connection Factories' and right clicking to get 'New->Connection Factory ...'
• Give the factory a name and make it a TopicConnectionFactory.

Now I had MQ installed and the broker running with a TopicConnectionFactory registered in the JNDI naming tree I had to set up a Topic.

• I navigated in MQ Explorer to 'JMS Administered Objects/location/Destinations' and right clicked to choose 'New->Destination ...'.
• I gave the topic a name - myTopic.
• Selected the type of destination to be a Topic.
• On the General property screen I gave a name for the topic within MQ series - this can be the same or different to the topic name given previously  Note that MQ Topic names are recommended to use '/' as a separator for readability.  If this is desired for the JNDI name then sub contexts should be created to give the same effect.
• On the Broker property screen I selected my Queue Manager and then I selected my stream to be the 'SYSTEM.BROKER.DEFAULT.STREAM'

The new support in Eclipse for Pub/Sub makes it so much easier than the old way of creating manually hooking up streams and queues to the broker.  Well done to the MQ Eclipse plug in team at IBM.

Registering MQ Series with BAM

With MQ configured the way I needed it the next step was to register the MQ libraries with BAM as a resource providor.  First within BAM Administrator I modified the Websphere MQ Enterprise Message Source Type to point to my MQ installation, at the same time removing the non-Websphere references from the startup parameters.  This left me with the following in startup parameters:
I then went into BAM Architect to define my message source.  I created a new Websphere MQ message source and provided the following settings:

• Initial Context Factory: com.sun.jndi.fscontext.RefFSContextFactory
• JNDI Service Provider URL: file:/C:/IBM/JNDI
  
• TopicConnectionFactory Name: mqTCF
  
• Topic Name: myTopic
  
• JMS Message Type: Text Message
  

I set a string message size of 2000 bytes as I know they are small messages.

Processing JMS Messages in BAM

Everything is now ready to start receiving messages in BAM.  For details of how to process JMS messages in BAM look this tutorial.

Important Note

Enterprise Link only appears to support Topics for subscription, not queues, so if you need to process queues then you will need to take messages off the queue and post them onto a topic.  One mechanism for doing this is the JMS Router in the oracle application server.

Hopefully this will fill a hole in how to get data from MQ series into BAM.  Have fun!

A Problem of Aggressive Cleanup

Just been working on an internal proof of concept and came across some very irritating behaviour in BPEL Designer in JDeveloper 10.1.3.3.  I deleted a Partner Link and accepted the suggestion of JDeveloper BPEL Designer to delete the associated WSDL file.  Unfortunately it turns out that this deletes not only the partner link specific WSDL file but also a shared WSDL file (DBAdapterOutboundHeader.wsdl) that is used by all DB Adapter Partner Links in the project.  Net result is that this breaks all the remaining DB Adapter based Partner Links in the project.  Fortunately there is a simple solution.  Copy the DBAdapterOutboundHeader.wsdl from any other project with a DB Adapter in it into the project/bpel directory and your problems will be solved.
For a view of some of the problems this may cause have a look at this blog.

Threading It All Together

or Thread Usage by BPEL Processes

Based on comments I have received and questions I have been asked it seems that when it comes to thread usage by BPEL processes there are lot of confused people around.  So I thought I would unravel a few mysteries about threading in BPEL.  Understanding the threading model is important for BPEL because the thread model can affect scalability of processes and the BPEL engine itself.
How threads are allocated depends on the interaction pattern of the link activating the receive activity.

Interaction Patterns

The interaction pattern can be thought of as how the BPEL process is called.  There are two types of interaction pattern, one-way and two-way or request-response as it is sometimes known.

One-Way Interaction

A one-way interaction pattern means that the process is invoked and is then left to run, possibly returning the results via another interaction, possibly not.  We often think of this as asynchronous because the caller does not wait for the process to complete.  Don't confuse the actual interaction pattern with the way the process works.  For example many processes have a request reply process model that is implemented through two one-way interactions.  The client calls the process and continues working.  The process does its job and then calls the client to pass back the results.  The process has a request-reply model but it works through two one-way interactions.  A one-way interaction pattern is characterised by a process that has a receive activity but no corresponding reply activity.  A request-reply process using one-way interaction patterns would be characterized by a receive activity with no corresponding reply but it would have a corresponding invoke.

Two-Way or Request-Response Interaction

A request-response interaction pattern means that the process is invoked and the caller then waits for the process to return a result before the caller continues its own processing.  We often think of this as synchronous because the caller appears to get the result immediately.  A request-response interaction pattern is characterised by a process that has a receive activity with a corresponding reply activity.   Note that it is possible to have a process that combines an initial request-response interaction to return an initial result, such as a correlation token, with a later one-way interaction that returns the final result to the caller.
More information on what this looks like is available in the BPEL docs.

Instrumentation

To investigate thread usage in BPEL processes I wrote a small amount of Java code that is embedded into the process inside an exec activity.  The code is shown below and stores the currently executing thread group and thread name into a variable called "CurrentThread".
  <bpelx:exec name="Java_Embedding_1" language="java" version="1.3">
    <![CDATA[
      Thread t = Thread.currentThread();
      setVariableData("CurrentThread",
                      t.getThreadGroup().getName()+":"+t.getName());
    ]]>
  </bpelx:exec>
This variable is then surfaced in the output of the process.

Simple Request-Response Thread Usage

My first test was to run a simple request-reply interaction pattern in a process.  In this case the  thread is taken from the inbound HTTP listener thread pool and a single thread is used to receive the request and process it.  The thread group is "HTTPThreadGroup" and the actual thread I got was "AJPRequestHandler-RMICallHandler-56".  The last digits vary.  So in this case the BPEL server receives the request on the same thread that is used for all processing within the process.  This process is available as ThreadExplorer1.zip.

Simple One-Way Thread Usage

My next test was to run a simple one-way interaction pattern, with another one-way onteraction to return the result.  in this case the message is received on one thread and then passed onto a pool of worker threads to execute the BPEL process.  The thread group is "main" and the actual thread I got was "WorkExecutorWorkerThread-120".  The last digits vary.  So in this case the BPEL server received the request and then queued it for subsequent execution.  This allows for greater scalability by limiting thread usage within the BPEL server.  This process is available as ThreadExplorer2.zip.

Combination of Request-Reply and One-Way Thread Usage

The obvious next question is what happens when I have both interaction patterns in the same process.  Well that is what I did in my next test.  I created a process that returns an immediate result (request-response interaction) and then later posts a further response as a one-way interaction.  In this case the message is received on the listener thread and processed on that thread up to the reply.  At that point the remainder of the work is passed off to a worker thread.  This is exactly as one would expect from combining the two previous usages.  This process is available as ThreadExplorer3.zip.

Effect of Flow on Threading

The next question is what happens in a flow activity.  Surely this will cause multiple threads to be spawned.  Well by default the answer is no.  ThreadExplorer4 process uses a request-reply interaction and within the processing it has a 3-way flow statement.  All statements in the flow are executed in the same thread.  This is exactly the same as if we had used a request-response interaction with a while activity instead of a flow.  This may be counter-intuitive but the reasoning behind this is as follows:

• For scalability we want to limit thread usage
• A flow is really intended to support waiting for multiple events such as responses from services with two one-way interactions and so we don't need to execute them in parallel, pseudo-parallel will do.
• If one branch of a flow is waiting for an event the same thread can start executing the next branch of the flow, avoiding the need for a JVM  thread context switch.

So there is little real need for a truly parallel flow statement.  This process is available as ThreadExplorer4.zip.

More Detail of Effect on Flow on Threading

If we put a wait or receive into our flows then we find that the thread itself is not suspended, but is returned to the thread pool.  If there are other legs of the flow capable of execution then the thread will be used to execute those before being returned to the worker pool.  This same behaviour applies when a leg of a flow wakes up due to an event such as a timer (wait activity) expiring or a message arrive (receive activity).  This is all shown in ThreadExplorer5 process which has a 3 way flow that uses the initial worker thread to put all three legs into a wait activity and then uses a different worker thread when they wake up to carry on their processing.  This process is available as ThreadExplorer5.zip.

Creating True Parallelism in an Flow

One common use case for flow is to make multiple request-reply calls in parallel to reduce their latency.  But as we saw in the previous section this doesn't work out of the box so how can it be achieved?  If we add a property "nonBlockingInvoke" to the partner link we are calling and set it to "true" then BPEL will use a seperate thread to make the call.  This is shown in ThreadExplorer6.  Note that when you run this the initial request is being processed on the thread that received the message, a listener thread.  When we do the invokes on a partner link with nonBlockingInvoke property set to true we see that those invokes are processed in a seperate thread.  Because we are calling another BPEL process the server does some optimisation and uses the same thread for all three invokes.  If the partner link were on a different server then 3 threads would have been used.  Note that these threads are all worker threads, not the thread from the listener.  Finally when the three flow legs join together a new worker thread is used to carry on processing.  This process is available as ThreadExplorer6.zip.
You can read more about the nonBlockingInvoke in the BPEL documentation.

More on Thread Processing in BPEL

You can read more about thread processing in BPEL in the documentation which gives more reasoning for the way the cases above behave as they do.  Hope this was useful to you.

Stringing Together XML

Someone just asked how to convert a string into XML.  A common use case is to retrieve XML stored as a string in the database.  Although you could argue that it would be better to store XML using the XML facilities of the Oracle database the fact is we still see lots of XML stored as VarChar2 or CLobs.  So how can we read these into BPEL?  Well in addition to explaining this I thought it was also a good excuse to explain the use of the append rule within an assign and also a use for the anyType.

Reading XML

Fortunately the developers foresaw the problem of converting strings to XML and have provided a handy function ora:parseEscapedXML which takes a string as a parameter and returns an XML document.  The string can be either a full XML document or just a document fragment.  Similarly it can have an explicit namespace or no namespace.
I have uploaded a simple BPEL process that takes an XML string as input and returns the DOM representation of that string.

Using an anyType

For the input I take a string type but for the output I have used an anyType.  This allows me to legitimatly assign it any value.  Note that although anyType is not supported by the graphical XPath builder in the XPath wizard it is supported by the editor, which will drill you down to the anyType.

Using Append instead of Copy

The other thing I have done in the assign is to use an append rule to add the parsed XML underneath the base output element.  This allows us to see the entire result.  If I had used a copy rule then the top level element of the parsed XML would have been thrown away and only its contents copied.  Using the append rule causes the entire subtree selected by the XPath expression to be placed underneath the target element.  Because the target is an anyType I don't have to worry about the XML conforming to any particular schema, I can dump any well formed XML in there.

Testing it Out

You may want to try it on the following input data to see how it works.  Note that there is no difference in the way documents and document fragments are handled.  the only requirement is that the XML be well formed.

Fragment Without a namespace

<NAME><FIRST>Antony</FIRST><LAST>Reynolds</LAST></NAME>

Fragment With a namespace

<NAME xmlns="http://blog.oracle.com/reynolds"><FIRST>Antony</FIRST><LAST>Reynolds</LAST></NAME>

Document Without a namespace

<?xml version="1.0" encoding="UTF-8"?><NAME><FIRST>Antony</FIRST><LAST>Reynolds</LAST></NAME>

Document With a namespace

<?xml version="1.0" encoding="UTF-8"?><NAME xmlns="http://blog.oracle.com/reynolds"><FIRST>Antony</FIRST><LAST>Reynolds</LAST></NAME>

Finding the End(point)

or The Difference Between Deployed and Undeployed BPEL WSDLs

A colleague just complained to me about his inability to create an ADF business object from a BPEL process.  He explained that when he pointed the JDeveloper wizard at the BPEL WSDL is gave an unknown endpoint error.  When I asked if he had pointed it at the file from the JDeveloper BPEL project rather than the WSDL file on the BPEL server he confirmed that this was indeed the case.  Pointing at the WSDL file on the server solved the problem.  Why is quite instructive.

A Tale of Two WSDLs

When you build a new BPEL process, you generally end up with a new WSDL file that defines the inputs and outputs of the BPEL process.  What this WSDL file is lacking however is any endpoint information.  Endpoint information details the logical network name and port number of the server on which the process is running, as well as the path to the process.  This data cannot be filled in until the process is deployed to a specific domain (part of the path of the request) and server (hostname and port number part of the request).  Hence when retrieving the WSDL from the JDeveloper file, you have no location where the process is available and it is this lack of data that the ADF wizard complains about.  When you retrieve the WSDL from the BPEL server then it includes the location information of the process and hence the ADF wizard is happier.
Obvious when you think about it, but it is a problem that plagued many people in the past so I thought I would share it with you.

More on Time!

Seems that dateTime is one of the most frustrating types in XML.  I just had a colleague ask how to convert a string to an XML dateTime type.  There are a rich range of functions to convert dateTimes to strings, but a paucity to do the reverse.  So I present here one approach to this.  I use XSLT variables to help in the parsing.  Here is the XSLT fragment that converts a US formatted date (MM/DD/YYYY) to an XML dateTime format (CCYY:MM:DDTHH:MM:SS).  Note that in an XML dateTime all fields are mandatory, the only optional pieces are the decimal part of the seconds (CCYY:MM:DDTHH:MM:SS.SS) and the timezone portion (CCYY:MM:DDTHH:MM:SSZ or CCYY:MM:DDTHH:MM:SS+02:00).
In the fragment below I take the input data from XPath expression /DateList/SingleDate/DateField.  This has the date in format MM/DD/YYYY.  I use this input to populate an element dateitem which is of dateTime type.
Lets go throught the XSLT a little at a time.

        <dateitem>
          <!-- Parse Date of format MM/DD/YYYY -->
<!-- I declare a variable TempInput to hold everything after the first '/' in the date (DD/YYYY). -->
          <xsl:variable name="TempInput">
            <xsl:value-of select='substring-after(/DateList/SingleDate/DateField,"/")'/>
          </xsl:variable>
<!--I set a variable Month to hold the month, generated by picking up everything before the first '/' in the date (MM). -->
          <xsl:variable name="Month">
            <xsl:value-of select='substring-before(/imp1:DateList/imp1:SingleDate/imp1:DateField, "/")'/>
          </xsl:variable>
<!--I define a variable Day to hold the day portion which I get by picking up everything before the '/' in my Temp variable (DD). -->
          <xsl:variable name="Day">
            <xsl:value-of select='substring-before($TempInput, "/")'/>
          </xsl:variable>
<!--I then pick up everything after the '/' in Temp variable (YYYY) and put it in a Year variable. -->
          <xsl:variable name="Year">
            <xsl:value-of select='substring-after($TempInput, "/")'/>
          </xsl:variable>
<!--Finally I construct an XML dateTime format by concating the Year, Month and Day variables in the correct XML format. -->
          <xsl:value-of select='concat($Year,"-",$Month,"-",$Day,"T00:00:00")'/>
        </top:dateitem>
Obviously the parsing could be made to handle time formats and timezones as well.
Hoping that this helps someone struggling with parsing text strings into dates.

I have created a simple ESB project that takes data from a file in US date format and loads it into two columns in a database table, one formatted as a string the other as a SQL Date.  When using the Database adapter the SQL Date format gets converted to an XML dateTime.  The project is available for download here.

Registering Interest

I have been involved in some interesting discussions recently on the use of a service registry.  Oracle SOA Suite ships with a limited use license for a service registry and there is the option to upgrade it to a full use license and so the question comes up, Antony why would we want to use a registry and can we use it for all our service lookups.
Lets start by looking at what a registry is.

Glorified Yellow Pages

Uncharitably a registry may be described as a glorified yellow pages.  It allows artifacts such as XML schema and service WSDLs to be stored in a searchable, categorized archive.  Artifacts are stored under different categories and may have keywords associated with them to assist in searching.  So at the end of the day the registry is just a repository of meta-data about artifacts.  In the same way a car is just a large amount of beaten metal with a power unit that drives wheels.  Calling it a repository of meta-data does not actually explain what it does or how it may be used.

Registry use Cases

Let me suggest a few registry use cases

• Design Time Service Discovery
  A registry can be used to catalogue existing services and associated artifacts.  This encourages re-use by making it easier to discover existing services.  The ability to promote items between registries also makes it possible to put in place a service approval process that vets new services before making them available to developers.  This again promotes re-use and generalisation of existing services.
  
• Run Time Service Discovery
  A registry can also be used at runtime to provide the physical endpoint for a service.  This makes it easy to change the physical provider of a particular service.  This also simplifies migration of services between development test and production environments as outlined in the next use case.
  
• Service Migration Mechanism
  The use of multiple registries provided a managed path for services to be promoted between environments, either from a development perspective or from a runtime perspective.

Design Time Considerations

Use of a registry at design time is generally a good idea but it does a require a certain amount of discipline in its use otherwise it becomes yet another dumping ground for all design decisions, good and bad!  However individual develpoment team can maintain their own local registry that has a well defined promotion process to a central registry, allowing teams to work on their development services which may later be migrated to a corporate registry.  In my experience most customers are not making use of a registry even in their design time environments and this is probably the best place to start using a registry.

Run Time Considerations

Using a registry in a runtime environment promises a greater degree of de-coupling.  However a similar amount of decoupling may be achieved through the use of an ESB alone.  If a registry is used to look up an ESB endpoint then there is a potential cost to be paid in terms of an additional lookup.  If an ESB endpoint is not looked up then there is the risk of coupling the data formats of unrelated services together, losing the use of an ESB to provide message transformation to/from canonical form.  Some informal tests I ran indicated that the additional overhead of a registry lookup does not add much to the service invocation, but in high volume environments it may be the straw that breaks the camels back.
A good policy may be to begin using a service registry in design time, later trying it out in non-high volume environments.  Using an ESB can make this migration to registry use easier, if less pure by having the ESB perform the registry lookup.

Using a Registry with Oracle BPEL PM

The current production release of Oracle BPEL PM has built in support for use of a registry.  Al that is required to make a service lookup occur through a registry is to perform the following.

1. In the BPEL Console, go to the "Manage BPEL Domain" and set the following properties
• uddiLocation - the inquiry address of the registry
• uddiUsername - if it is a secured registry set it to a username for performing lookups, if not set it to urn:unknown
• uddiPassword - if it is a secured registry set it to a password for performing lookups, if not set it to urn:unknown
2. In a BPEL process for each service endpoint (partner link) that you want to go through a UDDI lookup
• Add a property "registryServiceKey" to the partner link with the "Entity Key" value assigned in the UDDI repository

For more on potential performance impacts of registry lookup see Chintan Shahs blog entry. Note that I haven't seen such a bad degradation as he reports, but I could believe it for a complex WSDL and high system load, meaning unresponsive threading, particularly in a Windows environment.

An Example of Dynamic Lookup

To give a feel for dynamic lookup I have uploaded 3 JDeveloper Projects in the this file.

• JavaWS includes a simple Java web service (GreetingWS) that can be registered in a service registry.  When registering it in the registry you will need to note the service key and copy it into the partner link property in Dynamic BPEL.
  
• DynamicBPEL is a BPEL process that looks up the web service (GreetingWS) and invokes it in three different ways
1. Static, it uses the endpoint defined in a WSDL file
2. Dynamic, it sets the endpoint explicitly (passed in as a parameter to the process), showing how you can be very dynamic in the endpoint you invoke.
3. UDDI, it uses the UDDI repository to lookup the endpoint, showing how unintrusive it is on the rest of the process.
• TestDynamic is a BPEL process that you invoke to test the DynamicBPEL process.  It iterates over all three methods and calculates how long each method takes to complete the given number of iterations.  The first invocation is not counted to allow the system to "warm up" for each call.
  

More on Batch Processing in BPEL

I was asked a follow up question recently on my entry about batch processing in BPEL.  The individual had implemented a BPEL process similar to the one shown below.

A Use Case

The process needs to refresh the corporate phone book.  It does this by reading a file with the new phone details, deleting the old phone details from a database and then inserting the new phone details into the database.
A file is read into the process via ReceivePhoneBook activity.  The ClearPhoneBook activity uses custom SQL to truncate the phonebook table.  Finally after transforming the file format to database format the new phone details are inserted into the table using the InsertIntoPhoneBookActivity to insert multiple records into the table in a single call.

A Fly in the Ointment

This process works fine as long as all the records are received in a single message.  If they are received in multiple messages then not all the records will be written as multiple processes will receive part of the input file, but all the processes will truncate the table, resulting in the possibility of some records being inserted by one process and deleted by another process that starts a little later.  This is illustrated in the diagram below which shows three processes each processing a portion of the file.

Note that records 1-10 are written to the database at the same time as the table is truncated by the process receiving records 11-20, leaving the possibility that the write will complete only to be overridden by the truncate.  If records 1-10 are not overwritten by the second process they will definitely be overwritten by the third process which only starts truncating the table after records 1-10 have been inserted into the table.

A Solution

What we need to do is to separate the deletion of the existing data from the insertion of the new data.  We can do this by moving the truncation of the table (deleting the existing data) into a separate BPEL process.  Such a process is shown below.

For this to work we need to know when a new file starts to be loaded so that this process can be invoked before any record processing is performed.  Now there are lots of complicated ways that we could do this with singleton processes to maintain state and complex logic to make sure it all works.  Or we could use a newish feature of the BPEL process manager (introduced in 10.1.3.3 I believe) to get it to invoke our process.

Batch Manager

In the previous discussion I ignored the partner link that initiates the record deletion process.  This partner link implements the Batch Manager interface as specified in $ORACLE_HOME/bpel/system/xmllib/jca/BatchManager.wsdl.  To create the above process first create a new empty BPEL process and then in the services stream right click and select new partner link.  Click on the icon to browse for files from the local file system and select the BatchManager.wsdl file and choose to implement the BatchManagerInterfaceRole (i.e. choose this as "My Role" ).  This is the interface used by BPEL Process Manager to notify a process that a file has started to be read.  There are several methods available to receive different notifications

• onBatchReadStart - tells when a file is started to be read
• onBatchReadComplete - tells when a file has finished being read
• onBatchReadFailure - tells when a record or records cannot be processed by the adapter framework
  

For this scenario we are interested in onBatchReadStart.  Receiving this notification we know to truncate the table ready to receive the records.

Who to Tell?

How does the BPEL Process Manager know to call the notification process?  The answer is that it is configured as an activation agent property in the bpel.xml file of the process receiving the records from the file adapter.  To set up the notification it is necessary to add the following property tag to the bpel.xml at XPath location BPELSuitcase/BPELProcess/activationAgents/activationAgent :

• <property name="batchNotificationHandler">bpel://default|FileNotificationProcess</property>
  

Note that default is the name of your domain and FileNotificationProcess is the name of your process.  Adding this property will cause the BatchManager interface on the given process to be invoked when a file is read.

Final Steps

With notification configured we now need to modify our BPEL process to not truncate the table because this is being done via a separate process.  We also need to add a delay to the process to avoid race conditions that could cause records to be inserted into the database before the database has been truncated.  The modified process is shown below, complete with a 30 second delay to avoid problems with multiple processes being invoked at the same time.

A Worked Example

I have created a sample to let you explore how all this works.  To set it up do the following.

1. Download the project files in FileManipulation.zip.
2. Unzip FileManipulation.zip - this will create a FileManipulation directory with 3 sub-directories and a JDev 10.1.3.3 workspace
3. Open the workspace FileManipulation.jws in JDev 10.1.3.3
   
4. Create the following directories or modify the file adapter partner links in BigBatchProcess1.0 and BigBatchProcess1.1
   
• C:FilesInbound
• C:FilesOutbound
5. Create a test user in the database by running the script in FileManpulation/BigBatchProcess1.0/src/CreateUser.sql as a system user
6. Create a database connection in JDev called TestDS to connect to user Test (password test) in the database.  You may need to rerun the adapter wizard if you are not running XE database on port 1521
7. As user test run the script in FileManpulation/BigBatchProcess1.0/database/CreateTable.sql to create the phonebook table.
8. Deploy the BigBatchProcess1.0 to the BPEL process manager.
9. Test it by copying file FileManpulation/BigBatchProcess1.0/src/PhoneBook1.csv to C:FilesInbound.
10. Verify the number of records stored by executing command in database "select count(*) from phonebook.  There are 1000 records in the source file, you will probably receive a count less than 1000 due to race conditions around truncating the table and inserting records into it.  This is the problem we are trying to avoid.
    
11. Deploy the BigBatchProcess1.1 to the BPEL process manager as version 1.1.
12. Deploy the FileNotificationProcess1.0 to the BPEL process manager.
13. Test it by copying file FileManpulation/BigBatchProcess1.0/src/PhoneBook1.csv to C:FilesInbound.
14. Verify the number of records stored by executing command in database "select count(*) from phonebook.  There are 1000 records in the source file, you should now receive a count of 1000 in the database table, indicating thqat we have solved the problem.
    
15. Pat yourself on the back and have a nice drink.

Hope that some of you find the above useful.

Throttling Files

I'm sure everyone has been tempted to grab a file by the throat and squeeze it until it departs for that great filing cabinet in the sky, but that is something to discuss quietly with your psychiatrist.  In this entry I would like to investigate how to limit the concurrency of file handling procedures.

A Concurrency Problem

By default the interaction with the file adapter is a one way interaction.  The file adapter posts a message into the queue for a process and then carries on about its business.  This is good because it allows multiple files, or multiple batches from a single file to be processed concurrently.  However if there are a large number of concurrent processes started then this can cause performance problems as the process manager starts to worry about which process to execute when.  One sympton of this is undelivered messages visible on the BPEL console.

This is bad because it means we are feeding BPEL process manager faster than it can consume, and so it gags a little and throughput is reduced as it gags.

Stopping Force Feeding

So how do we tell the file adapter that BPELs mouth is full and please wait until I have eaten this bit.  Well the answer is in the observation that interactions with the file adapter are usually one way.  They do not need to be!  It is possible to modify the WSDL generated by the file adapter wizard to support a two way interaction.  This causes the file adapter thread to wait until the reply before sending another message.  For the reasons why changing from a one-way to a two-way interaction causes this behaviour look at my earlier entry on BPEL threading.  The answer is left as an exercise to the reader.

Changing the File Adapter WSDL

To change the one-way interaction to a two-way interaction we do the following in the generated WSDL file:

• Make sure that the definitions root element has a namespace reference to XML schema by adding the following
• xmlns:xsd="http://www.w3.org/2001/XMLSchema"
• Add a dummy message that will be used in a reply.
• <message name="Dummy_msg">
      <part name="PhoneRecords" type="xsd:string"/>
  </message>
• Add a reply to the read operation
• <output message="tns:Dummy_msg"/>
• Add a reply to the binding
• <output/>

This gives a file like the one below

Modifying the BPEL Process

Having modified the WSDL we must modify the BPEL to now do a reply.  The reply will release the file handling thread in the adapter to do further work.  The modified process is available as project BigBatchProcess1.2 in the FileManipulation.zip file.  Details of using this process can be found in an earlier entry on batch processing of files.  Note that there is an additional test file included in the src directory - phonebook2.csv - that has 5000 records in it to give a larger test case.

Other Scenarios

The above scenario is not the only one where we may wish to use a two-way interaction with the file adapter.  The following additional scenarios may also occur.

Reading a File Sequentially

We may need to process a file in strict record order yet still want to have it batched because it is a large file.  In this case the problem is that we may have multiple batches submitted concurrently and there is no guarantee of the order that they will be processed in.  In this case just making the interaction two-way is not enough because there may be more than one file processing thread.

Controlling the Number of File Processing Threads

The number of threads used to process files in the file adapter is controlled by a setting in the file $ORACLE_HOME/bpel/system/services/config/pc.properties.  To alter the number of threads change the following property:

• oracle.tip.adapter.file.numProcessorThreads=1

Note that this is a global setting and so may impact other file activation agents that have been configured in the system.  The net result of this is to limit file processing to a single thread.  Hence by using a single thread and a two-way interaction we can single thread the file processing and so guarantee processing of records in order in a file.

Processing Files in Date Order

Another scenario where these techniques are applied is when there is a requirement to process files in order of last modification date, for example when later files may contain reversing transactions for earlier files.  This can currently only be achieved in 10.1.3.1 via a patch but it should also be available at some point in 10.1.3.3 and later releases.

Final Thoughts

The use of two-way interactions and limiting threads processing files is very powerful but it needs to be considered in the context of other file interactions in the system.  One way to limit the impact is to use singleton processes to receive the processing requests, enabling messages to be queued in order, releasing the file processing threads for other processes to use whilst still ensuring correct ordering in the required processes.
Hope this helped.

Closing the Transaction

Just been working with a couple of colleagues on documenting how to migrate transactional processes from WebLogic Integration (WLI) to BPEL and thought I would blog about transaction support in BPEL.  First off there are a couple of contradictory statements that need to be made.

• BPEL does not have transaction delineation
• Oracle BPEL Process Manager supports transactions

The first statement relates to the BPEL language which has no way of marking the start or end of XA style transactions, instead it provides semi-automated recovery through the concept of compensation, but more on this in another entry.

The second statement relates to the Oracle implementation of BPEL.  BPEL is inherently a stateful language and the BPEL Process Manager uses XA transactions to manage that state.  It also allows XA transactions to extend beyond the boundaries of the active BPEL process.  In this post I will try to explain a little about how this works.

Transactions Everywhere

Every BPEL process executes in the context of one or more transactions.  When a process starts to execute the process manager does one of two things with transactions;

1. It starts a new transaction for the process.
2. It enlists the process in an already open transaction.

BPEL uses this transaction to update process state in the database and also log audit events in the database.  Even if all this is turned off the process manager will still have processes execute in a transaction context.  When invoked through a web service interface BPEL will use the first option as there will not be a transaction context available.  When invoked through the Java API in a transactional environment then the process manager may use either option depending on the parameters used in the call.

Lazy Boy

BPEL will try to squeeze as many activities into a single transaction as it possibly can.  Only a few activities actually cause the process manager to update the process state in the database and commit the transaction.  A given activity does not always cause a transaction to be committed, sometimes the process may keep the same transaction context because of the properties associated with the activity.

Enlisting Support with a Little Participation

When BPEL calls an adapter, the ESB or another process in the same BPEL domain then it has the option to either include the target in the same transaction, or to start a separate transaction.  This behavior is controlled by the “transaction” property in the partner link.  If the “transaction” property value is set to “participate” then the callee is enlisted into the current transaction, if it is unset or set to any other value then the callee executes in a seperate transaction.  This is great for ensuring that several database updates are all included as part of the same transaction, but it does have a downside.  If any of the callees rolls back the exception then the the current BPEL process state is also rolled back to the last commit.  If there has been no commit and this process was invoked through a synchronous interaction then the process will seem to disappear from the face of the earth.  Something to be aware of.  To minimise the impact of this the database adapter will throw an error rather than rollback the transaction if by so doing it would not affect transactional integrity (basically you can do one DB update without rolling back but if you do two then an error will cause a rollback).

Parallel Execution

You might think that a BPEL flow activity would cause a new transaction to be started as this allows several activities to be executed in parallel. However this is not the case as the BPEL Process Manager will try to execute the flows sequentially for efficiency purposes.  Only if some action in the flow would cause a new transaction to be started will the activities execute in parallel.  A common use case for this is when a flow is used to make several synchronous calls.  Normally they would all execute in the same transaction context, however this means that the services are called sequentially, which can add latency to the application.  To avoid this there is another partner link property, “nonBlockingInvoke”, which when set to “true” causes a new thread to be spawned to execute the invoke.  The upshot of this is that a new transaction is started and this allows for true parallelism in the flow.

Lets Talk About Commitment

The general rules about what causes a commit are pretty straightforward, the BPEL process will commit the transaction and start a new one after the following activites

• receive – unless it is the first receive in the process and the process was called with a “transaction=participate” property (possibly by marking it as such in the partner link) in which case the process participates in the already existing transaction.
• onMessage
• wait – for very short waits (up to a couple of seconds) the process manager will not commit the transaction.
• onAlarm
• invoke – only if the partner link is non-idempotent (can be marked with the “idempotent” partner link property).
• End of process flow – only if the process is not participating in the callers transaction (the corollary of the receive).

The table below summarises the transaction status when performing different activities.

Activity	Transaction Status in BPEL Process	Transaction Status in Target Process or Adapter
Receive	New Transaction	N/A
Receive with Property “transaction=participate”	Use Existing Transaction from Caller	N/A
Invoke Synchronous Process	Use Existing Transaction	New Transaction
Invoke Synchronous Process with Partner Link Property “transaction=participate”	Use Existing Transaction	Use Existing BPEL Transaction
Invoke Synchronous Process with Partner Link Property “idempotent=false”	New Transaction	New Transaction
Invoke Synchronous Process with Partner Link Property “nonBlockingInvoke=true”	New Transaction	New Transaction
Invoke Asynchronous Process	Use Existing Transaction	New Transaction
Invoke Asynchronous Process with Partner Link Property “transaction=participate”	Use Existing Transaction	New Transaction
Invoke Synchronous Process with Partner Link Property “idempotent=false”	New Transaction	New Transaction
Wait < a couple of seconds	Use Existing Transaction	N/A
Wait > a couple of seconds	New Transaction	N/A
Flow	Use Existing Transaction

Flow

You may be surprised to see flow on the list, and then even more surprised to see that it uses the existing BPEL transaction context.  The reason for this is that the BPEL process manager will try to execute all the flow legs sequentially and so the behavior of the flow will be determined by the activities in the flow rather than the actual flow itself.

References

You can read more about transactional semantics (complete with pretty pictures that I think do a good job of explaining what is happening) in the Oracle SOA Suite Best Practices Guide chapter 1.

Clemens also blogged about this a while back here and here.

Calling BPEL from the Oracle Service Bus

The latest SOA Suite from Oracle now includes the former AquaLogic Service Bus (ALSB), now known as the Oracle Service Bus (OSB).  The old Oracle Enterprise Service Bus (ESB) is still available but the preferred Service Bus in Oracle is now the OSB.  Currently OSB is only available on WebLogic platform whilst ESB is supported on all SOA Suite platforms.  This entry details the steps that I had go through to call BPEL (version 10.1.3.4) from ALSB (version 3.0 which can be downloaded from here).  When the Oracle badged version is released some of the problems I had should go away.  So here is what I did to get OSB to call BPEL.

Creating the Project and Directories

I began by creating a project in the service bus, I tend to separate components out into sub-folders such as WSDL, Schemas, Proxy and Business Services.  This makes it easier for me to find things.

Loading the WSDL

In the BPEL console I navigated to the WSDL for the process I wanted to call and saved it locally.  In my OSB projects WSDL folder I then uploaded the BPEL WSDL.  After doing this it complained that:

The WSDL "AddBPELProcess.wsdl" was successfully created with validation errors. View the WSDL/Conflicts to see detailed diagnostic messages.

Drilling down into the loaded WSDL I got a more specific error message explaining what the conflict was:

One of the WSDL dependencies is invalid.

By default JDeveloper creates a WSDL that imports its type definitions from a separate XML schema.  This XML schema file by default has the same name as the WDSL file but with a .xsd extension.  The service bus was complaining because it couldn’t find that schema.  I would obviously have to fix this.

Loading the Schema and Resolving the Reference

BPEL stores the schema at the same location as the WSDL file.  So if the URL of my WSDL file was “http://w2k3/orabpel/default/AddBPELProcess/1.0/AddBPELProcess?wsdl” then the URL of my XSD would be “http://w2k3/orabpel/default/AddBPELProcess/1.0/AddBPELProcess.xsd”.  I needed to first of all download from BPEL the schema file and then I uploaded it to the Schema folder I had created in my service bus project.  I then went back to my WSDL folder, drilled down into my BPEL process WSDL and went into the the “Edit References” screen.  Here I was able to browse for my uploaded schema and match it to the schema expected by the WSDL.  Saving these changes gave me a WSDL that the service bus was happy with.

Creating the Business Service

My BPEL process was to be a back end service or business service in the service bus.  having loaded the WSDL for the service I was now ready to create a business service.  I went to the BusinessServices folder and added a business service using the WSDL I had just uploaded.  I chose to use the port rather than the binding in the WSDL as this would provide me with the physical endpoint of the service from the WSDL, avoiding me having to specify it.  After creating the business service I then activated my changes so I could test out the business service from the service bus test screen.

Testing the Business Service

After activating my changes I clicked on the little bug icon next to my business service expecting to see a nice test screen, but all I got was an ugly error message.

Error Accessing Test Configuration

Underneath this message was some more detail.

An unexpected error occured accessing information about the WSDL of the service:
com.bea.wli.config.component.NotFoundException: Can not compute effective WSDL for : BusinessService Chapter19/BusinessServices/AddIntegers

Still not very helpful.  A quick search of the internet turned up this useful blog entry by Berthold Maier (thank goodness for my schoolboy German).

Fixing the WSDL

It turns out that the service bus didn’t like the partner link elements at the end of the WSDL.  Personally I think it should have ignored them if it didn’t understand them, but c’est la vie!  I edited the WSDL in the service bus and commented out the partner link information.

<!--
  <plnk:partnerLinkType name="AddBPELProcess">
    <plnk:role name="AddBPELProcessProvider">
      <plnk:portType name="tns:AddBPELProcess"/>
    </plnk:role>
  </plnk:partnerLinkType>
-->

I was then able to commit the changes and go back to the business service test screen which now appeared correctly and allowed me to invoke the BPEL process.

More Thoughts on WSDL

The above steps let me call my process, but I did have to manually edit the WSDL.  I could have avoided this by uploading to the service bus the WSDL from within my JDeveloper project which is missing the binding information.  But this raises another set of issues and I prefer to work with what is deployed, particularly as I cannot guarantee access to the JDeveloper version of the WSDL and even if I had access I have no guarantee that the deployed WSDL and the WSDL in JDeveloper are the same.

Summary

When calling BPEL from the service bus remember two key things

1. Upload any referenced schema or wsdl files into the service bus.
2. Remove the partner link information from the wsdl obtained from JDeveloper.

Apart from those two issues everything else seemed to work fine.

I have to say that I find the newly acquired Oracle Service Bus very capable and has lots of nice features such as support for throttling, configuration history and dashboards.  And best of all the next release will have tighter integration with BPEL, what's not to like?

Getting Latest Version of SOA Suite

I keep a VM image with snapshots of various clean installs of SOA Suite.  When a new patch set or point release comes out I then apply it to the original vanilla install.  This keeps me with nice clean SOA Suite installs.  I thought people might be interested in what is the current latest software release level and how to get it / install it.

Current SOA Suite release level is SOA Suite 10.1.3.4 and there is a patchset available known as MLR#1.  Don’t confuse MLR#1 with earlier patchsets for 10.1.3.3 which got up into the teens.

SOA Suite 10.1.3.4 is installed as a patch to 10.1.3.1 except in the case of BAM where it is a patch to 10.1.3.3.  Note that the patch can also be applied to pretty much any patched versions of SOA Suite such as 10.1.3.3 MLR #10

Installing SOA Suite 10.1.3.4 MLR#1

To get a vanilla install of 10.1.3.4 MLR#1 we do the following:

1. Download SOA Suite 10.1.3.1 from OTN.  It is CD1 for your platform (Windows and Linux links).
2. Unzip/extract SOA Suite and install by running installer.  Note that if you are either not on Windows or wish to use an existing database then you need to have an existing database installed and ready to go.
3. Verify install worked by bringing up BPEL console, ESB console, App Server Console, etc.
4. Download SOA Suite 10.1.3.4 patch from OTN.  It is labeled 10.1.3.4 patch for your platform (Windows and Linux links).
5. Unzip and install by running installer.  Run database upgrade scripts as instructed.
6. Verify update worked by bringing up BPEL console, ESB console, App Server Console, etc.
7. Download MLR#1 patch from MetaLink.  This can only be done if you have a MetaLink account for SOA Suite.  If you are evaluating the software then you should be fine without MLR#1.  MLR#1 is patch number 7375086 and is a generic patch but is tagged as a Linux patch because MetaLink doesn’t have a generic platform.  The patch itself is valid on all 10.1.3.4 platforms, not just Linux.  Check the readme for install instructions.
8. Unzip and apply the patch.
9. Verify update worked by bringing up BPEL console, ESB console, App Server Console, etc.

Congratulations, you are now the proud owner of a SOA Suite 10.1.3.4 MLR#1 installation.

Installing BAM 10.1.3.4

To get a vanilla install of 10.1.3.4 BAM we do the following:

1. Download BAM 10.1.3.3 from OTN.  It is CD1 and is only available for Windows.
2. Unzip/extract BAM and install by running installer.  Note that if you will need an existing database.
3. Verify install worked by bringing up BAM console.
4. Download BAM 10.1.3.4 patch from OTN.  It is labeled 10.1.3.4 patch for Windows and is the same as the 10.1.3.4 SOA Suite patch so you can use the one yopu downloaded in step 4 in the previous section.
5. Unzip and install by running installer.  Run database upgrade scripts as instructed.
6. Verify update worked by bringing up BAM console.

There is currently no MLR patch for BAM 10.1.3.4.

Checking Versions

You can verify that you have the latest version of SOA Suite by going to the home page and you should see the version number as “Welcome to Oracle SOA Suite (10.1.3.4.0)” – note that patch levels are not shown here and must be interrogated by using “opatch lsInventory”.

 

To check what version of BAM you are running log onto any of the BAM tools, Active Viewer for example, and select the About box.  This will show the release as 10.1.3.3.0 but will have 10.1.3.4.0 mentioned in the Updates section.

BPEL Deployment Framework

One of the coolest features in BPEL 10.1.3.4 is the deployment framework which makes the job of moving processes between dev, test and production environments much easier.  Modifying the bpel.xml file or altering descriptors through the console provide a degree of customisation for different environments, but it is all done on a single property at a time basis and requires a lot of work for each environment, especially when it is considered that individual WSDL files may also need to be updated. There is a better way in SOA Suite 10.1.3.4; the deployment framework.

The deployment framework combines the BPEL suitcase with a deployment plan that updates multiple files in the BPEL suitcase with the correct values for the deployment environment. Different deployment plans can be created and maintained for each deployment plan.

It is possible to generate a template BPEL deployment plan from a BPEL suitcase which can then be customised and used with the base BPEL suitcase at deployment time to update the various URLs and properties.

The steps to customise the BPEL Suitcase for each environment are as follows.

• Create a deployment template from the BPEL project or suitcase which will be used as the basis for the deployment plans.
• Create a deployment plan based on the template for each target environment.
• Attach the appropriate deployment plan to the BPEL suitcase or project prior to deploying in the target environment.
• Deploy the BPEL process into the target environment.

Creating a Deployment Plan Template

To create a deployment template we need to add the following commands to the build.xml ant file that is built by JDeveloper and found in the Resources folder of our BPEL project.

<target name="generate_plan_from_project">
  <generateplan planfile="${process.dir}/planfile.xml"
                verbose="true"
                overwrite="true"
                descfile="${process.dir}/bpel/bpel.xml"/>
</target>
<target name="generate_plan_from_suitcase">
  <generateplan planfile="${process.dir}/planfile.xml"
                verbose="true"
                overwrite="true"
                suitecase="${process.dir}/output/bpel_${BPELSuitcase.BPELProcess(id)}_${rev}.jar"/>
</target>

This creates two new ant targets, generate_plan_from_project and generate_plan_from_suitcase. These both create a template deployment plan, either directly from the project files, or from a generated suitcase. A BPEL suitcase is only generated when a BPEL project is deployed or when the project is “made”, so if using the option to generate from a suitcase it is necessary to ensure that the suitcase has previously been created. If the suitcase generated is a revision other than 1.0 then it is necessary to set the revision property in the build.properties file that is found in the Resources folder of the BPEL project in JDeveloper.

# Change below if deploying with process revision other than 1.0
rev = 1.1

The generateplan command in ant uses four attributes

• suitecase or descfile is the source information for the deployment plan.
• planfile is the location of the planfile template to be generated.
• overwrite will replace any existing planfile of the same name.
• verbose turns up the level of reporting.

A sample deployment plan template is shown below. Note the use of two elements:

• <replace> is used to replace the value of a property within a specific part of the bpel.xml
• <searchReplace> is used to <search> for a string in WSDL and XSD files and <replace> it with another string.

<?xml version="1.0" encoding="UTF-8"?>
<BPELDeploymentPlan xmlns="http://schemas.oracle.com/bpel/deployplan">
  <BPELProcess id="SimpleFileProcess">
    <configurations/>
    <partnerLinkBindings>
      <partnerLinkBinding name="ReadNewCustomerFileService">
        <property name="wsdlLocation">
          <replace>ReadNewCustomerFileService.wsdl</replace>
        </property>
      </partnerLinkBinding>
      <partnerLinkBinding name="WriteNewCustomerDBService">
        <property name="retryInterval">
          <replace>60</replace>
        </property>
        <property name="wsdlLocation">
          <replace>WriteNewCustomerDBService.wsdl</replace>
        </property>
      </partnerLinkBinding>
      <partnerLinkBinding name="CardValidatorService">
        <property name="wsdlLocation">
          <replace>CardValidatorService.wsdl</replace>
        </property>
        <property name="location">
          <replace>http://w2k3/chapter18/CardValidatorServiceSoapHttpPort</replace>
        </property>
      </partnerLinkBinding>
    </partnerLinkBindings>
  </BPELProcess>
  <wsdlAndSchema name="CardValidatorService.wsdl|DBAdapterOutboundHeader.wsdl|fileAdapterInboundHeader.wsdl|NewCustomerFile.xsd|ReadNewCustomerFileService.wsdl|WriteNewCustomerDBService.wsdl|WriteNewCustomerDBService_table.xsd">
    <searchReplace>
      <search/>
      <replace/>
    </searchReplace>
  </wsdlAndSchema>
</BPELDeploymentPlan>

Creating a Deployment Plan

Having created a template we can use this to create deployment plans for each specific environment. We do this by creating a copy of the deployment plan by selecting Save As from the file menu in JDeveloper and then editing the <search> and <searchReplace> tags to match our target environment.

We will search and replace all instances of our local development machine hostname, w2k3, with the name of our test server, testserver, across wsdl and xsd files. To do this we modify the search and replace elements as shown below.

<wsdlAndSchema name="*">
  <searchReplace>
    <search>w2k3</search>
    <replace>testserver</replace>
  </searchReplace>
</wsdlAndSchema>

This will cause the BPEL process manager to search all WSDL and schema files “*” in the suitcase at deployment time and replace the string “w2k3” with the string “testserver”. Note that it is possible to have multiple <searchReplace> elements.

Attaching a Deployment Plan to a BPEL Suitcase

Having created and saved a deployment plan specific for one or more specific environments we will want to deploy our process into an environment. Before doing this we must first attach the specific deployment plan to the BPEL suitcase. We do this using the following ant command.

<target name="attach_plan">
  <attachplan planfile="${planfile}" verbose="true"
               overwrite="true"
               suitecase="${process.dir}/output/bpel_${BPELSuitcase.BPELProcess(id)}_${rev}.jar"/>
</target>

This will create a file bpeldeployplan.xml in the BPEL suitcase. This is the deployment plan that will be used at deployment time by the BPEL process manager. Note that the name of the deployment plan to use is encoded as an Ant property planfile that must be set in the build.xml. Once attached the deployment plan will be executed when the BPEL suitcase is deployed.

Modifying Ant to Use Deployment Plan

In addition to adding the two tasks above to the build.xml file, it is possible to add the attachment of the plan file as part of the regular deploy process by modifying the dependencies of the process-deploy task by adding the attach_plan dependency after the compile dependency.

<target name="process-deploy"
          depends="validateTask, compile, attach_plan, deployProcess,
deployTaskForm, deployDecisionServices" />

Now when building and deploying with ant the deployment plan will be attached to the suitcase before the suitcase is deployed to the target environment. This allows us to provide a different deployment plan for each environment, which can then be deployed from the command line in a reproducible fashion.

Summary

The deployment plans allow a lot of automation in the move from development to production, and because they can be used with editable files and command lines they are easy to include in the version control systems and automated build environments.  Congratulations to the BPEL developers on a useful piece of functionality that appeals to those responsible for ensuring consistency of environments rather than developers.

Time Travel for Oracle Web Services Manager Administrators

I watched the Sarah Jane Adventures, a Doctor Who spinoff, with my family this week in which Sarah Jane has the opportunity to travel back in time to see her parents before they died.  Of course it all goes horribly wrong as usual and Sarah-Jane and her young assistants will have to save the world, again!  Reminds me of IT, you do something which seems like a good idea only to discover that it is a disaster.  Fortunately many of the tools we use have some sort of versioning that allows us to rollback the changes and restore the world to the way we want it.

Oracle Web Services Manager has just such a facility.  Every time a change is made to policies applied to a component in OWSM the previous set of policies is saved as a version, stored, ready and waiting to be restored if the change causes the fabric of space time to unravel.  When reviewing the policies applied to a component there is a View Versions label next to a link labelled Version.

  Clicking the Version link brings up a list of all the previous policy sets that were applied.  Not only does it list all the previous policy sets, it also tells us when they were applied.

Note that we have the option to view the details of a previous policy set by clicking on the view icon .

This shows us the policies that were applied to the component at a particular point in time.  By clicking on the View Details icon we can see the actual policy steps.

If after reviewing a previous policy set and the steps associated with the policies we realise that we need to restore that policy we can do so by selecting the restore link from the list of previous policy sets.

OWSM will ask for confirmation before restoring the previous policy set as the current working set.

Note that the current policy set is not obvious from the Versions page.  To check exactly what is being applied you need to open up the current policy and verify it is the version that you want before clicking OK.

You can drill down into the View Details to make absolutely sure that it is doing what you think before you commit the changes and so publish the new policies to the policy enforcement points.

So now you can travel back in time to correct errors in your OWSM policies.  But wait, there is more. When Sarah Jane went back in time she saved her parents and as a result destroyed her own future.  Similarly when we rolled back our policy to a previous time, we may also have lost the service mappings that relate policies to service endpoints.  The policy versions are actually policy set versions.  A policy set is the collection of policies available to a component.  If you fall back to a policy set that does not include a policy that you have applied to an endpoint, then that policy to endpoint mapping is also lost.  In the TV show Sarah Jane went back in time to undo her change.  Similarly you can also reverse your meddling in time by restoring a later policy and any lost endpoint mappings will also reappear.

Happy time travelling.

A Long Book Story

For the last 18 months Matt Wright and myself have been working on a SOA Suite book.  We started off targeting the 11g release, but delays and then the BEA acquisition led us to focus on the current 10g production release.  This week I sent off the last rewrites to the publisher so all the chapters are now complete.  Matt and I wanted to focus on how to use the SOA Suite to build a service oriented architecture, when it is released you can let us know how well we achieved that aim.

We are now packaging up the code samples and polishing up the sample application ready to be available with the book.

Downloading SOA Suite

As part of the book Matt & I have been writing I had to provide instructions on how to install SOA Suite.  It struck me that the list and download location of components that make up the SOA Suite would be useful to have so I am posting them here.

Acquiring the Software

The list of required files to install components of Oracle SOA Suite 10.1.3.4 on Windows is shown in the table below. The location column indicates the URL where the software can be located for the given component. The filename is the name of the file to download. The notes column gives additional information about what to download. Oracle database XE is only required if you do not already have an 11g or 10.1.2 Oracle database available.  Note that this list is basically the core SOA Suite on OC4J, BAM for Windows, Oracle Service Bus on WebLogic, Oracle JDeveloper and a patch set to upgrade teh core SOA Suite and BAM to 10.1.3.4 revision levels.

Component	Location	Filename	Notes
Oracle Database 10g Express Edition	http://www.oracle.com/technology/software/products/database/xe/htdocs/102xewinsoft.html	OracleXE.exe	Only required if you don’t already have a suitable database.
Oracle SOA Suite 10g (10.1.3.1)	http://www.oracle.com/technology/software/products/ias/htdocs/101310.html	soa_windows_x86_ 101310_disk1.zip	Download CD1.
Oracle SOA Suite 10.1.3.4 Patch	http://www.oracle.com/technology/software/products/ias/htdocs/101310.html	ias_windows_x86_ 101340.zip	Download 10.1.3.4 patch. This patch covers both SOA Suite 10.1.3.1 and BAM 10.1.3.3
Oracle Business Activity Monitoring 10g (10.1.3.3)	http://www.oracle.com/technology/software/products/bam/index.html	bam_windows_x86_ 101330.zip	Download Oracle Business Activity Monitoring 10g For Microsoft Windows (x86), not the download with Enterprise Link.
Oracle Service Bus 10gR3	http://www.oracle.com/technology/software/products/osb/index.html	osb103_wls103_ win32.exe	Download Oracle Service Bus 10gR3
Oracle JDeveloper 10.1.3.4	http://www.oracle.com/technology/software/products/jdev/htdocs/soft10134.html	jdevstudio10134. zip	Download the Windows Install, Studio Edition.

Hope that you find the above useful.

Oracle HTTP Adapter

The Secret BPEL Adapter

I had a phone call a while back from a partner who wanted to call a service that took as input an XML document and returned an XML document.  The normal way to do this of course is using SOAP to manage the interface definition for the input and output parameters.  What was needed in this case was an HTTP adapter that could call the service appropriately.  The problem is that when you look in JDeveloper or the user guide there is no mention of an HTTP adapter.  My friend at the partner wanted to know if there was an HTTP adapter to which I replied “No and Yes”!  This left him a little confused to say the least so let me share with you the explanation I gave him.  Along the way we will look at how to use the HTTP Adapter functionality from BPEL.

WSIF Bindings

The key to using HTTP POST and GET requests is in the WSIF bindings.  Web Service Invocation Framework (WSIF) provides a way for non-SOAP services to be described with a WSDL wrapper.  For example the database adapter uses WSIF to provide a WSDL wrapper around a TopLink JCA adapter.  The adapters listed in the developers guide and supported directly in JDeveloper have re-entrant wizard driven interfaces that makes them easy to use.  There are other bindings supported by WSIF that do not have nice wizards to help you build them, however with a bit of knowledge (always a dangerous thing) they can provide useful functionality.

Where to Find Samples

The Oracle BPEL Process Manager ships with 3 tutorials for non-wizard driven bindings.  They can be found in the $ORACLE_HOME/bpel/samples/tutorials/702.Bindings directory.  The three samples are provided in three sub-directories:

• EJBBinding – Calling an EJB session bean.
• HTTPBinding – Calling an HTTP based service.
• JavaBinding – Calling a plain old java object (POJO).

Lets look in more detail at how the HTTP binding works.

HTTP Refresher

HTTP has a number of verbs that describe the operation to be performed, these include GET, POST and HEAD amongst others.  Generally if we are retrieving data then we are interested in GET or POST.  When using GET we pass parameters as part of the target URL, for example http://localhost/EchoApp/echo?p1=1&p2=2.  This example passes two parameters to the service located at EchoApp/echo, p1 with a value of 1 and p2 with a value of 2.  When using POST then the parameters are passed as data in the body of the message rather than as part of the URL.  Originally GET was intended for non-mutative operations and POST for mutative operations, but POST is often used when there is a large amount of data to be sent even if it is not mutative, it is also used to improve security as the parameters are not visible on the URL.

Performing a GET

Lets look at how to execute a GET operation that passes some parameters and receives an XML document in response.

I have created a simple Echo application that returns as an XML document the parameters it received and the contents of the body of the HTTP request.  This can be downloaded here.  To test the echo application there is an EchoApp/index.html that provides a simple form to perform a POST operation.  Just typing EchoApp/echo into a browser bar will perform a GET operation.

A sample WSDL for a GET operation is available here.  Lets look at the key binding section.

Within the binding we specify the binding verb for the HTTP operation.  In this case we want to use a GET operation.

<binding name="HttpGetBinding" type="tns:HttpGetPortType">
  <http:binding verb="GET"/>

Having described the type of operation we must specify the path of the URL (/EchoApp/echo) that we wish to invoke.  The server and port number are specified later.  Note that multiple operations may be specified at different URL paths.

  <operation name="GetData">
    <http:operation location="/EchoApp/echo"/>

We must now describe how we want to process the parameters.  In this case we want all the parts to be encoded as parameters on the URL and so use http:urlEncoded.  As the return is an XML document we specify that the return type is of mime type XML by specifying mime:mimeXml and providing the name of the part in the response message.

    <input>
      <http:urlEncoded/>
    </input>
    <output>
      <mime:mimeXml part="Body"/>
    </output>
  </operation>
</binding>

If we have input and output message definitions as shown below then the request would be encoded as /EchoApp/echo?param1=VALUEP1&param2=VALUEP2.  Note that the input message should have multiple parts of simple types.  The response would be expected to be received as an XML document with <get:Request> as a root element.

<message name="HttpGetIn">
  <part name="param1" type="xsd:string"/>
  <part name="param2" type="xsd:string"/>
</message>
<message name="HttpGetOut">
  <part name="Body" element="get:Request"/>
</message>

The actual server location of the target is given in the service section of the WSDL as shown below.

<service name="GetService">
  <port name="HttpGetPort" binding="tns:HttpGetBinding">
    <http:address location="http://localhost:7001"/>
  </port>
</service>

The above definitions then let us call a service using the GET verb and receive an XML document as a response.  Using this WSDL we can call the HTTP GET based service as though it were any other WSDL described web service, BPEL does not care about the underlying transport bindings.

Performing a POST with Form Style input

Lets now look at how to execute a POST operation that passes some parameters as though submitted from an HTML form and receives an XML document in response.

A sample WSDL for a POST operation passing form style parameters is available here.  Lets look at the key binding section.  As before we specify the binding for the HTTP operation, but change the verb to be a POST operation.  The location is specified in the same way as for the GET operation.

<binding name="HttpPostParamBinding"
              type="tns:HttpPostParamPortType">
  <http:binding verb="POST"/>
  <operation name="PostData">
    <http:operation location="/EchoApp/echo"/>

We must now describe how we want to process the parameters.  In this case we want all the parts to be encoded as form style parameters in the body of the request and so use mime:content type=”application/x-www-form-urlencoded”.  As the return is an XML document we specify that the return type is of mime type XML by specifying mime:mimeXml and providing the name of the part in the response message.

  <input>
    <mime:content type="application/x-www-form-urlencoded"/>
  </input>
  <output>
    <mime:mimeXml part="Body"/>
  </output>
</binding>

If we have the same input and output message definitions as used in the GET example then the request URL would be encoded as /EchoApp/echo with the body containing the following “param1=VALUEP1&param2=VALUEP2”.  Note that the input message should have multiple parts of simple types.  The response is handled in the same way as with the GET.

The actual server location of the target is given in the service section of the WSDL in the same was as the GET request.

The above definitions then let us call a service using the POST verb and receive an XML document as a response.  Using this WSDL we can call the HTTP POST based service as though it were any other WSDL described web service.

Performing a POST with XML Document input

In the previous section we looked as using a POST to submit form style parameters.  The POST verb lets us put any content into the body of the request and so it is often useful to be able to send an XML document to the service.  Lets now look at how to execute a POST operation that passes an XML document and receives an XML document in response.

A sample WSDL for a POST operation using XML input is available here.  Lets look at the key binding section.  As before we specify the binding for the HTTP operation with a POST verb.  The location is specified in the same way as before.  This time though for the parameter handling we map the input to a part in the input message using mime:mimeXml to identify the input as an XML document.  The output is mapped as before

<binding name="HttpPostXMLBinding" type="tns:HttpPostXMLPortType">
  <http:binding verb="POST"/>
  <operation name="PostData">
    <http:operation location="/EchoApp/echo"/>
    <input>
      <mime:mimeXml part="param"/>
    </input>
    <output>
      <mime:mimeXml part="Body"/>
    </output>
  </operation>
</binding>

This binding will cause the body of the request to consist of the XML document specified in the given message part.  If we have the input message definition shown below then the request URL would be encoded as /EchoApp/echo with the body containing the an XML document with a root element of “<get:HttpClientBPELProcessProcessRequest>”.  Note that the input message should have a single part.  The response is handled in the same way as with the GET and the previous POST.

<message name="HttpPostXMLIn">
  <part name="param"
           element="get:HttpClientBPELProcessProcessRequest"/>
</message>
<message name="HttpPostXMLOut">
  <part name="Body" element="get:Request"/>
</message>

The actual server location of the target is given in the service section of the WSDL in the same way as the previous GET and POST requests.

The above definitions then let us call a service using the POST verb, passing an XML document as input and receiving an XML document as a response.  Using this WSDL we can call the HTTP POST based service as though it were any other WSDL described web service.

Sample Code

A complete example is available for download.  To use it first deploy the example echo WAR file and then deploy the BPEL process which will call the sample echo application three times, once via a GET, once via a POST with forms style parameters and once via a POST with XML input.  The three results are stored in a sequence and returned from the BPEL process.

Summary

In this entry I have spoken about using the HTTP WSIF binding to provide a an HTTP adapter in BPEL.  It should also work fine with ESB.  OSB currently has a different (and easier) way of calling HTTP based services.  When creating an HTTP WSIF process it is generally a good idea to start with a working one and modify it to reflect your data types and endpoints.

So in answer to the question that started all this; is there an HTTP Adapter?  No there isn’t a nice adapter wizard to use HTTP bindings, but yes there is support for HTTP bindings through WSIF.

What’s in the SOA Suite and What’s it for?

I had a really good question recently.  Someone was struggling with Oracle Service Bus having previously used BPEL.

I am still working with OSB, and I am struggling to get to grips with how it will be used in FUSION and beyond.....
…
I know there is no definitive answer, but I want to try and get an idea of how it is intended to use OSB and Orchestration..... (cos it aint easy!!!)

Lets look at the three components in the SOA Suite that have some overlap and that my questioner was referring to.

• Oracle Service Bus (OSB)
• Oracle Enterprise Service Bus (OESB)
• Oracle BPEL Process Manager (BPEL)

All three of these provide connectivity services, all of them support transformation.  OSB and BPEL provide orchestration capabilities.  So what my correspondent was really asking was

What should be done in which service bus and what should be done in BPEL.

It is worth looking at each component in turn and identifying its sweet spots as I see them.

Oracle Enterprise Service Bus

Prior to the BEA acquisition this was Oracles primary service bus.  Since the acquisition of BEA the role of the OESB is to provide mediation services (transformation & simple routing) between SOA Suite components, primarily between BPEL processes in the 10g release.  In the 11g release this component becomes known as the Mediator and acts as a component in an SCA assembly.  If running on a non-WebLogic server then OESB becomes the only ESB available in the SOA Suite and so then it becomes a straight split between OESB and BPEL, with OESB providing routing and transformation services.

Oracle Service Bus (former AquaLogic Service Bus)

This is Oracles primary service bus and is the preferred platform for service virtualization and interactions external to the SOA Suite.  Currently OSB is only available on WebLogic server but the intention is provide it on other platforms as well in the future.  OSB is the foundation of service bus functionality moving forward.  This means that we should be using OSB for the following:

• Endpoint routing (providing location transparency) so that we do not care about the physical location of the endpoint.
• Endpoint abstraction (interface transparency) so that we do not care about the exact data formats required by the endpoint because the OSB will take care of transformations.
• Load balancing so that we do not care about which of multiple service implementations will actually service a request.
• Throttling so that we do not care about how use of services is restricted.
• Enrichment so that we do not care about how additional data is provided to the request to match the expected request and response formats.
• Simple synchronous composition so that we do not care if our abstract service call is actually made up of two or more physical service calls.
• Protocol conversion so that we do not care what physical transports are being used.
• Sync/async abstraction so that we can treat services as fire and forget or query response according to the needs of the client.

Oracle BPEL Process Manager

BPEL Process Manager is the primary composition, orchestration and process engine in the SOA Suite and as such should be used for the following.

• Complex composition of synchronous flows that involve more than a couple of services.
• Long running compositions that may run for minutes, hours or days.
• Asynchronous compositions that require correlation of requests and responses.
• Process abstraction that enables us to track processes and their interactions with multiple services.
• Human workflow for when we need to interface to wetware.

Summary

Hopefully the above has given you some indication of what each component is best used for.  The short version is, for routing and transformation between SOA Suite components use either OESB or OSB.  For service virtualization use OSB and for process orchestration and long running interactions use BPEL.  See, not that hard was it?

Oracle SOA Suite Developer’s Guide

My friend Matt Wright just pointed out to me that I hadn’t mentioned that our book was now available.  Thanks to the guys at Packt Publishing for guiding us through the process and publishing the book.  Our focus when writing the book was to provide a practitioners guide to implementing SOA using the Oracle SOA Suite.  As such we have set each component of the SOA Suite within the context in which might be used.  This seems to be the area that a lot of companies starting down a SOA path struggle with.  They are unsure what bits of technology to use for what.

Dave Shaffer, Oracle VP for SOA, kindly wrote a foreword for us.

Writing the book has been a huge undertaking for both Matt and myself.  During the writing of the book the 11g release was seriously delayed so we switched versions from 11 to 10.1.3 (the current production release).  Matt decided that he was sick of 365 days of rain and moved his family from England to Australia.  Finally Oracle purchased BEA and caused a re-evaluation of the Service Bus technologies.

I am pleased with the way the book has turned out, and Matt and I learnt a huge amount as we prepared the book, both of us had to delve into area with we weren’t totally familiar.  Matt became a master of the Rules engine and I discovered the marvelous new deployment capabilities added to the 10.1.3.4 release.

We have made every effort to match the book to the current production 10.1.3.4 release so get yourself a copy and give us some feedback.

The Problem

The native format builder wizard in the file adapter is great at reading flat file structures but doesn’t support reading more structured file structures.  Sometimes we need to read more complex structures such as master-detail records.  Let’s look at how we can use the file adapter to read structured file formats.

For example imagine a laundry list file such as the one below:

P,101,James L,Shirt,2,Starch L,Socks,6,De-odorise L,Pants,2,Remove Stains

P,220,JoJo L,Sweatshirt,1,Handwash L,Shirt,3,No Iron L,Socks,2,Iron L,Pants,2,Steam Press L,Tie,1,Dry clean

P,305,Ruth L,Skirt,7,Iron Pleats L,Socks,8,Iron

To make it easier to see the record structure I have put boxes around the individual records.  This can be represented in XML as shown below

The native format builder wizard would either treat each line as either the same type of record (“Multiple records are of single type”) or as two different types of record (“Multiple records are of different types”).

In the first case we would have a single record type with four fields that fails to distinguish between people (lines pre-fixed with ‘P’) and laundry items (lines pre-fixed with ‘L’).  This does not reflect the fact that the two lines are of different types.  The second case, multiple records are of different types, is closer to what we want.  It creates two records types, the type being determined by the value of the first field.  The XML for this is shown below:

However this case creates a list of two record types with no recognition that one record is nested inside the other.

Comparing the two XML representations we can see what needs to be done; we need to add an Items element under Room and move the Item element to be under Items.  The question is how do we describe this using the native schema constructs supported by the file adapter.

Creating a Master-Detail Records Native Format Schema

The easiest way to deal with this is to use the native format builder wizard in the file adapter to create the basic outline of the records for us.  We choose “Multiple records are of different types” and provide appropriate names for the two records types and individual fields within the records.

Having created the basic native format schema we can now edit it to be exactly the way we want it for the laundry list.

We make the following changes in the generated native schema file:

• Replace the <choice> element by a <sequence> element.  In conjunction with other changes this will give us a list of elements of the same type, rather than of list of elements of two different types.

  • <xsd:choice minOccurs="1" maxOccurs="unbounded"     nxsd:choiceCondition="terminated"     nxsd:terminatedBy=",">     … </xsd:choice>

    <xsd:sequence>    … </xsd:sequence>

• Replace the conditionValue attributes with startsWith attributes and add a comma to the end of the attribute values.  This will allow the native schema processor to identify the start of master records and child records.  We also add a maxOccurs attribute to the elements modified so that they can have multiple instances in a sequence.

  • <xsd:element name="Item"     nxsd:conditionValue="L"> … </xsd:element>	<xsd:element name="Item"     nxsd:startsWith="L,"     maxOccurs="unbounded"> … </xsd:element>
    <xsd:element name="Room"     nxsd:conditionValue="P"> … </xsd:element>	<xsd:element name="Room"     nxsd:startsWith="P,"     maxOccurs="unbounded">     … </xsd:element>

• Add an <Items> element as a sequence to the <Room> element sequence and move the <Item> element to be inside the <Items> sequence.

  <xsd:element name="Room" nxsd:startsWith="P," maxOccurs="unbounded">
      <xsd:complexType>
          <xsd:sequence>
              …
              <xsd:element name="Items" maxOccurs="1">
                  <xsd:complexType>
                      <xsd:sequence>
                          <xsd:element name="Item" nxsd:startsWith="L," maxOccurs="unbounded">
                              …
                          </xsd:element>
                      </xsd:sequence>
                  </xsd:complexType>
              </xsd:element>
          </xsd:sequence>
      </xsd:complexType>
  </xsd:element>

This gives us a native schema format that looks like this:

Note that this is the structure that we were aiming for in the first place.

Sample Code

I have created a simple BPEL process that performs the following steps

• Read a laundry file using the wizard generated native schema format.  This creates an XML document with two distinct records types.
  • Note that in the process I do not delete the file so it is important to make sure that if you re-use the same file with the process you need to update its timestamp!
• Read the laundry file again using the modified native schema format.  This creates an XML document with master-detail style records which reflects the actual format of the file.
  • Note that I use a synch read with the filename provided by the inbound header from the previous file read.   I delete the input file after reading it.
• Write the laundry file using a pure XML schema to create an XML document file of the laundry list.

Sample code is zipped up as a JDeveloper project and can be downloaded here.  In addition to the normal BPEL artifacts within the project, I have included a sample laundry file (laundry.sample.txt) in the top level of the project directory.  After deploying to a BPEL server the process will look in C:\FileTransfer\InBound for the input file, after processing the input file it is written to C:\FileTransfer\Processed and the output file is generated in C:\FileTransfer\OutBound.  Either create appropriate directories or edit the project to use new directories.

Documentation

This entry has used the facilities of the native format schemas in SOA Suite.  These are documented in chapter 7 of the Oracle® Application Server Adapters for Files, FTP, Databases, and Enterprise Messaging User's Guide.

Good luck in using the adapters to process master-detail records!

I have just been with a customer who was using Oracle Enterprise Linux 5.  Now this shouldn’t be any different from other Linux installations except for one minor problem, the SOA Suite installer insists on checking that the Linux flavor is one explicitly supported by SOA Suite.  Well OEL wasn’t in the list when SOA Suite 10.1.3.1 came out and so the installer fails on the pre-requisite checks and won’t go any further, even if you applied all the required patches to the OS.  Fortunately there is a patch 6339508 which provides replacement pre-requisite tests.  To use it you need to add two parameters to the runInstaller command as shown below.

./runInstaller PREREQ_CONFIG_LOCATION=<PATCH_LOC>/prereq –paramFile <PATCH_LOC>/oraparam.ini

PATCH_LOC is the location where you unzipped the 6339508 patch.  With these new parameters the installer correctly recognizes Oracle Enterprise Linux as a supported platform.  Note that OEL needs some specific patches when used with SOA Suite and these need to be installed prior to running the installer.  Check the documentation for details.

I am seeing an increasing number of customers taking up OEL, I think because of the attraction of a single company supporting the whole technology infrastructure stack, OS, AS and DB.  I expect to see more of this still in the future.

Installing SOA Suite 10.1.3 with an 11g Database

Just at a customer who has an 11g RAC database that he wants to use for his SOA repository.  If you try and install SOA Suite into an 11g database, it tells you that the database is not supported and the irca configuration assistant fails to find a java library.  11g is a certified platform for SOA Suite 10.1.3.4 so here is how to get it installed.

IRCA

Before installing the SOA Suite executables you need to run the irca script to create the SOA Suite schemas in the 11g database.  The irca script needs to be able to find an ojdbc14.jar file.  This file is not shipped with the 11g database which provides libraries for Java 5 and Java 6 rather than the almost obsolete Java 1.4.  This leaves you with a couple of options;

• Use an Oracle 10g home as your Oracle home, if you have one on the machine, when running irca.
• Copy the JDBC libraries from an Oracle 10g home (<Oracle10g_Home>/jdbc/lib) to the Oracle 11g jdbc location (<Oracle11g_Home>/jdbc/lib).

Having done this then the irca script should run fine and create the ORABPEL, ORESB and ORAWSM schemas for you in an 11g database.  You are now ready to run the SOA Suite installer.

Installer

When you run the installer to create the SOA Suite instance it will fail when checking for the existence of the SOA Suite schemas unless you patch the installer files first.  The way to do this is to download patch 6265268 from MetaLink and follow the instructions which basically require you to modify the install media as follows:

• Replace DBConnectQueries.jar
  • Move
    • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar
  • to
    • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar.pre6265268
  • Copy
    • <PATCH>/DBConnectQueries.jar
  • from the patch to
  • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar
  • Note that the current patch documentation incorrectly refers to an 8.5 directory rather than the 8.4 that actually exists.
• Replace IP_DBQueries.jar
  • Move
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar
  • to
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar.pre6265268
  • Copy
    • <PATCH>/IP_DBQueries.jar
  • from the patch to
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar

You can now launch the installer and there will be no complaints about an 11g database.  Note that if you are installing a SOA Suite cluster then this will need to be done for each SOA Suite instance being installed.

What’s the Point?

As you are making these changes your mind keeps asking why you are doing this.  Apart from being told to do so by your DBAs there are some good reasons for using 11g.  The 11g database is the most manageable Oracle database ever, and several options only work on the 11g database.  11gR1 has been out for a long time now and so from a longevity perspective its best to put new deployments on an 11gR1 platform even if you have a release-1 strategy to avoid “bleeding edge” deployments.

Finally as you can see, it is not that hard to use an 11g database with SOA Suite.

OTN have just posted a podcast “Oracle SOA Suite Developer's Guide: An Architect's Perspective” featuring an interview by Bob Rhubart with myself and Matt Wright about our book, the Oracle SOA Suite Developers Guide.  Matt comes across sounding calm and assured and I come across as …, well I’ll let you decide for yourself.  In my defence the interview was about 11pm my time and 8am Matts time, with Bob splitting the difference at about 3pm.  Listen and Enjoy.

Building a SOA Suite Cluster

Having spent a couple of weeks working on a SOA Suite cluster thought I would share some thoughts around clustering and SOA Suite.  Clustering of both BPEL Process Manager and Oracle Service Bus is relatively straightforward but there are a few gotchas.  Both BPEL and SOA Suite are stateless in the way they implement clustering, however BPEL does of course persist state to a database.

SOA Suite Clusters

Both BPEL and OSB clusters expect to be fronted by a load balancer.  Both can provide load balancing through a front end web server but a hardware load balancer is the best approach as shown in the diagram.

In this example we have an Oracle Real Application Cluster database running on 3-nodes to provide a high availability database environment.  We then have two clusters.  A cluster of 5 BPEL Process Manager instances, all pointing to the same RAC database, and a cluster of 5 OSB instances.  The BPEL cluster and the OSB cluster are both fronted by two hardware load balancers.

The BPEL instances and the OSB instances are in an active-active mode, meaning that all nodes are processing requests at the same time.  The load balancers are in active-passive mode, meaning that one load balancer processes all the traffic with the other load balancer acting as a hot standby in case of failure of the first load balancer.

This configuration avoids a single point of failure as every component is duplicated.  The system has been sized to be able to sustain the expected load even in the event of losing a machine.  This is known as "n+1" architecture, meaning that we need "n" machines to meet the requirements and so we provide "n+1" machines to allow for machine failure.  In the example shown we actually have two machines more than we need in normal operation because the the RAC database is running an "n+1" configuration as well as the SOA Suite.  Note that by running OSB on the same machines as BPEL we reduce the amount of extra hardware needed for failover and also reduce the latency of OSB-BPEL communication.

In the event of a failure then only in-process requests would be impacted.  Any requests that are "idempotent" (meaning they can be resubmitted with no ill effects) can be set up to automatically retry, further reducing the impact of a software or hardware failure.  Both BPEL and OSB can be set to automatically retry requests in event of failure, making the failover transparent.

Note that clustering does not help if we have a site failure due to fire, flooding, power failure or air conditioning failure for example.  In those cases we would need to have some sort of disaster recovery site, perhaps using Oracle Data Guard to keep the sites in synch at the database level.

BPEL Clusters

A BPEL cluster is effectively defined by a shared dehydration store.  Synchronous interactions must be processed within a single BPEL server instance, as the client has connected to a socket and expects a response on that same socket.  Asynchronous interactions are like any other long running BPEL process and may start processing on one node and then have processing resumed on another node, either due to failure or some other event.  The dehydration store (an oracle RAC database in the example above) provides a common location for process state that allows any BPEL instance to resume execution of a process instance.

When installing a BPEL cluster the best place to start is the High Availability Guide.  This outlines that you create a BPEL cluster by doing the following:

• Get the address of the load balancer.
• Run the repository creation assistant to create the BPEL meta-data in the database.
• Install OHS and OC4J components on the machines.
• Install BPEL Process Manager into the app server instances installed previously.
• If using a RAC database make sure that the JNDI data sources are using all RAC nodes.  See the Enterprise Deployment Guide for instructions on using Fast Connection Failover.
• Configure BPEL in a cluster as outlined in the Enterprise Deployment Guide.
• Set up JGroups to make all nodes aware of each other in the cluster.  Note that in 11g Coherence will be used for this which will simplify configuration.
• Set enableCluster and ClusterName in the collaxa-config.xml file.
• Make sure the BPEL PM instances all use the load balancer address for server URL and callback URLs.  This ensures that in event of node failure requests and responses are rerouted to remaining instances.
• Deploy processes on each node to make sure that all components are available on all nodes.  If you don't do this some processes will work because they don't have dependencies on non-BPEL components, but others will not.

Once set up use the BPEL fault handling framework to make sure that any calls to OSB are automatically retried in event of failure.

OSB Clusters

An OSB domain may have a single OSB cluster.  This cluster is installed like any other cluster in WebLogic.  Details on configuring the cluster can be found in the Creating WebLogic Domains Using the Configuration Wizard documentation.  For normal operation the OSB cluster is completely stateless, however the metrics gathering and aggregation takes place in a singleton service that by default is assigned to the first machine created in the cluster.  If this server fails then metrics will stay in the message queues to which they are delivered until a new instance starts.

• Get address of the load balancer.
• Install OSB software onto each machine or into a single shared location.  If a shared location make sure that the reference to it is the same on each machine.
• Create an OSB domain
• Creating a cluster and machines and assign servers to machines as explained in the Creating WebLogic Domains Using the Configuration Wizard documentation.
• Use load balancer address for cluster address.
• If you are not using a shared file location for your OSB install then you need to copy the contents of the osb domain directory to all nodes.  This ensure that the correct scripts are available for the node manager to launch managed servers.
• Run node manager on each machine.
• You can now launch your admin server and start the managed servers one at a time.  It is recommended that you start the OSB server running the data collectors first.  This will avoid timeouts on the other machines in the cluster and ensure that metrics are available.
• If message queues are required to be highly available then they should use persistent storage either on a shared highly available disk (a SAN for example) or they should use database persistence.
• If you are using an Oracle database then you should configure your OSB domain to store metrics in the Oracle database rather than in pointbase as explained in Creating WebLogic Domains Using the Configuration Wizard.

When calling BPEL or external web services from OSB make sure that you specify retries to allow for node failure.  Intra-OSB calls should be done using "local" transport for efficiency.

Summary

Configuring a cluster is a bit more involved than configuring a single instance, but it is not massively more complicated and it does provide both scalability and high availability.  Both BPEL and OSB scale linearly with increased nodes.  The only limitation on BPEL is the load on the backend dehydration store.  So go ahead, enjoy a cluster!

Today sees not only the official launch of SOA Suite 11g but also the availability of the software.  You can download it all now from OTN.  Downloads are available for Windows, Linux, Solaris and HP-UX.  You will need the following to develop and run 11g SOA Suite applications.

• Oracle WebLogic Server 11g R1 (with or without Oracle Enterprise Pack for Eclipse - OEPE).
• Oracle SOA Suite 11g R1 (includes BPEL, Human Workflow, Business Rules, BAM, Mediator, B2B, OWSM, User Messaging Service and Enterprise Manager Fusion Middleware Control)
• Oracle Repository Creation Utility 11g R1 (creates the database schemas needed by SOA Suite and other components)
• Oracle JDeveloper 11g R1 this also needs the SOA plug in which should be available through the JDev extensions mechanism.
• Documentation is also available online or may be downloaded.

Watch the launch webcast and enjoy the latest release of the software, it a big step forward for SOA Suite.

Since moving to support I have learned a lot about clustering.  Some of the things I have learnt are;

• Lots of customers are running SOA Suite clusters
• Lots of them haven't read the High Availability Guide (10g or 11g)
• Lots of them haven't read the Enterprise Deployment Guide or EDG (10g or 11g)
• Many of them have problems because of the points above

Part of the problem for many customers is that setting up a cluster has a lot of steps and a few gotchas that can come back to bite you.  Just got off the phone with a customer who was having problems with a cluster install, nothing too serious but irritating and slowing him down.  Unless the HA & EDG are followed very carefully it is easy to make mistakes.  A few common problem areas I have seen are

• Failing to separate the design and run time of the ESB
  The ESB design time is a singleton and there must never be more than one instance of the ESB design time active against the same repository at the same time.
• Poor configuration of JGroups or Coherence
  In 10g JGroups is used to identify cluster membership, in 11g Coherence plays the same role.  If these are not configured consistently across the cluster then one part of the cluster may be unaware of the existence of other parts of the cluster with dire circumstances for some shared resources.
• Failure to set up virtual addresses correctly
  The cluster should have a single virtual address. This virtual address needs to be configured in the HTTP listener, into the OC4J servlet engine and into the BPEL URL settings, amongst other places.  Failure to do this can lead to odd behavior.
• Failure to test on a cluster
  Seems many companies have clusters in production but not in test and dev.  Surely no-one is that stupid you say.  Well they may configure test instances with a cluster but for resource reasons run only one node of the cluster, after all a one node cluster is still a cluster right...  They then wonder why they only get certain problems in production and can't reproduce them in test.
• Many customers fail to have suitable test load balancers
  Often customer will not have a hardware load balancer for use in their test or dev environments and will rely on a software load balancer.  Some of these software load balancers use IP stickiness to keep affinity between clients and servers.  This is bad when testing because it means that if you run a test script from a single machine it will only target a single machine in the cluster....  Make sure when testing with a software load balancer that it uses HTTP cookie affinity rather than IP address affinity.
• Poor testing with BPEL drivers
  Often we use BPEL processes as test harnesses to exercise functionality.  This is good but it doesn't work well in a cluster without some modifications to the driver process.  By default BPEL will optimize communications, within the same JVM it will use Java calls, between JVMs in the same cluster it will use ORMI, it only uses HTTP if it has to.  To test properly we need to distribute load through the load balancer and hence want to use HTTP.  In the invokes we can use the property optSoapShortcut=false to force calls to go through HTTP and hence the load balancer.

The above isn't an exhaustive list.  If you have others then feel free to share them, I would love to add more to the list.

In my last entry I spoke about some of the gotchas that are involved in setting up a cluster.  Over the next few entries I am going to describe how to build a SOA Suite 11g cluster for use in a test environment.  In this entry we will look at the target architecture and the required software.

Target Architecture

I am going to build my 11g cluster on 3 machines.

• Machine DB will host an 11gR1 database.  I will also use it to host a software load balancer (I will use WebCache).
• Machine SOA1 will host two WebLogic installations.  A WebLogic 11g installation will have a single SOA domain hosting a SOA Suite cluster, including BAM.  A WebLogic 10.3 installation will have a single OSB domain hosting an OSB cluster.
• Machine SOA2 will have the same software as SOA1 and will host the same two domains.

When OSB 11g is released then the need for two separate WebLogic installations will go away as the intention is for OSB and the rest of SOA Suite to run on the same WebLogic software version.

As there are two WebLogic domains then I will run an admin server on each machine, one domains admin server on SOA1 and the other domains admin server on SOA2.  This helps reduce the memory footprint.

Logically the architecture is shown below with a load balancer distributing load across the SOA and OSB clusters with a backend 11g database.

 

As for testing I don’t have a hardware load balancer then I run the load balancer on the same machine as the database to give the physical architecture shown below.

I will run this on 3 virtual machines on a server with 8gb memory, allowing 2gb for each virtual machine.

As a large number of customers seem to be running Linux these days I will use Oracle Enterprise Linux 5.3.  I will use 64-bit Linux for the DB machine and 32-bit Linux for the SOA machines.

It is quite common for clusters to be using a RAC database rather than a single instance database, but that was one VM too many for me to get my head around.

Software Required

So now we have identified the logical and physical architecture we need to identify what software we will require.  The software used is all available for download from OTN by clicking on the software link as shown in the table below.  Our target machine for the software is also shown in the table.

Software	Purpose	Target	Notes
Oracle WebLogic Server 11g Rel 1	Required for SOA Suite	SOA1, SOA2
SOA Suite	Core SOA Suite	SOA1, SOA2
Oracle Service Bus 10gR3	Service Bus	SOA1, SOA2	11g release will be available shortly.
Repository Creation Utility	Creates Meta-Data repository for SOA Suite	DB	May be run from any machine with network access to database.
Oracle Database 11g Release 1	Holds Meta-Data repository for SOA Suite	DB	Any database certified with SOA Suite may be used.
Web Tier Utilities	Contains Web Cache for use as a load balancer	DB	Another load balancer may used.
Enterprise Linux	Operating System	SOA1, SOA2, DB	Any OS certified with database or SOA Suite may be used.  DB machine may be a different OS to SOA machines.

 

Load Balancing

There are a number of software load balancers available, including functionality built into Linux so why did I use WebCache.  Well there are a number of reasons.

1. I like WebCache
2. It has a nice web based UI for configuring and monitoring
3. It supports cookie based affinity (see previous post for importance of this)
4. It does the job

Just be careful when using WebCache with SOA Suite that you do not use it to cache data.  To my knowledge no testing has been done within Oracle with using WebCache in conjunction with SOA Suite 11g so don’t deploy it in a production environment.

I have to confess that the idea of using WebCache as a load balancer was not mine, but my colleague Nick Cosmidis, so thanks Nick.

Other Resources

Setting up a cluster requires shared storage, ideally for the domain home but also for shared resources such as JMS message stores.  I could have used an iSCSI appliance to provide this but I chose instead to use the DB machine as a shared file server for the mid-tier components.

The cluster also requires IP addresses.  Obvious but there are different requirements for those IP addresses.  The IP address for the load balancer must be routable from all the clients of the cluster.  The database, SOA Suite and OSB instances can have non-routable IP addresses as long as they can talk to each other and the the load balancer.  The clients don’t have to be able to access the database, SOA Suite or OSB directly because they will go through the load balancer.

Virtualization

I am running this on a virtualized environment, a single 8GB machine hosting all three machines.  The only software virtualization fully supported by Oracle is Oracle Virtual Machine.  That is not to say it won’t work on other software virtualization environments such as VMware, just that it is not fully supported on those environments.  For more information on Oracle’s support policy with respect to virtualization in general check out this link.  For specific information on VMware support then check Note 249212.1 in MetaLink.

We always talk about the virtues of loose coupling with SOA, and the service interface is a key component of this.  Often we need to extract the service interface from a deployed BPEL process in order to call the process, or make use some of the some of the same services that the BPEL process calls.  When we deploy a BPEL process both the WSDLs implemented by the process and the WSDLs invoked by the service are all available through the BPEL console.

Navigating to your Process

Obtaining the WSDLs of a process is very easy.  Log in to BPEL console and select the processes tab.

Then select the desired process.

This will display the process details.

Obtaining the WSDL Implemented by Our Process

To obtain the WSDL interface to the process click on the WSDL tab.  Ensure that you have the correct version selected.

This will bring up the WSDL associated with the partner link implemented by this process.

Obtaining the WSDLs Called by Our Process

To obtain the WSDLs called by the process then go to the descriptor tab.

This provides a list of all the partner links in the BPEL process, including the partner links implemented by the BPEL process.

From here we can select the WSDL that is needed.   We can also see the properties associated with the partner link.

Note that often the BPEL designer will have created a wrapper WSDL that adds partner link information that is required by BPEL.  If this is the case, and it will be the case for most external WSDL documents, then it is necessary to examine the wrapper WSDL and extract the location attribute of the WSDL import statement.

This can then be plugged into URL of the wrapper WSDL to provide the actual service WSDL.  Usuall this can also be obtained by removing the “Ref” suffice from the partner link WSDL.

This will provide the actual WSDL that is being used by our BPEL process, rather than the wrapper that references it.

Why Bother?

Why go to all this trouble, what is the value in obtaining the WSDLs used by a BPEL process?

Well there are several reasons.  A few are outlined below

• There may be difference in behavior between different environments and we wish to confirm that they are using the same WSDL definitions.
• We may need to create a test environment that requires us to emulate WSDL services provided.
• We may want to check endpoint details to ensure that we are able to navigate through firewalls in our environment.
• We may not have immediate access to the BPEL project and want to check some WSDL interface settings.
• We may just be nosy!

Download the Whole BPEL Process

It is also possible to download the whole BPEL suitcase (packaged process) by selecting the Manage tab of the process we want to download under the Processes tab.

At the bottom of this screen we can download the suitcase by clicking the Export Process button.  This downloads the suitcase as a zip file.

Summary

Often we are looking at deployed processes without easy access to the original JDeveloper project.  Through the BPEL console we can verify both individual WSDL interfaces and also download the whole BPEL source, making it easy to check what is happening.  So remember, next time you are faced with odd behavior and want to look at the BPEL source or partner link details, you don’t need to call the developer!