Today sees not only the official launch of SOA Suite 11g but also the availability of the software.  You can download it all now from OTN.  Downloads are available for Windows, Linux, Solaris and HP-UX.  You will need the following to develop and run 11g SOA Suite applications.

• Oracle WebLogic Server 11g R1 (with or without Oracle Enterprise Pack for Eclipse - OEPE).
• Oracle SOA Suite 11g R1 (includes BPEL, Human Workflow, Business Rules, BAM, Mediator, B2B, OWSM, User Messaging Service and Enterprise Manager Fusion Middleware Control)
• Oracle Repository Creation Utility 11g R1 (creates the database schemas needed by SOA Suite and other components)
• Oracle JDeveloper 11g R1 this also needs the SOA plug in which should be available through the JDev extensions mechanism.
• Documentation is also available online or may be downloaded.

Watch the launch webcast and enjoy the latest release of the software, it a big step forward for SOA Suite.

Building a SOA Suite Cluster

Having spent a couple of weeks working on a SOA Suite cluster thought I would share some thoughts around clustering and SOA Suite.  Clustering of both BPEL Process Manager and Oracle Service Bus is relatively straightforward but there are a few gotchas.  Both BPEL and SOA Suite are stateless in the way they implement clustering, however BPEL does of course persist state to a database.

SOA Suite Clusters

Both BPEL and OSB clusters expect to be fronted by a load balancer.  Both can provide load balancing through a front end web server but a hardware load balancer is the best approach as shown in the diagram.

In this example we have an Oracle Real Application Cluster database running on 3-nodes to provide a high availability database environment.  We then have two clusters.  A cluster of 5 BPEL Process Manager instances, all pointing to the same RAC database, and a cluster of 5 OSB instances.  The BPEL cluster and the OSB cluster are both fronted by two hardware load balancers.

The BPEL instances and the OSB instances are in an active-active mode, meaning that all nodes are processing requests at the same time.  The load balancers are in active-passive mode, meaning that one load balancer processes all the traffic with the other load balancer acting as a hot standby in case of failure of the first load balancer.

This configuration avoids a single point of failure as every component is duplicated.  The system has been sized to be able to sustain the expected load even in the event of losing a machine.  This is known as “n+1” architecture, meaning that we need “n” machines to meet the requirements and so we provide “n+1” machines to allow for machine failure.  In the example shown we actually have two machines more than we need in normal operation because the the RAC database is running an “n+1” configuration as well as the SOA Suite.  Note that by running OSB on the same machines as BPEL we reduce the amount of extra hardware needed for failover and also reduce the latency of OSB-BPEL communication.

In the event of a failure then only in-process requests would be impacted.  Any requests that are “idempotent” (meaning they can be resubmitted with no ill effects) can be set up to automatically retry, further reducing the impact of a software or hardware failure.  Both BPEL and OSB can be set to automatically retry requests in event of failure, making the failover transparent.

Note that clustering does not help if we have a site failure due to fire, flooding, power failure or air conditioning failure for example.  In those cases we would need to have some sort of disaster recovery site, perhaps using Oracle Data Guard to keep the sites in synch at the database level.

BPEL Clusters

A BPEL cluster is effectively defined by a shared dehydration store.  Synchronous interactions must be processed within a single BPEL server instance, as the client has connected to a socket and expects a response on that same socket.  Asynchronous interactions are like any other long running BPEL process and may start processing on one node and then have processing resumed on another node, either due to failure or some other event.  The dehydration store (an oracle RAC database in the example above) provides a common location for process state that allows any BPEL instance to resume execution of a process instance.

When installing a BPEL cluster the best place to start is the High Availability Guide.  This outlines that you create a BPEL cluster by doing the following:

• Get the address of the load balancer.
• Run the repository creation assistant to create the BPEL meta-data in the database.
• Install OHS and OC4J components on the machines.
• Install BPEL Process Manager into the app server instances installed previously.
• If using a RAC database make sure that the JNDI data sources are using all RAC nodes.  See the Enterprise Deployment Guide for instructions on using Fast Connection Failover.
• Configure BPEL in a cluster as outlined in the Enterprise Deployment Guide.
• Set up JGroups to make all nodes aware of each other in the cluster.  Note that in 11g Coherence will be used for this which will simplify configuration.
• Set enableCluster and ClusterName in the collaxa-config.xml file.
• Make sure the BPEL PM instances all use the load balancer address for server URL and callback URLs.  This ensures that in event of node failure requests and responses are rerouted to remaining instances.
• Deploy processes on each node to make sure that all components are available on all nodes.  If you don’t do this some processes will work because they don’t have dependencies on non-BPEL components, but others will not.

Once set up use the BPEL fault handling framework to make sure that any calls to OSB are automatically retried in event of failure.

OSB Clusters

An OSB domain may have a single OSB cluster.  This cluster is installed like any other cluster in WebLogic.  Details on configuring the cluster can be found in the Creating WebLogic Domains Using the Configuration Wizard documentation.  For normal operation the OSB cluster is completely stateless, however the metrics gathering and aggregation takes place in a singleton service that by default is assigned to the first machine created in the cluster.  If this server fails then metrics will stay in the message queues to which they are delivered until a new instance starts.

• Get address of the load balancer.
• Install OSB software onto each machine or into a single shared location.  If a shared location make sure that the reference to it is the same on each machine.
• Create an OSB domain
• Creating a cluster and machines and assign servers to machines as explained in the Creating WebLogic Domains Using the Configuration Wizard documentation.
• Use load balancer address for cluster address.
• If you are not using a shared file location for your OSB install then you need to copy the contents of the osb domain directory to all nodes.  This ensure that the correct scripts are available for the node manager to launch managed servers.
• Run node manager on each machine.
• You can now launch your admin server and start the managed servers one at a time.  It is recommended that you start the OSB server running the data collectors first.  This will avoid timeouts on the other machines in the cluster and ensure that metrics are available.
• If message queues are required to be highly available then they should use persistent storage either on a shared highly available disk (a SAN for example) or they should use database persistence.
• If you are using an Oracle database then you should configure your OSB domain to store metrics in the Oracle database rather than in pointbase as explained in Creating WebLogic Domains Using the Configuration Wizard.

When calling BPEL or external web services from OSB make sure that you specify retries to allow for node failure.  Intra-OSB calls should be done using “local” transport for efficiency.

Summary

Configuring a cluster is a bit more involved than configuring a single instance, but it is not massively more complicated and it does provide both scalability and high availability.  Both BPEL and OSB scale linearly with increased nodes.  The only limitation on BPEL is the load on the backend dehydration store.  So go ahead, enjoy a cluster!

OTN have just posted a podcast “Oracle SOA Suite Developer's Guide: An Architect's Perspective” featuring an interview by Bob Rhubart with myself and Matt Wright about our book, the Oracle SOA Suite Developers Guide.  Matt comes across sounding calm and assured and I come across as …, well I’ll let you decide for yourself.  In my defence the interview was about 11pm my time and 8am Matts time, with Bob splitting the difference at about 3pm.  Listen and Enjoy.

Installing SOA Suite 10.1.3 with an 11g Database

Just at a customer who has an 11g RAC database that he wants to use for his SOA repository.  If you try and install SOA Suite into an 11g database, it tells you that the database is not supported and the irca configuration assistant fails to find a java library.  11g is a certified platform for SOA Suite 10.1.3.4 so here is how to get it installed.

IRCA

Before installing the SOA Suite executables you need to run the irca script to create the SOA Suite schemas in the 11g database.  The irca script needs to be able to find an ojdbc14.jar file.  This file is not shipped with the 11g database which provides libraries for Java 5 and Java 6 rather than the almost obsolete Java 1.4.  This leaves you with a couple of options;

• Use an Oracle 10g home as your Oracle home, if you have one on the machine, when running irca.
• Copy the JDBC libraries from an Oracle 10g home (<Oracle10g_Home>/jdbc/lib) to the Oracle 11g jdbc location (<Oracle11g_Home>/jdbc/lib).

Having done this then the irca script should run fine and create the ORABPEL, ORESB and ORAWSM schemas for you in an 11g database.  You are now ready to run the SOA Suite installer.

Installer

When you run the installer to create the SOA Suite instance it will fail when checking for the existence of the SOA Suite schemas unless you patch the installer files first.  The way to do this is to download patch 6265268 from MetaLink and follow the instructions which basically require you to modify the install media as follows:

• Replace DBConnectQueries.jar
  • Move
    • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar
  • to
    • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar.pre6265268
  • Copy
    • <PATCH>/DBConnectQueries.jar
  • from the patch to
  • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar
  • Note that the current patch documentation incorrectly refers to an 8.5 directory rather than the 8.4 that actually exists.
• Replace IP_DBQueries.jar
  • Move
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar
  • to
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar.pre6265268
  • Copy
    • <PATCH>/IP_DBQueries.jar
  • from the patch to
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar

You can now launch the installer and there will be no complaints about an 11g database.  Note that if you are installing a SOA Suite cluster then this will need to be done for each SOA Suite instance being installed.

What’s the Point?

As you are making these changes your mind keeps asking why you are doing this.  Apart from being told to do so by your DBAs there are some good reasons for using 11g.  The 11g database is the most manageable Oracle database ever, and several options only work on the 11g database.  11gR1 has been out for a long time now and so from a longevity perspective its best to put new deployments on an 11gR1 platform even if you have a release-1 strategy to avoid “bleeding edge” deployments.

Finally as you can see, it is not that hard to use an 11g database with SOA Suite.

I have just been with a customer who was using Oracle Enterprise Linux 5.  Now this shouldn’t be any different from other Linux installations except for one minor problem, the SOA Suite installer insists on checking that the Linux flavor is one explicitly supported by SOA Suite.  Well OEL wasn’t in the list when SOA Suite 10.1.3.1 came out and so the installer fails on the pre-requisite checks and won’t go any further, even if you applied all the required patches to the OS.  Fortunately there is a patch 6339508 which provides replacement pre-requisite tests.  To use it you need to add two parameters to the runInstaller command as shown below.

./runInstaller PREREQ_CONFIG_LOCATION=<PATCH_LOC>/prereq –paramFile <PATCH_LOC>/oraparam.ini

PATCH_LOC is the location where you unzipped the 6339508 patch.  With these new parameters the installer correctly recognizes Oracle Enterprise Linux as a supported platform.  Note that OEL needs some specific patches when used with SOA Suite and these need to be installed prior to running the installer.  Check the documentation for details.

I am seeing an increasing number of customers taking up OEL, I think because of the attraction of a single company supporting the whole technology infrastructure stack, OS, AS and DB.  I expect to see more of this still in the future.

One of Oracles chosen charities in the UK is the NSPCC, the National Society for the Prevention of Cruelty to Children.  The NSPCC is the UK's leading charity specialising in child protection and the prevention of cruelty to children. It is the only children's charity with statutory powers, enabling it to act to safeguard children.  In the UK it is well known for running Child Line, a 24-hour helpline for children in distress or danger. Trained volunteer counsellors comfort, advise and protect children and young people who may feel they have nowhere else to turn.

This year together with a number of other Oracle UK employees I will be participating in the Three Peaks Challenge in July - climbing to the top of Ben Nevis, Scafell Pike and Snowdon, all within 24 hours.

This will be no mean feat - it involves making your own sandwiches as well as walking for 15 hours/30 miles up/down over 10,000 feet interspersed with 500miles of driving. I've been walking up and down stairs in the office for weeks in preparation.

If you would like to sponsor me in this somewhat madcap activity, then donations can be made via PayPal, Credit or Debit Card at my Justgiving site.  If you’re a UK taxpayer, Justgiving makes sure 25% in Gift Aid, plus a 3% supplement, are added to your donation.

So go ahead, sponsor me!

The Problem

The native format builder wizard in the file adapter is great at reading flat file structures but doesn’t support reading more structured file structures.  Sometimes we need to read more complex structures such as master-detail records.  Let’s look at how we can use the file adapter to read structured file formats.

For example imagine a laundry list file such as the one below:

P,101,James L,Shirt,2,Starch L,Socks,6,De-odorise L,Pants,2,Remove Stains

P,220,JoJo L,Sweatshirt,1,Handwash L,Shirt,3,No Iron L,Socks,2,Iron L,Pants,2,Steam Press L,Tie,1,Dry clean

P,305,Ruth L,Skirt,7,Iron Pleats L,Socks,8,Iron

To make it easier to see the record structure I have put boxes around the individual records.  This can be represented in XML as shown below

The native format builder wizard would either treat each line as either the same type of record (“Multiple records are of single type”) or as two different types of record (“Multiple records are of different types”).

In the first case we would have a single record type with four fields that fails to distinguish between people (lines pre-fixed with ‘P’) and laundry items (lines pre-fixed with ‘L’).  This does not reflect the fact that the two lines are of different types.  The second case, multiple records are of different types, is closer to what we want.  It creates two records types, the type being determined by the value of the first field.  The XML for this is shown below:

However this case creates a list of two record types with no recognition that one record is nested inside the other.

Comparing the two XML representations we can see what needs to be done; we need to add an Items element under Room and move the Item element to be under Items.  The question is how do we describe this using the native schema constructs supported by the file adapter.

Creating a Master-Detail Records Native Format Schema

The easiest way to deal with this is to use the native format builder wizard in the file adapter to create the basic outline of the records for us.  We choose “Multiple records are of different types” and provide appropriate names for the two records types and individual fields within the records.

Having created the basic native format schema we can now edit it to be exactly the way we want it for the laundry list.

We make the following changes in the generated native schema file:

• Replace the <choice> element by a <sequence> element.  In conjunction with other changes this will give us a list of elements of the same type, rather than of list of elements of two different types.

  • <xsd:choice minOccurs="1" maxOccurs="unbounded"     nxsd:choiceCondition="terminated"     nxsd:terminatedBy=",">     … </xsd:choice>

    <xsd:sequence>    … </xsd:sequence>

• Replace the conditionValue attributes with startsWith attributes and add a comma to the end of the attribute values.  This will allow the native schema processor to identify the start of master records and child records.  We also add a maxOccurs attribute to the elements modified so that they can have multiple instances in a sequence.

  • <xsd:element name="Item"     nxsd:conditionValue="L"> … </xsd:element>	<xsd:element name="Item"     nxsd:startsWith="L,"     maxOccurs="unbounded"> … </xsd:element>
    <xsd:element name="Room"     nxsd:conditionValue="P"> … </xsd:element>	<xsd:element name="Room"     nxsd:startsWith="P,"     maxOccurs="unbounded">     … </xsd:element>

• Add an <Items> element as a sequence to the <Room> element sequence and move the <Item> element to be inside the <Items> sequence.

  <xsd:element name="Room" nxsd:startsWith="P," maxOccurs="unbounded">
      <xsd:complexType>
          <xsd:sequence>
              …
              <xsd:element name="Items" maxOccurs="1">
                  <xsd:complexType>
                      <xsd:sequence>
                          <xsd:element name="Item" nxsd:startsWith="L," maxOccurs="unbounded">
                              …
                          </xsd:element>
                      </xsd:sequence>
                  </xsd:complexType>
              </xsd:element>
          </xsd:sequence>
      </xsd:complexType>
  </xsd:element>

This gives us a native schema format that looks like this:

Note that this is the structure that we were aiming for in the first place.

Sample Code

I have created a simple BPEL process that performs the following steps

• Read a laundry file using the wizard generated native schema format.  This creates an XML document with two distinct records types.
  • Note that in the process I do not delete the file so it is important to make sure that if you re-use the same file with the process you need to update its timestamp!
• Read the laundry file again using the modified native schema format.  This creates an XML document with master-detail style records which reflects the actual format of the file.
  • Note that I use a synch read with the filename provided by the inbound header from the previous file read.   I delete the input file after reading it.
• Write the laundry file using a pure XML schema to create an XML document file of the laundry list.

Sample code is zipped up as a JDeveloper project and can be downloaded here.  In addition to the normal BPEL artifacts within the project, I have included a sample laundry file (laundry.sample.txt) in the top level of the project directory.  After deploying to a BPEL server the process will look in C:\FileTransfer\InBound for the input file, after processing the input file it is written to C:\FileTransfer\Processed and the output file is generated in C:\FileTransfer\OutBound.  Either create appropriate directories or edit the project to use new directories.

Documentation

This entry has used the facilities of the native format schemas in SOA Suite.  These are documented in chapter 7 of the Oracle® Application Server Adapters for Files, FTP, Databases, and Enterprise Messaging User's Guide.

Good luck in using the adapters to process master-detail records!

Last Wednesday the Open Group made available their SOA Source Book.  They describe it as “a collection of source material produced by the SOA Working Group for use by enterprise architects working with Service-Oriented Architecture”.  Having looked at it I have to say it seems to have a high information density and would be a good place to start getting someone's head around SOA concepts.  Well worth a look.

Tuxedo Connections or the On Ramp to Tux

Tuxedo can be considered as the original and purest service oriented architecture.  The key abstraction in Tuxedo is the service and everything is made to fit into the service mould.  It seems strange then that people think of Tuxedo as a legacy application.  Tuxedo is highly regarded by the senior management team in Oracle who view it as a key tool to support extreme transaction processing.  The question is then, how does this relate to the rest of the SOA world which does not subscribe to the Tuxedo technologies such ATMI, C++ or COBOL.

The diagram below shows the different interfaces into and out of the Tuxedo world.  Lets look at them briefly and how they relate to the rest of the SOA world which is focussed on XML, SOAP and HTTP.

Native Client Interfaces

The native client interfaces to Tuxedo are the C, C++, .Net or COBOL client interfaces using ATMI (Application to Transaction Monitor Interface).  There is also a version of ATMI for Java, called Jolt.  These interfaces allow clients to invoke Tuxedo services and get responses.  They do not allow Tuxedo to invoke services in these clients except by listening on Tuxedo message queues, these interfaces are asymmetric.

Legacy Client and Server Interfaces

A Tuxedo domain can interface to other Tuxedo domains, treating their services as though they were its own services.  This capability is extended to other systems such as mainframe systems.  The external systems see Tuxedo services as native services and invoke them as they would any other service, similarly Tuxedo sees these external systems as native Tuxedo services and invokes them as it would any other service.  This provides relatively seamless integration between legacy environments and Tuxedo and allows either side to operate as a server to the other, in other words these interfaces are symmetric.

Open System Interfaces

In addition to treating legacy mainframe interfaces and other Tuxedo domains in the same way as local services Tuxedo can also do this for such open system standards as CORBA and java code running in WebLogic Server.  CORBA applications can invoke Tux services through the CORBA API and Tuxedo services can invoke CORBA objects.  The WebLogic Tuxedo Connector (WTC) extends the capabilities of Jolt to become fully symmetric in that EJBs in WebLogic can be invoked as services from Tuxedo.

A Transactional Note

Note that all the interfaces we have spoken about so far are transactional, as in they are part of the Tuxedo transaction infrastructure, invoking a remote mainframe transaction may cause an XA transaction to be started within Tuxedo.  When calling an EJB in WebLogic this also is part of the overall Tuxedo XA transaction infrastructure.

Web Service Access to Tuxedo

There are two alternative ways to get Web Services to access Tuxedo.  The most obvious is to use SALT (Service Architecture Leveraging Tuxedo) which exposes Tuxedo services as web services, and allows Tuxedo to invoke Web Services as though they were Tuxedo services.  This is a symmetric interface and takes care of all the XML to Tuxedo translations but it is not transactional.  The web service call is not part of the transaction.  A web service request to Tuxedo may cause a Tux transaction to be initiated, but webs services don’t currently provide a transactional context.  Similarly when Tux makes a call to a web service, that call is not part of any Tuxedo transaction.

So what if you want transactionality and access to web services.  This is where the service bus comes in.  The Oracle Service Bus (OSB) takes advantage of the WebLogic Tuxedo Connector to provide a fast efficient and transactional interface to and from the Tuxedo world.  This allows a pipeline to make two seperate calls to Tuxedo as part of the same transaction.  Note that there are a couple fo wrinkles to making this happen and I will deal with those in a later post.

Summary – Long Live Tux, King of Services

Not only is Tuxedo the original service oriented architecture but despite being more than 20 years old, through SALT and WebLogic Tuxedo Connector it still speaks the modern lingo of service buses, Java, XML, SOAP and HTTP.  So if you have a Tuxedo investment don’t write it off, but look at how you can more easily make your Tux services available to the new fangled XML based web service world.

Oracle SOA Suite Developer’s Guide

My friend Matt Wright just pointed out to me that I hadn’t mentioned that our book was now available.  Thanks to the guys at Packt Publishing for guiding us through the process and publishing the book.  Our focus when writing the book was to provide a practitioners guide to implementing SOA using the Oracle SOA Suite.  As such we have set each component of the SOA Suite within the context in which might be used.  This seems to be the area that a lot of companies starting down a SOA path struggle with.  They are unsure what bits of technology to use for what.

Dave Shaffer, Oracle VP for SOA, kindly wrote a foreword for us.

Writing the book has been a huge undertaking for both Matt and myself.  During the writing of the book the 11g release was seriously delayed so we switched versions from 11 to 10.1.3 (the current production release).  Matt decided that he was sick of 365 days of rain and moved his family from England to Australia.  Finally Oracle purchased BEA and caused a re-evaluation of the Service Bus technologies.

I am pleased with the way the book has turned out, and Matt and I learnt a huge amount as we prepared the book, both of us had to delve into area with we weren’t totally familiar.  Matt became a master of the Rules engine and I discovered the marvelous new deployment capabilities added to the 10.1.3.4 release.

We have made every effort to match the book to the current production 10.1.3.4 release so get yourself a copy and give us some feedback.