We are getting a new team member at the start of next month and he asked me if I could recommend any resources for him to get up to speed on SOA and Oracle SOA Suite. So here is a list of resources that I provided him.

Web Sites

• Oracle SOA Architect Center
• Oracle SOA Suite Home Page
• SOA Specs Information (Thomas Erl)

Blogs

The following bloggers I find useful

• My Blog :-)
• Matt Wright
• Ramkumar Menon
• IT Eye  (often Mike van Alst on SOA)
• Mike van Alst
• Clemens Utschig
• Marc Kelderman
• SOA/BPM PM Blog (often Clemens Utschig)
• Richard Veryard

Books

I have created a book list on Amazon.

We always talk about the virtues of loose coupling with SOA, and the service interface is a key component of this.  Often we need to extract the service interface from a deployed BPEL process in order to call the process, or make use some of the some of the same services that the BPEL process calls.  When we deploy a BPEL process both the WSDLs implemented by the process and the WSDLs invoked by the service are all available through the BPEL console.

Navigating to your Process

Obtaining the WSDLs of a process is very easy.  Log in to BPEL console and select the processes tab.

Then select the desired process.

This will display the process details.

Obtaining the WSDL Implemented by Our Process

To obtain the WSDL interface to the process click on the WSDL tab.  Ensure that you have the correct version selected.

This will bring up the WSDL associated with the partner link implemented by this process.

Obtaining the WSDLs Called by Our Process

To obtain the WSDLs called by the process then go to the descriptor tab.

This provides a list of all the partner links in the BPEL process, including the partner links implemented by the BPEL process.

From here we can select the WSDL that is needed.   We can also see the properties associated with the partner link.

Note that often the BPEL designer will have created a wrapper WSDL that adds partner link information that is required by BPEL.  If this is the case, and it will be the case for most external WSDL documents, then it is necessary to examine the wrapper WSDL and extract the location attribute of the WSDL import statement.

This can then be plugged into URL of the wrapper WSDL to provide the actual service WSDL.  Usuall this can also be obtained by removing the “Ref” suffice from the partner link WSDL.

This will provide the actual WSDL that is being used by our BPEL process, rather than the wrapper that references it.

Why Bother?

Why go to all this trouble, what is the value in obtaining the WSDLs used by a BPEL process?

Well there are several reasons.  A few are outlined below

• There may be difference in behavior between different environments and we wish to confirm that they are using the same WSDL definitions.
• We may need to create a test environment that requires us to emulate WSDL services provided.
• We may want to check endpoint details to ensure that we are able to navigate through firewalls in our environment.
• We may not have immediate access to the BPEL project and want to check some WSDL interface settings.
• We may just be nosy!

Download the Whole BPEL Process

It is also possible to download the whole BPEL suitcase (packaged process) by selecting the Manage tab of the process we want to download under the Processes tab.

At the bottom of this screen we can download the suitcase by clicking the Export Process button.  This downloads the suitcase as a zip file.

Summary

Often we are looking at deployed processes without easy access to the original JDeveloper project.  Through the BPEL console we can verify both individual WSDL interfaces and also download the whole BPEL source, making it easy to check what is happening.  So remember, next time you are faced with odd behavior and want to look at the BPEL source or partner link details, you don’t need to call the developer!

In my last entry I spoke about some of the gotchas that are involved in setting up a cluster.  Over the next few entries I am going to describe how to build a SOA Suite 11g cluster for use in a test environment.  In this entry we will look at the target architecture and the required software.

Target Architecture

I am going to build my 11g cluster on 3 machines.

• Machine DB will host an 11gR1 database.  I will also use it to host a software load balancer (I will use WebCache).
• Machine SOA1 will host two WebLogic installations.  A WebLogic 11g installation will have a single SOA domain hosting a SOA Suite cluster, including BAM.  A WebLogic 10.3 installation will have a single OSB domain hosting an OSB cluster.
• Machine SOA2 will have the same software as SOA1 and will host the same two domains.

When OSB 11g is released then the need for two separate WebLogic installations will go away as the intention is for OSB and the rest of SOA Suite to run on the same WebLogic software version.

As there are two WebLogic domains then I will run an admin server on each machine, one domains admin server on SOA1 and the other domains admin server on SOA2.  This helps reduce the memory footprint.

Logically the architecture is shown below with a load balancer distributing load across the SOA and OSB clusters with a backend 11g database.

As for testing I don’t have a hardware load balancer then I run the load balancer on the same machine as the database to give the physical architecture shown below.

I will run this on 3 virtual machines on a server with 8gb memory, allowing 2gb for each virtual machine.

As a large number of customers seem to be running Linux these days I will use Oracle Enterprise Linux 5.3.  I will use 64-bit Linux for the DB machine and 32-bit Linux for the SOA machines.

It is quite common for clusters to be using a RAC database rather than a single instance database, but that was one VM too many for me to get my head around.

Software Required

So now we have identified the logical and physical architecture we need to identify what software we will require.  The software used is all available for download from OTN by clicking on the software link as shown in the table below.  Our target machine for the software is also shown in the table.

Load Balancing

There are a number of software load balancers available, including functionality built into Linux so why did I use WebCache.  Well there are a number of reasons.

1. I like WebCache
2. It has a nice web based UI for configuring and monitoring
3. It supports cookie based affinity (see previous post for importance of this)
4. It does the job

Just be careful when using WebCache with SOA Suite that you do not use it to cache data.  To my knowledge no testing has been done within Oracle with using WebCache in conjunction with SOA Suite 11g so don’t deploy it in a production environment.

I have to confess that the idea of using WebCache as a load balancer was not mine, but my colleague Nick Cosmidis, so thanks Nick.

Other Resources

Setting up a cluster requires shared storage, ideally for the domain home but also for shared resources such as JMS message stores.  I could have used an iSCSI appliance to provide this but I chose instead to use the DB machine as a shared file server for the mid-tier components.

The cluster also requires IP addresses.  Obvious but there are different requirements for those IP addresses.  The IP address for the load balancer must be routable from all the clients of the cluster.  The database, SOA Suite and OSB instances can have non-routable IP addresses as long as they can talk to each other and the the load balancer.  The clients don’t have to be able to access the database, SOA Suite or OSB directly because they will go through the load balancer.

Virtualization

I am running this on a virtualized environment, a single 8GB machine hosting all three machines.  The only software virtualization fully supported by Oracle is Oracle Virtual Machine.  That is not to say it won’t work on other software virtualization environments such as VMware, just that it is not fully supported on those environments.  For more information on Oracle’s support policy with respect to virtualization in general check out this link.  For specific information on VMware support then check Note 249212.1 in MetaLink.

Since moving to support I have learned a lot about clustering.  Some of the things I have learnt are;

• Lots of customers are running SOA Suite clusters
• Lots of them haven't read the High Availability Guide (10g or 11g)
• Lots of them haven't read the Enterprise Deployment Guide or EDG (10g or 11g)
• Many of them have problems because of the points above

Part of the problem for many customers is that setting up a cluster has a lot of steps and a few gotchas that can come back to bite you.  Just got off the phone with a customer who was having problems with a cluster install, nothing too serious but irritating and slowing him down.  Unless the HA & EDG are followed very carefully it is easy to make mistakes.  A few common problem areas I have seen are

• Failing to separate the design and run time of the ESB
  The ESB design time is a singleton and there must never be more than one instance of the ESB design time active against the same repository at the same time.
• Poor configuration of JGroups or Coherence
  In 10g JGroups is used to identify cluster membership, in 11g Coherence plays the same role.  If these are not configured consistently across the cluster then one part of the cluster may be unaware of the existence of other parts of the cluster with dire circumstances for some shared resources.
• Failure to set up virtual addresses correctly
  The cluster should have a single virtual address. This virtual address needs to be configured in the HTTP listener, into the OC4J servlet engine and into the BPEL URL settings, amongst other places.  Failure to do this can lead to odd behavior.
• Failure to test on a cluster
  Seems many companies have clusters in production but not in test and dev.  Surely no-one is that stupid you say.  Well they may configure test instances with a cluster but for resource reasons run only one node of the cluster, after all a one node cluster is still a cluster right...  They then wonder why they only get certain problems in production and can't reproduce them in test.
• Many customers fail to have suitable test load balancers
  Often customer will not have a hardware load balancer for use in their test or dev environments and will rely on a software load balancer.  Some of these software load balancers use IP stickiness to keep affinity between clients and servers.  This is bad when testing because it means that if you run a test script from a single machine it will only target a single machine in the cluster....  Make sure when testing with a software load balancer that it uses HTTP cookie affinity rather than IP address affinity.
• Poor testing with BPEL drivers
  Often we use BPEL processes as test harnesses to exercise functionality.  This is good but it doesn't work well in a cluster without some modifications to the driver process.  By default BPEL will optimize communications, within the same JVM it will use Java calls, between JVMs in the same cluster it will use ORMI, it only uses HTTP if it has to.  To test properly we need to distribute load through the load balancer and hence want to use HTTP.  In the invokes we can use the property optSoapShortcut=false to force calls to go through HTTP and hence the load balancer.

The above isn't an exhaustive list.  If you have others then feel free to share them, I would love to add more to the list.

I haven't written anything for a couple of months, but in my defence I would like to point out that I have changed jobs at Oracle.  I now work in global support services, supporting the Oracle SOA products.  As part of this change I have moved from Easter Compton in England to Monument, Colorado, moving myself, wife and 4 children a quarter of a the way around the world has proven more challenging than expected.

A few weeks ago I was in Orlando on an HA training course for 11g SOA Suite and I'll be sharing some stuff about this in future blogs.  I also put together a 1 day course for support engineers around SOA Architectures and again I will post some of this stuff over the next couple of weeks.

But this week I am in San Francisco for Open World.  If those who know want to stop by and see me I will be in the Support Stars Bar on Monday afternoon and Tuesday morning.  On Wednesday at 12:30 I have a book signing at the bookstore.  So stop by and say hello.

This is my first US Open World and I am looking forward to the chance to catch up with old friends and hopefully make some new ones.

As part of the three peaks challenge which I completed this week the Oracle team were challenged to get a picture of someone reading Matt and my book - the SOA Suite Developers Guide -on top of each peak.  Thought I would share the story and the pictures with you.

The objective of the three peaks challenge is to walk up the highest peak in each of England, Scotland and Wales in a single 24 hour period.

Peak 1 - Ben Nevis

Ben Nevis in Scotland is the highest peak of the three at 1344m.  We started our ascent at 5pm on Friday and reached the summit by 7:30pm.  As can be seen on the picture of myself and Neil Spink reading the SOA Suite book, weather at the top was clear so that we could see for miles.  The weather was hot and sunny and I struggled a little on the way down but we were all off the mountain by 9:45pm and on the road to Scafell Pike in the Lake District, not far from my brother-in-laws house.  We left for Scafell at 9:51pm.

Peak 2 - Scafell Pike

Scafell Pike in England is the smallest of the three peaks at 978m.  We arrived at 3:32am, just as the sky was lightening up in the east.  The moon was also very bright with little cloud cover so we had no problem seeing our route and we reached the summit by 5:30am just as the sun was rising above the surrounding peaks of the Lake District.  There was little cloud about and Daniel Roberts and myself could easily see the book at the top.  We were all back down the mountain by 7:17pm and on the road to Snowdon.  Our drivers might have been controlling 17 seater mini-buses, but to the occupants we felt like we were participating in a particularly tough rally as we wound our way out through the tortuous lanes on our way to North Wales.

Peak 3 - Snowdon

The final peak, Snowdon, is the tallest mountain in England and Wales at 1,085m and was challenging in two ways.  When we arrived at 11:50am we had already spent more than 8 hours on mountains and more than 11 hours crammed into what seemed to be a shrinking mini-bus, so we were not at our most energetic.  Also at Snowdon the weather was failing and it was beginning to rain.  However arrival at Snowdon energised us and within a minute of arrival we were striking out towards the mountain.  The initial approach up the Pyg Track was fairly easy going, with just a scramble at the end to reach the summit by 1:45pm.  The summit was cloudy and wet so no-one wanted to hang around for long, especially myself and Andy Gale who found the book rather damp going.  The descent was also very quick, using the Miners Track to drop down off the mountain very rapidly, followed by what felt like a long run/walk around the mountain back to the start point.  We had all arrived back by 4:02pm, meaning that as a group we completed the challenge in 23 hours and 2 minutes.

Thanks

Special thanks must go to our drivers, Colin, James, Martin and Jackie who got us between peaks rapidly and safely.  I was very appreciative of Andy Gale who made sure I kept pace on Scafell Pike and Snowdon.  Finally the determination award must go to Neil Spink who despite blisters and damaged knees climbed Snowdon and completed the challenge.

Sponsorship

Apart from wanting to show that we were not all over the hill (most of us were well over 40!) we also wanted to raise money for the NSPCC.  We are still a few hundred pounds shy of raising £10,000 for this charity so please feel free to go to my justgiving page and sponsor me.

Statistics

The following is from a GPS tracker we took with us.

Ben Nevis - start at 51m, Top 1,347m  (include the trig point I guess), distance 15.1km or 9.38 miles

Scafell Pike - start at 29m, Top  985m, distance 9 km or.5.59 miles

Snowdon - Lowest Point 376m, Top 1,088m , distance 12.4 KM or 7.7miles

Total Climb = 1296 + 956 + 712 = 2,964m or 9,724 ft

Total Distance = 22.67miles

Today sees not only the official launch of SOA Suite 11g but also the availability of the software.  You can download it all now from OTN.  Downloads are available for Windows, Linux, Solaris and HP-UX.  You will need the following to develop and run 11g SOA Suite applications.

• Oracle WebLogic Server 11g R1 (with or without Oracle Enterprise Pack for Eclipse - OEPE).
• Oracle SOA Suite 11g R1 (includes BPEL, Human Workflow, Business Rules, BAM, Mediator, B2B, OWSM, User Messaging Service and Enterprise Manager Fusion Middleware Control)
• Oracle Repository Creation Utility 11g R1 (creates the database schemas needed by SOA Suite and other components)
• Oracle JDeveloper 11g R1 this also needs the SOA plug in which should be available through the JDev extensions mechanism.
• Documentation is also available online or may be downloaded.

Watch the launch webcast and enjoy the latest release of the software, it a big step forward for SOA Suite.

Building a SOA Suite Cluster

Having spent a couple of weeks working on a SOA Suite cluster thought I would share some thoughts around clustering and SOA Suite.  Clustering of both BPEL Process Manager and Oracle Service Bus is relatively straightforward but there are a few gotchas.  Both BPEL and SOA Suite are stateless in the way they implement clustering, however BPEL does of course persist state to a database.

SOA Suite Clusters

Both BPEL and OSB clusters expect to be fronted by a load balancer.  Both can provide load balancing through a front end web server but a hardware load balancer is the best approach as shown in the diagram.

In this example we have an Oracle Real Application Cluster database running on 3-nodes to provide a high availability database environment.  We then have two clusters.  A cluster of 5 BPEL Process Manager instances, all pointing to the same RAC database, and a cluster of 5 OSB instances.  The BPEL cluster and the OSB cluster are both fronted by two hardware load balancers.

The BPEL instances and the OSB instances are in an active-active mode, meaning that all nodes are processing requests at the same time.  The load balancers are in active-passive mode, meaning that one load balancer processes all the traffic with the other load balancer acting as a hot standby in case of failure of the first load balancer.

This configuration avoids a single point of failure as every component is duplicated.  The system has been sized to be able to sustain the expected load even in the event of losing a machine.  This is known as "n+1" architecture, meaning that we need "n" machines to meet the requirements and so we provide "n+1" machines to allow for machine failure.  In the example shown we actually have two machines more than we need in normal operation because the the RAC database is running an "n+1" configuration as well as the SOA Suite.  Note that by running OSB on the same machines as BPEL we reduce the amount of extra hardware needed for failover and also reduce the latency of OSB-BPEL communication.

In the event of a failure then only in-process requests would be impacted.  Any requests that are "idempotent" (meaning they can be resubmitted with no ill effects) can be set up to automatically retry, further reducing the impact of a software or hardware failure.  Both BPEL and OSB can be set to automatically retry requests in event of failure, making the failover transparent.

Note that clustering does not help if we have a site failure due to fire, flooding, power failure or air conditioning failure for example.  In those cases we would need to have some sort of disaster recovery site, perhaps using Oracle Data Guard to keep the sites in synch at the database level.

BPEL Clusters

A BPEL cluster is effectively defined by a shared dehydration store.  Synchronous interactions must be processed within a single BPEL server instance, as the client has connected to a socket and expects a response on that same socket.  Asynchronous interactions are like any other long running BPEL process and may start processing on one node and then have processing resumed on another node, either due to failure or some other event.  The dehydration store (an oracle RAC database in the example above) provides a common location for process state that allows any BPEL instance to resume execution of a process instance.

When installing a BPEL cluster the best place to start is the High Availability Guide.  This outlines that you create a BPEL cluster by doing the following:

• Get the address of the load balancer.
• Run the repository creation assistant to create the BPEL meta-data in the database.
• Install OHS and OC4J components on the machines.
• Install BPEL Process Manager into the app server instances installed previously.
• If using a RAC database make sure that the JNDI data sources are using all RAC nodes.  See the Enterprise Deployment Guide for instructions on using Fast Connection Failover.
• Configure BPEL in a cluster as outlined in the Enterprise Deployment Guide.
• Set up JGroups to make all nodes aware of each other in the cluster.  Note that in 11g Coherence will be used for this which will simplify configuration.
• Set enableCluster and ClusterName in the collaxa-config.xml file.
• Make sure the BPEL PM instances all use the load balancer address for server URL and callback URLs.  This ensures that in event of node failure requests and responses are rerouted to remaining instances.
• Deploy processes on each node to make sure that all components are available on all nodes.  If you don't do this some processes will work because they don't have dependencies on non-BPEL components, but others will not.

Once set up use the BPEL fault handling framework to make sure that any calls to OSB are automatically retried in event of failure.

OSB Clusters

An OSB domain may have a single OSB cluster.  This cluster is installed like any other cluster in WebLogic.  Details on configuring the cluster can be found in the Creating WebLogic Domains Using the Configuration Wizard documentation.  For normal operation the OSB cluster is completely stateless, however the metrics gathering and aggregation takes place in a singleton service that by default is assigned to the first machine created in the cluster.  If this server fails then metrics will stay in the message queues to which they are delivered until a new instance starts.

• Get address of the load balancer.
• Install OSB software onto each machine or into a single shared location.  If a shared location make sure that the reference to it is the same on each machine.
• Create an OSB domain
• Creating a cluster and machines and assign servers to machines as explained in the Creating WebLogic Domains Using the Configuration Wizard documentation.
• Use load balancer address for cluster address.
• If you are not using a shared file location for your OSB install then you need to copy the contents of the osb domain directory to all nodes.  This ensure that the correct scripts are available for the node manager to launch managed servers.
• Run node manager on each machine.
• You can now launch your admin server and start the managed servers one at a time.  It is recommended that you start the OSB server running the data collectors first.  This will avoid timeouts on the other machines in the cluster and ensure that metrics are available.
• If message queues are required to be highly available then they should use persistent storage either on a shared highly available disk (a SAN for example) or they should use database persistence.
• If you are using an Oracle database then you should configure your OSB domain to store metrics in the Oracle database rather than in pointbase as explained in Creating WebLogic Domains Using the Configuration Wizard.

When calling BPEL or external web services from OSB make sure that you specify retries to allow for node failure.  Intra-OSB calls should be done using "local" transport for efficiency.

Summary

Configuring a cluster is a bit more involved than configuring a single instance, but it is not massively more complicated and it does provide both scalability and high availability.  Both BPEL and OSB scale linearly with increased nodes.  The only limitation on BPEL is the load on the backend dehydration store.  So go ahead, enjoy a cluster!

OTN have just posted a podcast “Oracle SOA Suite Developer's Guide: An Architect's Perspective” featuring an interview by Bob Rhubart with myself and Matt Wright about our book, the Oracle SOA Suite Developers Guide.  Matt comes across sounding calm and assured and I come across as …, well I’ll let you decide for yourself.  In my defence the interview was about 11pm my time and 8am Matts time, with Bob splitting the difference at about 3pm.  Listen and Enjoy.

Installing SOA Suite 10.1.3 with an 11g Database

Just at a customer who has an 11g RAC database that he wants to use for his SOA repository.  If you try and install SOA Suite into an 11g database, it tells you that the database is not supported and the irca configuration assistant fails to find a java library.  11g is a certified platform for SOA Suite 10.1.3.4 so here is how to get it installed.

IRCA

Before installing the SOA Suite executables you need to run the irca script to create the SOA Suite schemas in the 11g database.  The irca script needs to be able to find an ojdbc14.jar file.  This file is not shipped with the 11g database which provides libraries for Java 5 and Java 6 rather than the almost obsolete Java 1.4.  This leaves you with a couple of options;

• Use an Oracle 10g home as your Oracle home, if you have one on the machine, when running irca.
• Copy the JDBC libraries from an Oracle 10g home (<Oracle10g_Home>/jdbc/lib) to the Oracle 11g jdbc location (<Oracle11g_Home>/jdbc/lib).

Having done this then the irca script should run fine and create the ORABPEL, ORESB and ORAWSM schemas for you in an 11g database.  You are now ready to run the SOA Suite installer.

Installer

When you run the installer to create the SOA Suite instance it will fail when checking for the existence of the SOA Suite schemas unless you patch the installer files first.  The way to do this is to download patch 6265268 from MetaLink and follow the instructions which basically require you to modify the install media as follows:

• Replace DBConnectQueries.jar
  • Move
    • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar
  • to
    • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar.pre6265268
  • Copy
    • <PATCH>/DBConnectQueries.jar
  • from the patch to
  • <MEDIA>/stage/Queries/DBConnectQueries/8.4/1/DBConnectQueries.jar
  • Note that the current patch documentation incorrectly refers to an 8.5 directory rather than the 8.4 that actually exists.
• Replace IP_DBQueries.jar
  • Move
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar
  • to
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar.pre6265268
  • Copy
    • <PATCH>/IP_DBQueries.jar
  • from the patch to
    • <MEDIA>/stage/Queries/IP_DBQueries/3.0/1/IP_DBQueries.jar

You can now launch the installer and there will be no complaints about an 11g database.  Note that if you are installing a SOA Suite cluster then this will need to be done for each SOA Suite instance being installed.

What’s the Point?

As you are making these changes your mind keeps asking why you are doing this.  Apart from being told to do so by your DBAs there are some good reasons for using 11g.  The 11g database is the most manageable Oracle database ever, and several options only work on the 11g database.  11gR1 has been out for a long time now and so from a longevity perspective its best to put new deployments on an 11gR1 platform even if you have a release-1 strategy to avoid “bleeding edge” deployments.

Finally as you can see, it is not that hard to use an 11g database with SOA Suite.