Antony Reynolds' Blog

I haven't written anything for a couple of months, but in my defence I would like to point out that I have changed jobs at Oracle.  I now work in global support services, supporting the Oracle SOA products.  As part of this change I have moved from Easter Compton in England to Monument, Colorado, moving myself, wife and 4 children a quarter of a the way around the world has proven more challenging than expected.

A few weeks ago I was in Orlando on an HA training course for 11g SOA Suite and I'll be sharing some stuff about this in future blogs.  I also put together a 1 day course for support engineers around SOA Architectures and again I will post some of this stuff over the next couple of weeks.

But this week I am in San Francisco for Open World.  If those who know want to stop by and see me I will be in the Support Stars Bar on Monday afternoon and Tuesday morning.  On Wednesday at 12:30 I have a book signing at the bookstore.  So stop by and say hello.

This is my first US Open World and I am looking forward to the chance to catch up with old friends and hopefully make some new ones.

Since moving to support I have learned a lot about clustering.  Some of the things I have learnt are;

• Lots of customers are running SOA Suite clusters
• Lots of them haven't read the High Availability Guide (10g or 11g)
• Lots of them haven't read the Enterprise Deployment Guide or EDG (10g or 11g)
• Many of them have problems because of the points above

Part of the problem for many customers is that setting up a cluster has a lot of steps and a few gotchas that can come back to bite you.  Just got off the phone with a customer who was having problems with a cluster install, nothing too serious but irritating and slowing him down.  Unless the HA & EDG are followed very carefully it is easy to make mistakes.  A few common problem areas I have seen are

• Failing to separate the design and run time of the ESB
  The ESB design time is a singleton and there must never be more than one instance of the ESB design time active against the same repository at the same time.
• Poor configuration of JGroups or Coherence
  In 10g JGroups is used to identify cluster membership, in 11g Coherence plays the same role.  If these are not configured consistently across the cluster then one part of the cluster may be unaware of the existence of other parts of the cluster with dire circumstances for some shared resources.
• Failure to set up virtual addresses correctly
  The cluster should have a single virtual address. This virtual address needs to be configured in the HTTP listener, into the OC4J servlet engine and into the BPEL URL settings, amongst other places.  Failure to do this can lead to odd behavior.
• Failure to test on a cluster
  Seems many companies have clusters in production but not in test and dev.  Surely no-one is that stupid you say.  Well they may configure test instances with a cluster but for resource reasons run only one node of the cluster, after all a one node cluster is still a cluster right...  They then wonder why they only get certain problems in production and can't reproduce them in test.
• Many customers fail to have suitable test load balancers
  Often customer will not have a hardware load balancer for use in their test or dev environments and will rely on a software load balancer.  Some of these software load balancers use IP stickiness to keep affinity between clients and servers.  This is bad when testing because it means that if you run a test script from a single machine it will only target a single machine in the cluster....  Make sure when testing with a software load balancer that it uses HTTP cookie affinity rather than IP address affinity.
• Poor testing with BPEL drivers
  Often we use BPEL processes as test harnesses to exercise functionality.  This is good but it doesn't work well in a cluster without some modifications to the driver process.  By default BPEL will optimize communications, within the same JVM it will use Java calls, between JVMs in the same cluster it will use ORMI, it only uses HTTP if it has to.  To test properly we need to distribute load through the load balancer and hence want to use HTTP.  In the invokes we can use the property optSoapShortcut=false to force calls to go through HTTP and hence the load balancer.

The above isn't an exhaustive list.  If you have others then feel free to share them, I would love to add more to the list.

In my last entry I spoke about some of the gotchas that are involved in setting up a cluster.  Over the next few entries I am going to describe how to build a SOA Suite 11g cluster for use in a test environment.  In this entry we will look at the target architecture and the required software.

Target Architecture

I am going to build my 11g cluster on 3 machines.

• Machine DB will host an 11gR1 database.  I will also use it to host a software load balancer (I will use WebCache).
• Machine SOA1 will host two WebLogic installations.  A WebLogic 11g installation will have a single SOA domain hosting a SOA Suite cluster, including BAM.  A WebLogic 10.3 installation will have a single OSB domain hosting an OSB cluster.
• Machine SOA2 will have the same software as SOA1 and will host the same two domains.

When OSB 11g is released then the need for two separate WebLogic installations will go away as the intention is for OSB and the rest of SOA Suite to run on the same WebLogic software version.

As there are two WebLogic domains then I will run an admin server on each machine, one domains admin server on SOA1 and the other domains admin server on SOA2.  This helps reduce the memory footprint.

Logically the architecture is shown below with a load balancer distributing load across the SOA and OSB clusters with a backend 11g database.

As for testing I don’t have a hardware load balancer then I run the load balancer on the same machine as the database to give the physical architecture shown below.

I will run this on 3 virtual machines on a server with 8gb memory, allowing 2gb for each virtual machine.

As a large number of customers seem to be running Linux these days I will use Oracle Enterprise Linux 5.3.  I will use 64-bit Linux for the DB machine and 32-bit Linux for the SOA machines.

It is quite common for clusters to be using a RAC database rather than a single instance database, but that was one VM too many for me to get my head around.

Software Required

So now we have identified the logical and physical architecture we need to identify what software we will require.  The software used is all available for download from OTN by clicking on the software link as shown in the table below.  Our target machine for the software is also shown in the table.

Load Balancing

There are a number of software load balancers available, including functionality built into Linux so why did I use WebCache.  Well there are a number of reasons.

1. I like WebCache
2. It has a nice web based UI for configuring and monitoring
3. It supports cookie based affinity (see previous post for importance of this)
4. It does the job

Just be careful when using WebCache with SOA Suite that you do not use it to cache data.  To my knowledge no testing has been done within Oracle with using WebCache in conjunction with SOA Suite 11g so don’t deploy it in a production environment.

I have to confess that the idea of using WebCache as a load balancer was not mine, but my colleague Nick Cosmidis, so thanks Nick.

Other Resources

Setting up a cluster requires shared storage, ideally for the domain home but also for shared resources such as JMS message stores.  I could have used an iSCSI appliance to provide this but I chose instead to use the DB machine as a shared file server for the mid-tier components.

The cluster also requires IP addresses.  Obvious but there are different requirements for those IP addresses.  The IP address for the load balancer must be routable from all the clients of the cluster.  The database, SOA Suite and OSB instances can have non-routable IP addresses as long as they can talk to each other and the the load balancer.  The clients don’t have to be able to access the database, SOA Suite or OSB directly because they will go through the load balancer.

Virtualization

I am running this on a virtualized environment, a single 8GB machine hosting all three machines.  The only software virtualization fully supported by Oracle is Oracle Virtual Machine.  That is not to say it won’t work on other software virtualization environments such as VMware, just that it is not fully supported on those environments.  For more information on Oracle’s support policy with respect to virtualization in general check out this link.  For specific information on VMware support then check Note 249212.1 in MetaLink.