High Availability Application Server

It seems that I get asked a lot about how to set up Oracle Application Server in a High Availability (HA) environment.  A couple of weeks ago I was asked to provide someone to assist one of our partners in setting this up in the Enterprise Technology Centre (ETC) in Thames Valley Park.  No-one else being available I assigned myself.

In the Beginning

The ETC provided us with some hardware.  This is outlined in the image below.

We had two Application Server J2EE mid-tier installations on seperate machines (the AS nodes), a single infrastructure node and two Real Application Cluster (RAC) database nodes.  We also had an F5 Networks BigIP load balancer.

Initial Deployments

Our first job was to deploy the application to one of the application servers and configure it so that it worked.  We did this before clustering to make sure that clustering didn't introduce any unusual behavior.  Part of the application configuration was to set up single sign on with a custom single sign on module on the mid-tier.  Once we had set up the app and got it working with the RAC database and infrastructure and single sign we were ready to set up a cluster.

Clustering

Clustering was very simple.  We used ASControl to create a cluster and added the first mid tier (the one we had deployed the application to) into the cluster.  This defined the configuration of the cluster.  We then tested the application to make sure it still worked before adding the second node to the cluster.  Adding the second node to the cluster automatically caused the application to be deployed to the second node.

The application now ran fine on both nodes but there were still a couple of things to sort out.

Firstly we had to put in the address of the specific app server we wanted to go to.  Secondly single sign on only allows a single URL to be registered for a custom login module, so all authentication was done on one of the app servers no matter which one we actually accessed.  To resolve these issues we needed a virtual hostname that would be load balanced across both servers.

We also configured the cluster to replicate session state but couldn't verify it was working until we set up the virtual IP address.

Setting up the Load Balancer

Fortunatley we had the big IP box.  We configured a server group consisting of the two app servers in the cluster and bound them to the same virtual IP address.  We then altered the configuration of the custom login module in SSO so that it also pointed to the virtual IP address.  We were then able to put in a single address and be load balanced across both app servers for application access and single sign on.

Note that we set up sticky load balancing to ensure that we always hit the same HTTP server from the same client.

Testing

We then tested the application by running a load runner script against it.  The load runner script authenticated once per simulated user and then repeatedly iterated through a transaction set.  Whilst the script was running we checked that the load was being balanced across both HTTP servers (OHS), both OC4J containers and both database nodes.

We then took down one of the HTTP servers (on app server 1).  All web traffic was then sent by the load balancer to app server 2.  We then took down the OC4J container on app server 2.   Now all users were using the OHS on app server 2 but the OC4J on app server 1.

Apart from a slow recognition by the load balancer that the OHS was down there were no problems and the script continued to execute without errors, the sessions failing over transparently between boxes.

The Database

Unfortunately time prevented us exploring the different options for the database connectivity.  We were going to try at Failure Notification (FaN) and got as far as configuring the notification service on the RAC instances before we ran out of time.

A Job Well Done

All this was done on App Server 10.1.2.0.2.  If we had used 10.1.3 we would have had more options for the session replication, but that appeared to work fine.  At the end of the day we were pleased with how straight forward it had been to configure it all in a HA environment.

The partner was pleased at being able to demonstrate both scalability and high availability with the solution and the system is now being commisioned in the live environment for the customer.