Building a SOA/BPM/BAM Cluster Part I – Preparing the Environment
By Antony Reynolds on May 17, 2010
An increasing number of customers are using SOA Suite in a cluster configuration, I might hazard to say that the majority of production deployments are now using SOA clusters. So I thought it may be useful to detail the steps in building an 11g cluster and explain a little about why things are done the way they are.
In this series of posts I will explain how to build a SOA/BPM cluster using the Enterprise Deployment Guide.
This post will explain the setting required to prepare the cluster for installation and configuration.
The following software is required for an 220.127.116.11 SOA/BPM install.
|Oracle Database||Certified databases are listed here||SOA & BPM Suites require a working database installation.|
|18.104.22.168||If upgrading an 22.214.171.124 repository then a separate script is available.|
|126.96.36.199||Provides Web Server. 188.8.131.52 is an upgrade to 184.108.40.206, so 220.127.116.11 must be installed first.|
|18.104.22.168||Web Server, 22.214.171.124 Patch. You can use the 126.96.36.199 version without problems.|
|Oracle WebLogic Server 11gR1||10.3.3||This is the host platform for 188.8.131.52 SOA/BPM Suites.|
|SOA Suite||184.108.40.206||SOA Suite 220.127.116.11 is an upgrade to 18.104.22.168, so 22.214.171.124 must be installed first.|
|SOA Suite||126.96.36.199||SOA Suite 188.8.131.52 patch, requires 11.1.12 to have been installed.|
My installation was performed on Oracle Enterprise Linux 5.4 64-bit.
I will not cover setting up the database in this series other than to identify the database requirements. If setting up a SOA cluster then ideally we would also be using a RAC database. I assume that this is running on separate machines to the SOA cluster. Section 2.1, “Database”, of the EDG covers the database configuration in detail.
The database should have processes set to at least 400 if running SOA/BPM and BAM.
- alter system set processes=400 scope=spfile
The Repository Creation Utility creates the necessary database tables for the SOA Suite. The RCU can be run from any machine that can access the target database. In 11g the RCU creates a number of pre-defined users and schema with a user defiend prefix. This allows you to have multiple 11g installations in the same database.
After running the RCU you need to grant some additional privileges to the soainfra user. The soainfra user should have privileges on the transaction tables.
- grant select on sys.dba_pending_transactions to prefix_soainfra
- Grant force any transaction to prefix_soainfra
The cluster will be built on the following machines.
- EDG Name is the name used for this machine in the EDG.
- Notes are a description of the purpose of the machine.
|LB||External load balancer to distribute load across and failover between web servers.|
|WEBHOST1||Hosts a web server.|
|WEBHOST2||Hosts a web server.|
|SOAHOST1||Hosts SOA components.|
|SOAHOST2||Hosts SOA components.|
|BAMHOST1||Hosts BAM components.|
|BAMHOST2||Hosts BAM components.|
Note that it is possible to collapse the BAM servers so that they run on the same machines as the SOA servers. In this case BAMHOST1 and SOAHOST1 would be the same, as would BAMHOST2 and SOAHOST2. The cluster may include more than 2 servers and in this case we add SOAHOST3, SOAHOST4 etc as needed.
My cluster has WEBHOST1, SOAHOST1 and BAMHOST1 all running on a single machine.
The cluster will use the following software components.
- EDG Name is the name used for this machine in the EDG.
- Type is the type of component, generally a WebLogic component.
- Notes are a description of the purpose of the component.
|AdminServer||Admin Server||Domain Admin Server|
|WLS_WSM1||Managed Server||Web Services Manager Policy Manager Server|
|WLS_WSM2||Managed Server||Web Services Manager Policy Manager Server|
|WLS_SOA1||Managed Server||SOA/BPM Managed Server|
|WLS_SOA2||Managed Server||SOA/BPM Managed Server|
|WLS_BAM1||Managed Server||BAM Managed Server running Active Data Cache|
|WLS_BAM2||Managed Server||BAM Manager Server without Active Data Cache|
|Node Manager||Will run on all hosts with WLS servers|
|OHS1||Web Server||Oracle HTTP Server|
|OHS2||Web Server||Oracle HTTP Server|
|LB||Load Balancer||Load Balancer, not part of SOA Suite|
The above assumes a 2 node cluster.
The SOA cluster requires an extensive amount of network configuration. I would recommend assigning a private sub-net (internal IP addresses such as 10.x.x.x, 192.168.x.x or 172.168.x.x) to the cluster for use by addresses that only need to be accessible to the Load Balancer or other cluster members. Section 2.2, "Network", of the EDG covers the network configuration in detail.
- EDG Name is the hostname used in the EDG.
- IP Name is the IP address name used in the EDG.
- Type is the type of IP address:
- Fixed is fixed to a single machine.
- Floating is assigned to one of several machines to allow for server migration.
- Virtual is assigned to a load balancer and used to distribute load across several machines.
- Host is the host where this IP address is active. Note for floating IP addresses a range of hosts is given.
- Bound By identifies which software component will use this IP address.
- Scope shows where this IP address needs to be resolved.
- Cluster scope addresses only have to be resolvable by machines in the cluster, i.e. the machines listed in the previous section. These addresses are only used for inter-cluster communication or for access by the load balancer.
- Internal scope addresses
- Notes are comments on why that type of IP is used.
|EDG Name||IP Name||Type||Host||Bound By||Scope||Notes|
|ADMINVHN||VIP1||Floating||SOAHOST1-SOAHOSTn||AdminServer||Cluster||Admin server, must be able to migrate between SOA server machines.|
|SOAHOST1||IP1||Fixed||SOAHOST1||NodeManager, WLS_WSM1||Cluster||WSM Server 1 does not require server migration.|
|SOAHOST2||IP2||Fixed||SOAHOST1||NodeManager, WLS_WSM2||Cluster||WSM Server 2 does not require server migration|
|SOAHOST1VHN||VIP2||Floating||SOAHOST1-SOAHOSTn||WLS_SOA1||Cluster||SOA server 1, must be able to migrate between SOA server machines|
|SOAHOST2VHN||VIP3||Floating||SOAHOST1-SOAHOSTn||WLS_SOA2||Cluster||SOA server 2, must be able to migrate between SOA server machines|
|BAMHOST1VHN||VIP4||Floating||BAMHOST1-BAMHOSTn||WLS_BAM1||Cluster||BAM server 1, must be able to migrate between BAM server machines|
|BAMHOST2||IP3||Fixed||BAMHOST2||NodeManager, WLS_BAM2||Cluster||BAM server 2 does not require server migration|
|soa.mycompany.com||VIP5||LB||LB||Public||External access point to SOA cluster.|
|admin.mycompany.com||VIP6||Virtual||LB||LB||Internal||Internal access to WLS console and EM|
|soainternal.mycompany.com||VIP7||Virtual||LB||LB||Internal||Internal access point to SOA cluster|
Floating IP addresses are IP addresses that may be re-assigned between machines in the cluster. For example in the event of failure of SOAHOST1 then WLS_SOA1 will need to be migrated to another server. In this case VIP2 (SOAHOST1VHN) will need to be activated on the new target machine. Once set up the node manager will manage registration and removal of the floating IP addresses with the exception of the AdminServer floating IP address.
Note that if the BAMHOSTs and SOAHOSTs are the same machine then you can obviously share the hostname and fixed IP addresses, but you still need separate floating IP addresses for the different managed servers. The hostnames don’t have to be the ones given in the EDG, but they must be distinct in the same way as the ETC names are distinct. If the type is a fixed IP then if the addresses are the same you can use the same hostname, for example if you collapse the soahost1, bamhost1 and webhost1 onto a single machine then you could refer to them all as HOST1 and give them the same IP address, however SOAHOST1VHN can never be the same as BAMHOST1VHN because these are floating IP addresses.
Notes on DNS
IP addresses that are of scope “Cluster” just need to be in the hosts file (/etc/hosts on Linux, C:\Windows\System32\drivers\etc\hosts on Windows) of all the machines in the cluster and the load balancer. IP addresses that are of scope “Internal” need to be available on the internal DNS servers, whilst IP addresses of scope “Public” need to be available on external and internal DNS servers.
Shared File System
At a minimum the cluster needs shared storage for the domain configuration, XA transaction logs and JMS file stores. It is also possible to place the software itself on a shared server. I strongly recommend that all machines have the same file structure for their SOA installation otherwise you will experience pain! Section 2.3, "Shared Storage and Recommended Directory Structure", of the EDG covers the shared storage recommendations in detail.
The following shorthand is used for locations:
- ORACLE_BASE is the root of the file system used for software and configuration files.
- MW_HOME is the location used by the installed SOA/BPM Suite installation. This is also used by the web server installation. In my installation it is set to <ORACLE_BASE>/SOA11gPS2.
- ORACLE_HOME is the location of the Oracle SOA components or the Oracle Web components. This directory is installed under the the MW_HOME but the name is decided by the user at installation, default values are Oracle_SOA1 and Oracle_Web1. In my installation they are set to <MW_HOME>/Oracle_SOA and <MW_HOME>/Oracle _WEB.
- ORACLE_COMMON_HOME is the location of the common components and is located under the MW_HOME directory. This is always <MW_HOME>/oracle_common.
- ORACLE_INSTANCE is used by the Oracle HTTP Server and/or Oracle Web Cache. It is recommended to create it under <ORACLE_BASE>/admin. In my installation they are set to <ORACLE_BASE>/admin/Web1, <ORACLE_BASE>/admin/Web2 and <ORACLE_BASE>/admin/WC1.
- WL_HOME is the WebLogic server home and is always found at <MW_HOME>/wlserver_10.3.
Key file locations are shown below.
|<ORACLE_BASE>/admin/domain_name/aserver/domain_name||Shared location for domain. Used to allow admin server to manually fail over between machines. When creating domain_name provide the aserver directory as the location for the domain. |
In my install this is <ORACLE_BASE>/admin/aserver/soa_domain as I only have one domain on the box.
|<ORACLE_BASE>/admin/domain_name/aserver/applications||Shared location for deployed applications. Needs to be provided when creating the domain. |
In my install this is <ORACLE_BASE>/admin/aserver/applications as I only have one domain on the box.
|<ORACLE_BASE>/admin/domain_name/mserver/domain_name||Either unique location for each machine or can be shared between machines to simplify task of packing and unpacking domain. This acts as the managed server configuration location. Keeping it separate from Admin server helps to avoid problems with the managed servers messing up the Admin Server. |
In my install this is <ORACLE_BASE>/admin/mserver/soa_domain as I only have one domain on the box.
|<ORACLE_BASE>/admin/domain_name/mserver/applications||Either unique location for each machine or can be shared between machines. Holds deployed applications. |
In my install this is <ORACLE_BASE>/admin/mserver/applications as I only have one domain on the box.
|<ORACLE_BASE>/admin/domain_name/soa_cluster_name||Shared directory to hold the following |
dd – deployment descriptors
jms – shared JMS file stores
fadapter – shared file adapter co-ordination files
tlogs – shared transaction log files
In my install this is <ORACLE_BASE>/admin/soa_cluster.
|<ORACLE_BASE>/admin/instance_name||Local folder for web server (OHS) instance. |
In my install this is <ORACLE_BASE>/admin/web1 and <ORACLE_BASE>/admin/web2.
I also have <ORACLE_BASE>/admin/wc1 for the Web Cache I use as a load balancer.
|<ORACLE_BASE>/product/fmw||This can be a shared or local folder for the SOA/BPM Suite software. I used a shared location so I only ran the installer once. |
In my install this is <ORACLE_BASE>/SOA11gPS2
All the shared files need to be put onto a shared storage media. I am using NFS, but recommendation for production would be a SAN, with mirrored disks for resilience.
To reduce the hardware requirements it is possible to collapse the BAMHOST, SOAHOST and WEBHOST machines onto a single physical machine. This will require more memory but memory is a lot cheaper than additional machines. For environments that require higher security then stay with a separate WEBHOST tier as per the EDG. Similarly for high volume environments then keep a separate set of machines for BAM and/or Web tier as per the EDG.
Notes on Dev Environments
In a dev environment it is acceptable to use a a single node (non-RAC) database, but be aware that the config of the data sources is different (no need to use multi-data source in WLS). Typically in a dev environment we will collapse the BAMHOST, SOAHOST and WEBHOST onto a single machine and use a software load balancer. To test a cluster properly we will need at least 2 machines.
For my test environment I used Oracle Web Cache as a load balancer. I ran it on one of the SOA Suite machines and it load balanced across the Web Servers on both machines. This was easy for me to set up and I could administer it from a web based console.