Antony Reynolds' Blog

Installing an 11g SOA Cluster – Part III Configuration

Antony Reynolds
Senior Director Integration Strategy

image Configuring a SOA 11g PS2 SOA Cluster – Part III WSM Configuration

In this post I will go through the steps required to set up a SOA cluster.  In previous posts I covered how to install RAC database for use by SOA, how to prepare the environment for SOA and how to install the software required.  The diagram shows the configuration I am creating, note that the Enterprise Deployment Guide (EDG) expects the web servers (OHS Servers) to be on a separate machine to the WebLogic servers (Admin and WSM servers).

Configuration Approach

Because configuring the SOA cluster in a high available environment has a lot of activities we will follow the Enterprise Deployment Guide (EDG) practice of configuring the environment a piece at a time and then validating what we have done.  We will do the following:

  1. Create a WebLogic domain and configure it for Web Services Manager by creating a policy management cluster.
  2. Create Oracle HTTP Server instances and configure them to talk to the newly created SOA domain.
  3. Extend the WebLogic domain to support SOA & BPM functionality.
  4. Extend the WebLogic domain to support BAM functionality.
  5. Configure the node manager to support starting and failing over all managed servers.

Create the WebLogic Domain

Our first step is to create a new WebLogic domain and within it configure a Web Service Manager policy management cluster.  This will require us to create a domain with an Admin Server and managed servers for the OWSM policy management cluster.

Enabling Access to ifconfig and arping Commands

To start and stop the network adapter for the admin server we need root privileges.  We can provide these just for the ifconfig and arping commands by using sudo which allows to run certain commands with root privileges.  To set this up on each machine run the visudo command as root to edit the /etc/sudoers permission file and add the following:

## SOA Suite Settings
oracle ALL=NOPASSWD: /sbin/ifconfig, /sbin/arping

This grants the oracle user access to the ifconfig and arping commands without having to provide a password by using sudo. Obviously this needs to be done on all middleware servers.  (EDG)

Enable Admin IP Address

The Admin server needs to be able to be started on any of the SOA servers so we have assigned it an IP address that can float between machines.  Before we configure our cluster we need to assign the Admin server IP address to a machine in the cluster.  For my environment I issued the following commands as the root user:

sudo /sbin/ifconfig eth1:9 netmask
sudo /sbin/arping -q -U -c 3 -I eth1

The ifconfig command associates a sub-interface (eth1:9 in my case) with a particular IP address and associated netmask.  I used eth1:9 to avoid any possible future clashes with node manager which will be responsible for assigning floating IP addresses for managed servers that can migrate between machines.  The arping command sends an unsolicited update (-U) 3 time (-c 3) on interface eth1 (-I eth1) alerting nodes on the sub-net of the association between this IP address and the adapter, this is important if the IP address was previously assigned to another adapter, such as occurs during migration of the admin server from one machine to another.  It is a good idea to create this as a shell script on shared storage, perhaps in the AServer folder, because you will need it to start the admin server on different machines. (EDG)

Creating the Domain

Having allocated an IP address for the admin server we now need to configure the initial domain. (EDG)

Running Config Script and Selecting Templates edg

To create the domain we run the config script found in the oracle_common/common/bin directory.  This starts the Fusion Middleware Configuration Wizard which allows us to “Create a New WebLogic Domain”.  We select the “Oracle WSM Policy Manager” template which should also automatically select the Oracle JRF template.  The WSM Policy Manager template as you might expect configures the domain to support the policy manager.  This in turn relies upon the Java Required Files being configured in the domain.  The Java Required Files act as an abstraction layer isolating the higher level Oracle components from the underlying application server.  We also need to select the “Enterprise Manager” template to provide the admin support for SOA Suite. (EDG)

Specifying Domain Location & JVM

On the “Specify Domain Name and Location” screen make sure that we set the domain location to be /u01/app/oracle/admin/soa-domain/aserver and the domain name to soa_domain.  The domain name is of course arbitrary.  After providing a password for the weblogic user we then select the JDK and WebLogic domain startup mode.  Set the startup mode to be “Production Mode” and select the JRockit JDK.  If you followed my preparation instructions for a 64-bit JVM then this should be your only option.  Note that you can change the VM used later, but it is easier to set it here.

Configuring RAC Data Sources

On the “Configure JDBC Component Schema” screen select the OWSM MDS Schema and then select the “Configure selected component schemas as RAC multi data source schemas in the next panel”.  This allows us to set up OWSM to use a RAC database.  We can then enter the Service Name as soaedg.soa.oracle.com (this was configured in the previous blog entry), the username can be left as DEV_MDS if we used DEV as our prefix, provide the password of the database user and then enter the host details.  The host details should be the virtual listener addresses that we set up, see my earlier post on configuring a RAC database for SOA.  In my case host names are rac1-vip.soa.oracle.com and rac2-vip.soa.oracle.com with instance names of RAC1 and RAC2 and port number 1521.  Before advancing to the next screen make sure that all nodes of the RAC database are up.  The WebLogic server makes use of multiple data source pools to connect to a RAC database, each pool targeting a specific RAC instance.  It is important to verify that all nodes are correctly working so that in event of database node failure the remaining nodes can continue to service database requests.

Configuring the Admin Server

After the database configuration we are asked about the optional configuration we wish to perform, select “Administration Server”, “Managed Servers, Clusters and Machines” and “Deployment and Services”.  This will allow us to create tie the admin server to the correct address, create the managed servers for the OWSM policy management cluster and target the right services to the cluster.

On the “Configure the Administration Server” screen we want to change the listen address to be the hostname we created for the admin server, in my case admin.soa.oracle.com.  This makes the Admin Server listen only on this address.  This stops the admin server from binding to other managed server addresses and also makes sure that it is actually listening on the hostname that we have assigned for the Admin Server.  Unless there is likely to a port clash with another WebLogic server domain running on the same machine then leave the listening port to be 7001.

Creating the OWSM Managed Servers and Cluster

I added the following managed servers which I will use to run the OWSM policy manager.

Server Name Listen Address Listen Port
WSM1 wsm1.soa.oracle.com 7010
WSM2 wsm2.soa.oracle.com 7010

Because WSM1 and WSM2 are running on different fixed IP addresses they can listen on the same port number.  If like me you used the same IP address for multiple fixed IP managed servers (WSM2 and BAM2 for example) it is important to make sure that you assign different port number to the managed servers that run different roles.

Note that the EDG has the managed servers listen on the SOA hostname address while I have them listen on WSM hostname address.  In my configuration that is the same as the SOA hostname address but by having this additional hostname I can just change the IP address of a WSM hostname and retarget the server to another physical machine without having to make any other configuration changes, either in the WSM managed server configuration or in the front-end web server configuration.

After creating the servers I created a cluster called WSM_Cluster, leaving the cluster settings at their defaults, and then assigned the two WSM managed servers to the WSM cluster.

Creating “Physical” Machines and Targeting Servers

We are currently creating a cluster with two physical machines, so I need to create a WebLogic representation of these machines.  The machines correspond to a node manager, and later we will configure a node manager to run on each machine.  Because I was running on Linux I chose to create “Unix Machine”s.  I added my two machines calling them SOAHost1 and SOAHost2 and identifying their node manager listen addresses as soa-cluster1.soa.oracle.com and soa-cluster2.soa.oracle.com.  I left the other settings at their default values.  I also added an extra machine called AdminHost that listens on localhost.  This allows us to move the Admin Server between machines without having to retarget it.

Having created the machines I then assigned the servers to machines as follows

Server Role Target Machine
AdminServer WebLogic Admin Server AdminHost
WSM1 WSM Policy Manager Server SOAHost1
WSM2 WSM Policy Manager Server SOAHost2

Targeting Deployments, Services and Resource to WSM Cluster

Make sure that the following items are correctly targeted to the appropriate server/cluster.

Application Deployment Target
wsm-pm WSM_Cluster
Library Deployment Target
oracle.wsm.seedpolicies#11.1.1@11.1.1 WSM_Cluster
Shutdown Class Target
JOC-Shutdown WSM_Cluster
Startup Class Target
JOC-Startup WSM_Cluster
OWSM Startup Class WSM_Cluster
JDBC Target
All JDBC System Resource WSM_Cluster, AdminServer

All other items are targeted only at the Admin Server.

Create the Domain

We can now finally hit the create button to create the domain.  Note that this will create the soa_domain on shared storage with domain configuration in /u01/app/oracle/admin/soa_domain/aserver/soa_domain and applications stored in /u01/app/oracle/admin/soa_domain/aserver/applications/soa_domain

Starting & Configuring the Domain

Configure Admin Server Security

Before starting the admin server we need to configure the security properties file for the Admin server by creating /u01/app/oracle/admin/aserver/domains/soa_domain/servers/AdminServer/security/boot.properties with the following contents


This prevents us being prompted for a username and password when booting the admin server. (EDG)

Configure and Start Node Manager

Set Node Manager to Use Scripts

Before using the Node Manager we must set it to use the start scripts by running setNMProps.sh from the Oracle Common home /u01/app/oracle/product/fmw/oracle_common/common/bin/setNMProps.sh.  This forces the node manager to use start scripts rather than directly launching the JVM by setting the “StartScriptEnabled” property equal to “true” in nodemanager.properties found in the WebLogic home common/bin directory /u01/app/oracle/product/fmw/wlserver_10.3/common/nodemanager.  This is important because the start scripts set up the (extensive) environment required by the SOA Suite.  Failure to run this may lead to ClassNotFoundErrors when starting WebLogic.  Beware that the setNMProps.sh script is very naive in the way it runs.  It checks for the existence of the string StartScriptEnabled=true in the nodemanager.properties file.  If you comment out this line it will still think that it is set and do nothing.

Important:  This command must be run in every middleware home, for example if you are using shared storage for FMW binaries and have two copies being shared you need to run this command in both copies.  If you have multiple servers pointing to the same FMW binaries you only need to run this script once for each set of binaries.  (EDG)

Create Node Manager Directory

If you are using a shared directory for FMW software then you should create a directory outside of that to hold the node manager configuration and log files.  This needs to be local to each machine so a good location would be $ORACLE_BASE/admin/NodeManager.  Having created the directory we need to copy the contents of the $WLS_HOME/common/nodemanager to that directory.

cp /u01/app/oracle/product/fmw/wlserver_10.3/common/nodemanager/* /u01/app/oracle/admin/NodeManager

Having created a new directory we need to tell the node manager to use it by editing the $WLS_HOME/server/bin/startNodeManager.sh script and editing the line that sets the node manager home to point to the new home as shown below:


Start the Node Manager

Start the node manager by running the following command from the WebLogic server/bin directory

/u01/app/oracle/product/fmw/wlserver_10.3/server/bin/startNodeManager.sh `hostname`

This will restrict the node manager to listening on a single IP address rather than multiple IP addresses.  I suggest you put this into a script to make life easier. (EDG)

Start the Admin Server via Script

We can then start the Admin Server by running startWebLogic.sh.  This may take a while to come to the running state.

With Admin Server running we can go to the console http://admin.soa.oracle.com:7001/console and set the node manager username and password by navigating to the security tab under the domain (soa_domain) and selecting the General tab.  Under the advanced settings set the node manager username and node manager password to any values you want, I used username nmadmin.  Once this is set then we can start the Admin Server using node manager.

Shutdown the Admin Server either by killing the JVM directly or using the console to shutdown the server. (EDG)

Start the Admin Server via Node Manager

Using WLST we can start the Admin Server with the following script

nmConnect(username, password, ‘soa-cluster1’, ‘5556’, ‘soa_domain’, ‘/u01/app/oracle/admin/soa_domain/’)

We run this script from the WebLogic Scripting Tool wlst.sh in Oracle Common Home /u01/app/oracle/product/fmw/oracle_common/common/bin. (EDG)

With the admin server up and running we can check that it is working and the configuration is correct by going to the weblogic console at http://admin:7001/console and the EM console at http://admin:7001/em. (EDG)

Creating Separate Domain Directories for Managed Servers

We want to create a separate domain directory for the managed servers to isolate them from the Admin Server.  These directories will be local to the node on which a managed server is running.  This approach avoids interference between nodes.

./pack.sh -managed=true

This command packages up our domain as a template for use in a managed server domain and stores it in the soadomaintemplate.jar file.  This file can be transported to other nodes to provide access to the managed server domain on those nodes.  This process is simplified by putting the generated template jar file on a shared location as I have done above.

./unpack.sh -domain=/u01/app/oracle/admin/soa-domain/mserver/soa_domain

This command unpacks the SOA domain for use by the managed servers and registers the local domain with the node manger by updating the $WLS_HOME/common/nodemanager/nodemanager.domains file.Note that we keep a separate managed server domain for each node, so we must run unpack on each node to create the local files even though the nodemanager.domains file already knows about the domain if we are using shared storage for the middleware binaries.  Note that if we have moved the node manager directory then we will need to copy the nodemanager.domains file to the new node manager directory.  (EDG)

Apply JRF to WSM Cluster

We need to apply the Java Required Files (the abstraction layer between SOA Suite and underlying app server) to the WSM cluster.  This is done from the Fusion Middleware Control page, selecting the cluster under the WebLogic domain tree and clicking the Apply JRF Template button. (EDG)

Disable Hostname Verification on Servers

If you are not going to set up certificates to enable certificate based hostname verification of your servers, to validate that the servers are connecting to who they think they are connecting to, then you need to disable hostname verification.  This is done in the WebLogic console for each of the following servers:

  • AdminServer
  • WSM1
  • WSM2

To disable hostname verification then go to the settings for the server, choose the SSL tab, and set Hostname Verification to “None”  in the Advanced section and save the settings for each server before applying the changes through the WebLogic change console.  After this you need to restart the Admin server. (EDG)

Starting the WSM Cluster

At this point you should have the node manager running on the first server and both servers configured with managed server domains.  We can now start the node manager on the second server (using the script you created).  Once the node manager is running on all servers then we can start WSM1 and WSM2 managed servers from the WebLogic console. Validate that they are successfully up and running and that the policies are deployed by going to http://wsm1:7010/wsm-pm and http://wsm2:7010/wsm-pm.  If you get a page not found error (404) then there may be a problem with your targeting of applications to the WSM cluster. (EDG)

Configuring Java Object Cache for WSM Cluster

Web Services Manager uses the Oracle Java Object Cache to provide distributed caching to improve performance.  Hopefully this will be replaced by Coherence in a future release.  In the meantime we need to configure it using WebLogic Scripting Tool (WLST) in $MW_HOME/Oracle_SOA/common/bin.  Issue the connect() command and provide the Admin server username (weblogic) and password as requested.  When prompted for the server URL enter t3://admin:7001 rather than using localhost as we told the Admin server to listen on hostname admin, not localhost.  Then execute the script to configure the cache, having checked that both WSM servers are up and running.


Accept the script defaults, when prompted for the CLuster Name enter the name of the cluster you created – I used WSM_Cluster.  When prompted for the Discover Port enter 9991 or any other unused port outside the reserved address range.  You should see the output describing the configuration of the two WSM servers. (EDG)

Creating & Configuring Oracle HTTP Servers

Having created an OWSM cluster we can now create the front end web server instances.

Creating the OHS Server Instance

We begin by running the configuration wizard for the web tier on each machine Web_Tier_ORACLE_HOME/bin/config.sh.

Configure Components

We are not using Oracle Web Cache so we can remove that from the list of components to configure.  We want to configure Oracle HTTP Server and Associate Selected Components with WebLogic Domain.  We could use Web Cache as a load balancer but I opted for a standalone load balancer.

Specify WebLogic Domain

We now need to specify the details of our WebLogic domain.

  • Domain Host Name : admin
  • Domain Port No : 7001
  • User Name : weblogic
  • Password : <weblogic_password>

The domain host name is the Admin Server hostname.

Specify Component Details

We now specify the location of the Oracle HTTP Server instance files.  We store this in the Admin directory.

  • Instance Home Location : $ORACLE_BASE/admin/OHSN
  • Instance Name : ohs_instanceN
  • OHS Component Name : ohsN

If we have two web servers then we will have component names oh1 and ohs2.

Configure Ports

Either provide a staticports.ini file for complete control over the ports or use the auto port configuration.

Configure New Instance

After specifying how or if we wish to be alerted to security updates we can review our configuration before clicking on Configure to create our new OHS instance.

Difference from EDG

Note that the EDG suggests creating the Web instance as part of the Web tier install and then registering WebLogic in a separate step.  I have done the install in one step and the instance creation and WebLogic registration in a second step.  This approach works better than the EDG suggestion if you are using a shared software directory and have more Web instances than software locations. (EDG)

Configure OHS for SOA

Now that we have our OHS instances we need to configure them to forward requests to the SOA Suite.

Configure Virtual Hosts, Admin & EM Consoles & WSM Cluster

First we set up the OHS to work with virtual hosts, the WebLogic and EM consoles and the WSM cluster.  We define three virtual hosts, one each for admin, internal and external access.  This allows us to configure different rules for these different hostnames.  We configure this by adding the following to the end of the httpd.conf file of each OHS instance ($ORACLE_BASE/admin/OHSn/config/OHS/ohsn/httpd.conf).

# SOA Suite #
<IfModule mod_weblogic.c>
  MatchExpression /wsm-pm WebLogicCluster=wsm1:7010,wsm2:7010

# SOA Suite Virtual Hosts #
NameVirtualHost *:7777
<VirtualHost *:7777>
    ServerName soa-cluster.soa.oracle.com
    ServerAlias soa-cluster
    ServerAdmin you@your.address
    RewriteEngine On
    RewriteOptions inherit

<VirtualHost *:7777>
    ServerName soa-cluster-admin.soa.oracle.com
    ServerAlias soa-cluster-admin
    ServerAdmin you@your.address
    RewriteEngine On
    RewriteOptions inherit
  # Admin Server and EM only through this virtual host
  <Location /console>
    SetHandler weblogic-handler
    WebLogicHost admin
    WeblogicPort 7001
  <Location /consolehelp>
    SetHandler weblogic-handler
     WebLogicHost admin
    WeblogicPort 7001
  <Location /em>
    SetHandler weblogic-handler
    WebLogicHost admin
    WeblogicPort 7001

<VirtualHost *:7777>
    ServerName soa-cluster-internal.soa.oracle.com
    ServerAlias soa-cluster-internal
    ServerAdmin you@your.address
    RewriteEngine On
    RewriteOptions inherit

# SOA Suite WSM Cluster #
<Location /wsm-pm>
    SetHandler weblogic-handler
    WebLogicCluster wsm1:7010,wsm2:7010

Note that we have departed from the EDG by putting our Location tags in the httpd.conf file so that we can restrict access to the consoles.  In particular the consoles can only be accessed through the soa-cluster-admin virtual host.  The ServerAlias’s allow us to use either canonical hostnames or short hostnames to access our virtual hosts.

After editing this file we then need to restart the OHS on each machine:

cd $ORACLE_BASE/admin/OHSn/bin

./opmnctl restartproc process-type=OHS

Finally we can check that we can access all three virtual hosts through the load balancer.

  • https://soa-cluster.soa.oracle.com/wsm-pm
  • http://soa-cluster-internal.soa.oracle.com
  • http://soa-cluster-admin.soa.oracle.com

Then check all three virtual hosts can access the WSM PM cluster.

  • https://soa-cluster.soa.oracle.com/wsm-pm
  • http://soa-cluster-internal.soa.oracle.com/wsm-pm
  • http://soa-cluster-admin.soa.oracle.com/wsm-pm

Finally we can check that the EM and WebLogic consoles are only accessible through the admin virtual host.

  • https://soa-cluster.soa.oracle.com/console (should fail)
  • http://soa-cluster-internal.soa.oracle.com/console (should fail)
  • http://soa-cluster-admin.soa.oracle.com/console
  • https://soa-cluster.soa.oracle.com/em (should fail)
  • http://soa-cluster-internal.soa.oracle.com/em (should fail)
  • http://soa-cluster-admin.soa.oracle.com/em

Note that if you are using a similar configuration to mine in the load balancer then at this point you will find that attempts to access http://soa-cluster.soa.oracle.com will automatically redirect you to the secure version of the web site. (EDG and EDG)

Configure Admin Server HTTP Frontend

Sometimes the WebLogic console needs to set up a canonical URL to refer to a page.  In this case it will construct the URL using the Frontend Host property in the Protocols HTTP section of the AdminServer settings, so we need to set this to point to the load balancer URL (soa-cluster-admin.soa.oracle.com) by using the console.  After activating the change then the console and enterprise manager can only be reliably accessed through the load balancer. (EDG)

Configuring OS for Admin Server Failover

We now have an OWSM cluster set up and running.  We would like to be able to test failover of the Admin Server.  As this requires shutting down the admin network interface on one server and starting it on another I have created some scripts to do this and configured the operating system to  allow this to be done by the oracle user without having to change to the root user.

Releasing the Admin IP Address on Server 1

After stopping the Admin Server the IP address can be released by issuing the command

sudo /sbin/ifconfig eth1:9 down

Failure to release the IP address may lead to unusual errors or inability to assign the IP address to the second server. (EDG)

Bringing Up Admin Server on Server 2

We can now assign the Admin Server IP address on server 2 and start the Admin Server using our startup script we created earlier.  Note that you need to make sure you execute the arping command to flush the arp caches for things to work properly.

We can now test that the admin server is still accessible. (EDG)

Failing Back Admin Server to Server 1

We can now fail back the Admin Server to server 1 using the same procedure that we followed to move it to server 2.

  1. Release IP on original server
  2. Assign IP on new new server
  3. Flush arp caches

The admin server is then ready to run on server 1 again. (EDG)

WSM Cluster Now Complete

We now have a working WSM cluster and are ready for the next stage, configuring a SOA/BPM cluster which I will cover in the next post.

Join the discussion

Comments ( 3 )
  • Bob Procter Tuesday, June 21, 2011

    Where Weblogic uses floating IP addresses for each server instance, is it the floating IP or the physical IP of the machine that is used as the source address for outbound calls from the server?

  • guest Saturday, June 25, 2011

    I haven't verified this but I believe it will use the machine IP address, but it depends on the network configuration, for example I usually configure WLS on two networks and it will choose the IP address that has appropriate connectivity.

    Also for security reasons it is a good idea to route outbound traffic through a proxy server for example and allow inbound traffic to come direct via the firewalls, so again the answer to which adapter will be used is "it depends".

    Sorry not to be more definitive.


  • guest Friday, May 25, 2012

    Thanks Antony, we have been trying to install SOA for weeks now due to Web Tier failed to start opmnctl start all.

    We have installed the same softwares on the same OS without any issues,

    below are the steps we followed on 64 bits Red Hat 5.6:

    1) Installed required version of JRockit,

    2) Installed Web Logic 10.3.4

    3) install SOA

    4) Create a new Weblogic domain (run config.sh)

    5a) Installed Web Tier and apply and then run config.sh failed (no any error generated)

    since 5a failed, we also tried

    5b ) Install and Config Web Tier as the EDG guilde on but still failed on the same spot

    we have open 1 sev1 SR for many days but make no progress, no downgrade to sev2 working with the same support, but still make no progress.

    Can you please provide suggestions what we should do for the trouble shooting? we have run out of idea now,

    Any suggestions will be very appreciated.



Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.