Tuesday Apr 29, 2014

Enabling WAN Replication for Oracle Service Bus Result Cache

The result cache is a very cool functionality introduced in Oracle Service Bus to allow ESB developers to automatically cache responses from a external service in OSB's built-in in-memory data grid caching system, which is Oracle Coherence. No matter which external service you are planning dealing with, an web service, an REST API, an directory in the file system or an CICS transaction, if the result cache functionality is activated for that external service, the response payload of an specific request message will be putted in the caching system for future reuse if the same request message is received again. The result cache functionality also allows you to define a expiration criteria, so the response payloads entries can eventually expire.


ESB developers will activate this functionality in OSB neither to protect critical back-end external services, to offload it or to short its response time. In the scenario that wants to protect back-end external services, perhaps those services have some cost associate every time you send a message for them. This cost would have various meanings, like per-request-basis (a paid external service that allows customer's credit history querying), IT budget (an CICS transaction service in which each call consumes MIPS) or even performance costs. In the case of performance costs is that when we start thinking in offloading. When services are originally designed, we measure some approximate throughput and average latency, and we put enough hardware resources to sustain that measure. When a ESB is situated in front of those services, you are enabling more channels to interact with that service and maybe the new amount of channels can be too high for the existing hardware resources. Finally, you could enable this functionality to short the response time of some services. If some services are sensitive in terms of response time latency, so the result cache is a must have.

A common practice used by customers around the world is to have replicas of their system architecture in different data centers, allowing them to survive in case of catastrophes. But only having a replica of their system architecture in different data centers is not enough. There is a need to provide business continuity, which means that every single detail of the system architecture should be constantly synchronized between the data centers, so when a backup data center take place in a catastrophe scenario, the down time should be minimal. There is also scenarios when even small periods of down time are not acceptable. All the data centers should be in stand-by/active mode to take over the entire processing in any moment. The challenge here is to keep two types of things synchronized: system architecture artifacts and system transactions. System architecture artifacts are any piece of data that the run-time system architecture needs to properly work. Common examples of artifacts are XML configuration files, applications, log files, data files and storage. System transactions are a unit-of-work of a business transaction. A business transaction represents a single or multiple business processes of the organization, and most of the times a business transaction are associated to a monetary need. E-commerce sites for instance are good examples of business transactions that are associated to a monetary need. If the site loses a single transaction, that lost represents less incoming money. And that is a situation that no CFO/CEO likes to tolerate.

Back to the result cache functionality, imagine that you have OSB deployed in two or more data centers operating in active-active mode. A corporate load balancer distributes load across each data center though its exposed services. When a request arrives in one data center, OSB take that request and start processing it, causing one or more entries to be stored in the result cache for future reuse. If the same request arrives in another data center, the desire is that OSB pick the already processed result from the result cache instead of processing it again. This is true because from the customer/client point of view, it is the same service and invocation request. But what will really happen is that the request will be processed again since result cache by default do not replicate entries across data centers, only across clusters in the same local network. So the challenge here is to find a way to enable entries being replicated from one local network (a.k.a "LAN") to a remote network (a.k.a, "WAN") even if this remote network is geographically distant.

In this article, I will show step by step how to enable result cache data replication across different data centers connected through a WAN. Thanks to OSB's great product architecture, this configuration is very straightforward and you will not have to change nothing in your SOA services, neither even in the OSB deployment. Everything is done out-of-the-box by Oracle Coherence. This article will help you even if WAN replication is not your primary objective. If you have different OSB domains (in the same or different networks) in which some services are exactly the same in those domains, the same technique should apply. All the examples created in this article were based on Oracle Service Bus 11gR1 default installation, which comprises WebLogic 10.3.6, Coherence 3.7.1.1 and Service Bus 11.1.1.7.

Patching Oracle Coherence from Middleware's Home

Before starting using the Push Replication Pattern feature available in Coherence Incubator (it will be explained in the next topic) we need to patch the Coherence installation that come with WebLogic. When you install the WebLogic pre-requisite for OSB which is the WebLogic 11gR1 + Coherence package installer, the Coherence 3.7.1.1 version is installed in the middleware home location. We need to patch this Coherence installation so we can take advantage of the latest features of the Push Replication Pattern.

Update Coherence to the 3.7.1.11 version. You can get access to this version in the Oracle Support website. After logged in the Oracle Support Self-Service portal, go to the "Patches and Updates" tab and search for the following patch number: 17897749. Download this patch and update the Coherence installation according to the instructions available inside of the patch file.

Installing the Oracle Coherence Push Replication Pattern

The Push Replication Pattern is a extension for the Oracle Coherence product to allow remote clusters to exchange data across WAN networks. It is part of the Coherence Incubator project, an very cool initiative to enhance the Coherence product through community based feedback. It hosts a collection of projects with implementations of real world needs, in a form of design patterns. Even being open in terms of source code access, it is responsibility of Oracle engineers to provide new features, correction of bugs and documentation.

You need to download a compatible version of Coherence Incubator to the Coherence 3.7.1.11 release. Use the following link to get instructions about how to download the source code. After downloading the source code, you need to compile and build the run-time packages. To accomplish that, you will need the Apache Maven project management tool. With Apache Maven properly installed, follow the instructions of this link to compile and build the Coherence Incubator run-time packages.

Setting Up a Coherence Cluster with WAN Replication Support

Let's set up a Coherence cluster that allows data replication across a WAN network. The first thing to do is the definition of cache configuration files for both sites. The idea for those cache configuration files is that it should contains definitions for publishing and receiving endpoints. That means that one site should expose one or more endpoints to receive events from the other site and also define a remove invocation service to connect to the other site to publish events. It is a bi-directional communication across the sites in which the Push Replication Pattern takes care about when to publish/receive events using the endpoints. The listing code below shows the cache configuration file for site-01:

<?xml version="1.0" encoding="UTF-8"?>

<cache-config
    xmlns:element="class://com.oracle.coherence.environment.extensible.namespaces.XmlElementProcessingNamespaceContentHandler" 
    xmlns:event="class://com.oracle.coherence.patterns.eventdistribution.configuration.EventDistributionNamespaceContentHandler"
    xmlns:cr="class:com.oracle.coherence.environment.extensible.namespaces.InstanceNamespaceContentHandler">

    <caching-scheme-mapping>

        <cache-mapping>
            <cache-name>/osb/service/ResultCache</cache-name>
            <scheme-name>distributed-scheme-with-wan-replication</scheme-name>
            <event:distributor>
                <event:distributor-name>{cache-name}</event:distributor-name>
                <event:distributor-external-name>{site-name}-{cluster-name}-{cache-name}</event:distributor-external-name>
                <event:distributor-scheme>
                    <event:coherence-based-distributor-scheme/>
                </event:distributor-scheme>
                <event:distribution-channels>
                    <event:distribution-channel>
                        <event:channel-name>site-01-channel</event:channel-name>
                        <event:starting-mode system-property="channel.starting.mode">enabled</event:starting-mode>
                        <event:channel-scheme>
                            <event:remote-cluster-channel-scheme>
                                <event:remote-invocation-service-name>site-02-sync-proxy-service</event:remote-invocation-service-name>
                                <event:remote-channel-scheme>
                                    <event:local-cache-channel-scheme>
                                        <event:target-cache-name>/osb/service/ResultCache</event:target-cache-name>
                                        <event:conflict-resolver-scheme>
                                            <cr:class classname="com.oracle.coherence.patterns.eventdistribution.channels.cache.BruteForceConflictResolver" />
                                        </event:conflict-resolver-scheme>
                                    </event:local-cache-channel-scheme>
                                </event:remote-channel-scheme>
                            </event:remote-cluster-channel-scheme>
                        </event:channel-scheme>
                    </event:distribution-channel>
                </event:distribution-channels>
            </event:distributor>
        </cache-mapping>

    </caching-scheme-mapping>

    <caching-schemes>

        <distributed-scheme>
            <scheme-name>distributed-scheme-with-wan-replication</scheme-name>
            <service-name>wan-sync-enabled-distributed-cache-svc</service-name>
            <backing-map-scheme>
                <read-write-backing-map-scheme>
                    <internal-cache-scheme>
                        <local-scheme />
                    </internal-cache-scheme>
                    <cachestore-scheme>
                        <class-scheme>
                            <class-name>com.oracle.coherence.patterns.pushreplication.PublishingCacheStore</class-name>
                            <init-params>
                                <init-param>
                                    <param-type>java.lang.String</param-type>
                                    <param-value>{cache-name}</param-value>
                                </init-param>
                            </init-params>
                        </class-scheme>
                    </cachestore-scheme>
                </read-write-backing-map-scheme>
            </backing-map-scheme>
            <autostart>true</autostart>
        </distributed-scheme>

        <remote-invocation-scheme>
            <service-name>site-02-sync-proxy-service</service-name>
            <initiator-config>
                <tcp-initiator>
                    <remote-addresses>
                        <socket-address>
                            <address>soa.suite.machine</address>
                            <port>30002</port>
                        </socket-address>
                    </remote-addresses>
                    <connect-timeout>2s</connect-timeout>
                </tcp-initiator>
                <outgoing-message-handler>
                    <request-timeout>5s</request-timeout>
                </outgoing-message-handler>
            </initiator-config>
        </remote-invocation-scheme>
        
        <proxy-scheme>
            <service-name>site-01-trans-proxy-service</service-name>
            <acceptor-config>
                <tcp-acceptor>
                    <local-address>
                        <address>soa.suite.machine</address>
                        <port>20001</port>
                    </local-address>
                </tcp-acceptor>
            </acceptor-config>
            <autostart>true</autostart>
        </proxy-scheme>

        <proxy-scheme>
            <service-name>site-01-sync-proxy-service</service-name>
            <acceptor-config>
                <tcp-acceptor>
                    <local-address>
                        <address>soa.suite.machine</address>
                        <port>30001</port>
                    </local-address>
                </tcp-acceptor>
            </acceptor-config>
            <autostart>true</autostart>
        </proxy-scheme>

    </caching-schemes>

</cache-config>

Save this cache configuration file as coherence-cache-config-site-01.xml. Before we continue, let's spend some time understanding the code. If you look at the top of the configuration file you will see the mapping for the cache /osb/services/ResultCache. This cache name matches with the one the come bundled with OSB. Also in the cache mapping, you will see a section that starts with the tag event:distributor. This XML tag is part of the Coherence Incubator implementation as you probably have seen in the namespaces declaration section. The event:distributor section basically states for declaring which remote sites should receive events from created, modified, removed or expired entries of the local cache. In the declaration, it is defined that the site-02 will be updated through a remote invocation service declared as site-02-sync-proxy-service later in the configuration file.

Special attemption for the event:conflict-resolver-scheme section. This should be used when you are expecting that entries from one site conflicts with entries of another site, most of the time because synchronization failures due unstable network links. Using this section, you can plug custom implementations that would decide which entry should be considered. The BruteForceConflictResolver class used in this example is a out-of-the-box implementation that came with the Event Distribution Pattern, another pattern that is part of the Coherence Incubator project.

Finally, you also have two proxy-scheme declarations in the configuration file. The purpose of the site-01-trans-proxy-service is for receiving local events from the same site. As for the site-01-sync-proxy-service, it is used to receive remote events from the foreign sites. Using two different proxies, one for transaction and another for synchronization gives you the ability to fine tune each proxy throughput independently, configuring for instance a different pool of threads for each one. In theory, you should balance the same number of threads for both proxies to ensure a well synchronized cluster. The Push Replication Pattern executes its synchronization job between sites completely asynchronous, meaning that the thread that updates the local cache does not have to wait the thread the replicates the entry for a remote site. That is the reason why is so important have different proxies.

Now let's create the cache configuration file for the site-02. The listing code below is almost identical to the previous listing, except from the fact that this time we are defining how site-02 will synchronize with site-01:

<?xml version="1.0" encoding="UTF-8"?>

<cache-config
    xmlns:element="class://com.oracle.coherence.environment.extensible.namespaces.XmlElementProcessingNamespaceContentHandler" 
    xmlns:event="class://com.oracle.coherence.patterns.eventdistribution.configuration.EventDistributionNamespaceContentHandler"
    xmlns:cr="class:com.oracle.coherence.environment.extensible.namespaces.InstanceNamespaceContentHandler">

    <caching-scheme-mapping>

        <cache-mapping>
            <cache-name>/osb/service/ResultCache</cache-name>
            <scheme-name>distributed-scheme-with-wan-replication</scheme-name>
            <event:distributor>
                <event:distributor-name>{cache-name}</event:distributor-name>
                <event:distributor-external-name>{site-name}-{cluster-name}-{cache-name}</event:distributor-external-name>
                <event:distributor-scheme>
                    <event:coherence-based-distributor-scheme/>
                </event:distributor-scheme>
                <event:distribution-channels>
                    <event:distribution-channel>
                        <event:channel-name>site-02-channel</event:channel-name>
                        <event:starting-mode system-property="channel.starting.mode">enabled</event:starting-mode>
                        <event:channel-scheme>
                            <event:remote-cluster-channel-scheme>
                                <event:remote-invocation-service-name>site-01-sync-proxy-service</event:remote-invocation-service-name>
                                <event:remote-channel-scheme>
                                    <event:local-cache-channel-scheme>
                                        <event:target-cache-name>/osb/service/ResultCache</event:target-cache-name>
                                        <event:conflict-resolver-scheme>
                                            <cr:class classname="com.oracle.coherence.patterns.eventdistribution.channels.cache.BruteForceConflictResolver" />
                                        </event:conflict-resolver-scheme>
                                    </event:local-cache-channel-scheme>
                                </event:remote-channel-scheme>
                            </event:remote-cluster-channel-scheme>
                        </event:channel-scheme>
                    </event:distribution-channel>
                </event:distribution-channels>
            </event:distributor>
        </cache-mapping>

    </caching-scheme-mapping>

    <caching-schemes>

        <distributed-scheme>
            <scheme-name>distributed-scheme-with-wan-replication</scheme-name>
            <service-name>wan-sync-enabled-distributed-cache-svc</service-name>
            <backing-map-scheme>
                <read-write-backing-map-scheme>
                    <internal-cache-scheme>
                        <local-scheme />
                    </internal-cache-scheme>
                    <cachestore-scheme>
                        <class-scheme>
                            <class-name>com.oracle.coherence.patterns.pushreplication.PublishingCacheStore</class-name>
                            <init-params>
                                <init-param>
                                    <param-type>java.lang.String</param-type>
                                    <param-value>{cache-name}</param-value>
                                </init-param>
                            </init-params>
                        </class-scheme>
                    </cachestore-scheme>
                </read-write-backing-map-scheme>
            </backing-map-scheme>
            <autostart>true</autostart>
        </distributed-scheme>

        <remote-invocation-scheme>
            <service-name>site-01-sync-proxy-service</service-name>
            <initiator-config>
                <tcp-initiator>
                    <remote-addresses>
                        <socket-address>
                            <address>soa.suite.machine</address>
                            <port>30001</port>
                        </socket-address>
                    </remote-addresses>
                    <connect-timeout>2s</connect-timeout>
                </tcp-initiator>
                <outgoing-message-handler>
                    <request-timeout>5s</request-timeout>
                </outgoing-message-handler>
            </initiator-config>
        </remote-invocation-scheme>
        
        <proxy-scheme>
            <service-name>site-02-trans-proxy-service</service-name>
            <acceptor-config>
                <tcp-acceptor>
                    <local-address>
                        <address>soa.suite.machine</address>
                        <port>20002</port>
                    </local-address>
                </tcp-acceptor>
            </acceptor-config>
            <autostart>true</autostart>
        </proxy-scheme>

        <proxy-scheme>
            <service-name>site-02-sync-proxy-service</service-name>
            <acceptor-config>
                <tcp-acceptor>
                    <local-address>
                        <address>soa.suite.machine</address>
                        <port>30002</port>
                    </local-address>
                </tcp-acceptor>
            </acceptor-config>
            <autostart>true</autostart>
        </proxy-scheme>

    </caching-schemes>

</cache-config>

Save this cache configuration file as coherence-cache-config-site-02.xml. Now that we have cache configuration files from both sites in place, we can set up the Coherence cluster that will hold the WAN replication enabled caches. For the site-01, create one shell script file named coherence-cache-server-site-01.sh and write the following code:

JAVA_HOME=/oracle/fmw/jrockit-jdk1.6.0_45-R28.2.7-4.1.0
COHE_HOME=/oracle/fmw/coherence_3.7
COHE_INCU=/oracle/fmw/coherence-incubator
OWLS_HOME=/oracle/fmw/wlserver_10.3
ALSB_HOME=/oracle/fmw/Oracle_OSB1

CLASSPATH=$COHE_HOME/lib/coherence.jar:$OWLS_HOME/server/lib/wljmsclient.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-common/target/coherence-common-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-eventdistributionpattern/target/coherence-eventdistributionpattern-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-messagingpattern/target/coherence-messagingpattern-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-pushreplicationpattern/target/coherence-pushreplicationpattern-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$ALSB_HOME/lib/osb-coherence-client.jar

JAVA_ARGS="-d64 -Xms256m -Xmx256m -XgcPrio:deterministic -XpauseTarget=250ms"
JAVA_ARGS="$JAVA_ARGS -Dtangosol.coherence.cacheconfig=coherence-cache-config-site-01.xml"
JAVA_ARGS="$JAVA_ARGS -Dtangosol.coherence.distributed.localstorage=true"

$JAVA_HOME/bin/java -cp $CLASSPATH $JAVA_ARGS com.tangosol.net.DefaultCacheServer

The given shell script code is self explanatory, so I will not enter in too much details. Just keep in mind that this type of cluster was designed to scale out, so if you need more storage capacity in the Coherence layer, just raise up more JVM nodes with the same configuration. Since there are no cluster defined, each JVM node that come up with will join the cluster automatically. Also, adjust the minimum and maximum heap sizes accordingly to suit your needs. Not to mention that you will need to adjust the global variables to your specific path needs.

For the site-02, create one shell script file named coherence-cache-server-site-02.sh and write the following code:

JAVA_HOME=/oracle/fmw/jrockit-jdk1.6.0_45-R28.2.7-4.1.0
COHE_HOME=/oracle/fmw/coherence_3.7
COHE_INCU=/oracle/fmw/coherence-incubator
OWLS_HOME=/oracle/fmw/wlserver_10.3
ALSB_HOME=/oracle/fmw/Oracle_OSB1

CLASSPATH=$COHE_HOME/lib/coherence.jar:$OWLS_HOME/server/lib/wljmsclient.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-common/target/coherence-common-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-eventdistributionpattern/target/coherence-eventdistributionpattern-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-messagingpattern/target/coherence-messagingpattern-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$COHE_INCU/coherence-pushreplicationpattern/target/coherence-pushreplicationpattern-11.3.1-SNAPSHOT.jar
CLASSPATH=$CLASSPATH:$ALSB_HOME/lib/osb-coherence-client.jar

JAVA_ARGS="-d64 -Xms256m -Xmx256m -XgcPrio:deterministic -XpauseTarget=250ms"
JAVA_ARGS="$JAVA_ARGS -Dtangosol.coherence.cacheconfig=coherence-cache-config-site-02.xml"
JAVA_ARGS="$JAVA_ARGS -Dtangosol.coherence.distributed.localstorage=true"

$JAVA_HOME/bin/java -cp $CLASSPATH $JAVA_ARGS com.tangosol.net.DefaultCacheServer

Execute each script on its respective site. Keep they up and running while we start the configuration of how each local OSB will connect to those clusters to delegate its caching needs.

Changing Oracle Service Bus Default Caching Configuration

The last part of the configuration is both the most simple and important one. We need to teach OSB about how to connect to a external cluster (created and configured in the previous topic) instead of using its built-in Coherence cluster. Let's start with the site-01. Edit the internal Coherence cache configuration file used by OSB located in the following folder: <DOMAIN_HOME>/config/osb/coherence/osb-coherence-cache-config.xml. You will need to change the contents of the original file with the contents of the following list below:

<?xml version="1.0"?>

<!DOCTYPE cache-config SYSTEM "cache-config.dtd">

<cache-config>

    <caching-scheme-mapping>
        <cache-mapping>
            <cache-name>/osb/service/ResultCache</cache-name>
            <scheme-name>remote-scheme-with-wan-replication</scheme-name>
        </cache-mapping>
    </caching-scheme-mapping>

    <caching-schemes>
        <near-scheme>
            <scheme-name>remote-scheme-with-wan-replication</scheme-name>
            <invalidation-strategy>all</invalidation-strategy>
            <autostart>true</autostart>
            <front-scheme>
                <local-scheme />
            </front-scheme>
            <back-scheme>
                <remote-cache-scheme>
                    <scheme-name>distributed-scheme-with-wan-replication</scheme-name>
                    <service-name>ORA-OSB-deployments</service-name>
                    <initiator-config>
                        <tcp-initiator>
                            <remote-addresses>
                                <socket-address>
                                    <address>soa.suite.machine</address>
                                    <port>20001</port>
                                </socket-address>
                            </remote-addresses>
                        </tcp-initiator>
                    </initiator-config>
                </remote-cache-scheme>
            </back-scheme>
        </near-scheme>
    </caching-schemes>
	
</cache-config>

Let's understand what is being done here. Internally, OSB was built to invoke a cache named /osb/services/ResultCache when the result cache functionality is activated for a business service. Since we have changed its caching scheme, now when the cache is accessed, it will trigger remote invocations over TCP to the distributed cache available in the 20001 port. With the usage of a near-scheme type of cache, OSB can benefit from the best of worlds: part of the most recently data stored on its heap for rapid retrieval and the other part stored in a remote distributed cache. This configuration provides both high performance and scalability with the plus of easy administration, since all the data is stored in a cluster separated of OSB.

Here is the OSB cache configuration file for site-02:

<?xml version="1.0"?>

<!DOCTYPE cache-config SYSTEM "cache-config.dtd">

<cache-config>

    <caching-scheme-mapping>
        <cache-mapping>
            <cache-name>/osb/service/ResultCache</cache-name>
            <scheme-name>remote-scheme-with-wan-replication</scheme-name>
        </cache-mapping>
    </caching-scheme-mapping>

    <caching-schemes>
        <near-scheme>
            <scheme-name>remote-scheme-with-wan-replication</scheme-name>
            <invalidation-strategy>all</invalidation-strategy>
            <autostart>true</autostart>
            <front-scheme>
                <local-scheme />
            </front-scheme>
            <back-scheme>
                <remote-cache-scheme>
                    <scheme-name>distributed-scheme-with-wan-replication</scheme-name>
                    <service-name>ORA-OSB-deployments</service-name>
                    <initiator-config>
                        <tcp-initiator>
                            <remote-addresses>
                                <socket-address>
                                    <address>soa.suite.machine</address>
                                    <port>20002</port>
                                </socket-address>
                            </remote-addresses>
                        </tcp-initiator>
                    </initiator-config>
                </remote-cache-scheme>
            </back-scheme>
        </near-scheme>
    </caching-schemes>
	
</cache-config>

As you can see, it is the same code with the same techniques. The only difference is that instead of pointing to the Coherence cluster of site-01 on port 20001, it points to the Coherence cluster of site-02 on port 20002. That's all what we need to have OSB delegating its caching needs to a remote cluster. The diagram below gives you an overview of what we have done so far.

Start OSB in both sites. During start up, OSB will connect to the Coherence cluster and establish a connection. Because of this, consider as a deployment procedure start first the Coherence cluster to after the OSB cluster. Now that we have WAN replication properly configured, let's start some tests.

Testing the WAN Replication Behavior in Oracle Service Bus

In order to test the WAN replication behavior, I have developed a simple web service which takes ten seconds to complete each request. The idea is to have this web service as a OSB business service with result cache activated. Then, you need to create a proxy service in which its only job is to route its requests to the business service. Both the proxy service and the business service should be deployed at all the sites, along with the Web Service deployment. Here is the snippet code from the web service implementation:

package com.oracle.fmw.soa.osb.demo.services;

import java.util.Date;

import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;

import com.oracle.fmw.soa.osb.demo.domain.Customer;

@WebService(name = "VerySlowWebService", serviceName = "VerySlowWebService")
public class VerySlowWebService {
	
	@SuppressWarnings("deprecation")
	@WebMethod(operationName = "findCustomerBySSN")
	public @WebResult(name = "customer") Customer findCustomerBySSN(
			@WebParam(name = "ssn") String ssn) {
		
		System.out.println("---> Entering in the Web Service Method Invocation...");
		
		try {
			
			Thread.sleep(10000);
			
		} catch (Exception ex) {
			
			ex.printStackTrace();
			
		}
		
		Customer customer = new Customer();
		customer.setSsn("832552291");
		customer.setFirstName("Ricardo");
		customer.setLastName("Ferreira");
		customer.setBirthDate(new Date(1981, 10, 05));
		
		return customer;
		
	}

}

A simple battery of tests to validate if everything is working should be:

  • Using the proxy service from site-01, make a request with "123456789" as the value of the SSN parameter. That request should take ~10 seconds to complete.
  • Using the proxy service from site-02, make a request with "123456789" as the value of the SSN parameter. That request should take ~01 second or less to complete.
  • Using the proxy service from site-02, make a request with "987654321" as the value of the SSN parameter. That request should take ~10 seconds to complete.
  • Using the proxy service from site-01, make a request with "987654321" as the value of the SSN parameter. That request should take ~01 second or less to complete.
  • Using the proxy service from site-01, make a request with "111111111" as the value of the SSN. Wait for the expiration of that entry in site-01. When it expires, check in the site-02 if the entry also expired.

Thinking in making things easier for you, I have made available all the project artifacts and OSB projects. Click in the links below to download them.

Monday Oct 28, 2013

Oracle Coherence, Split-Brain and Recovery Protocols In Detail

This article provides a high level conceptual overview of Split-Brain scenarios in distributed systems. It will focus on a specific example of cluster communication failure and recovery in Oracle Coherence. This includes a discussion on the witness protocol (used to remove failed cluster members) and the panic protocol (used to resolve Split-Brain scenarios).

Note that the removal of cluster members does not necessarily indicate a Split-Brain condition. Oracle Coherence does not (and cannot) detect a Split-Brain as it occurs, the condition is only detected when cluster members that previously lost contact with each other regain contact.

Cluster Topology and Configuration

In order to create an good didactic for the article, let's assume a cluster topology and configuration. In this example we have a six member cluster, consisting of one JVM on each physical machine. The member IDs are as follows:

Member ID  IP Address
 1  10.149.155.76
 2  10.149.155.77
 3  10.149.155.236
 4  10.149.155.75
 5  10.149.155.79
 6  10.149.155.78

Members 1, 2, and 3 are connected to a switch, and members 4, 5, and 6 are connected to a second switch. There is a link between the two switches, which provides network connectivity between all of the machines.


Member 1 is the first member to join this cluster, thus making it the senior member. Member 6 is the last member to join this cluster. Here is a log snippet from Member 6 showing the complete member set:

2010-02-26 15:27:57.390/3.062 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=main, member=6): Started DefaultCacheServer...

SafeCluster: Name=cluster:0xDDEB

Group{Address=224.3.5.3, Port=35465, TTL=4}

MasterMemberSet
  (
  ThisMember=Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)
  OldestMember=Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
  ActualMemberSet=MemberSet(Size=6, BitSetCount=2
    Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
    Member(Id=2, Timestamp=2010-02-26 15:27:17.847, Address=10.149.155.77:8088, MachineId=1101, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:296, Role=CoherenceServer)
    Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer)
    Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)
    Member(Id=5, Timestamp=2010-02-26 15:27:49.095, Address=10.149.155.79:8088, MachineId=1103, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:3229, Role=CoherenceServer)
    Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)
    )
  RecycleMillis=120000
  RecycleSet=MemberSet(Size=0, BitSetCount=0
    )
  )

At approximately 15:30, the connection between the two switches is severed:


Thirty seconds later (the default packet timeout in development mode) the logs indicate communication failures across the cluster. In this example, the communication failure was caused by a network failure. In a production setting, this type of communication failure can have many root causes, including (but not limited to) network failures, excessive GC, high CPU utilization, swapping/virtual memory, and exceeding maximum network bandwidth. In addition, this type of failure is not necessarily indicative of a split brain. Any communication failure will be logged in this fashion. Member 2 logs a communication failure with Member 5:

2010-02-26 15:30:32.638/196.928 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=PacketPublisher, member=2): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=5, Timestamp=2010-02-26 15:27:49.095, Address=10.149.155.79:8088, MachineId=1103, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:3229, Role=CoherenceServer)
by MemberSet(Size=2, BitSetCount=2
  Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
  Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)
  )

The Coherence clustering protocol (TCMP) is a reliable transport mechanism built on UDP. In order for the protocol to be reliable, it requires an acknowledgement (ACK) for each packet delivered. If a packet fails to be acknowledged within the configured timeout period, the Coherence cluster member will log a packet timeout (as seen in the log message above). When this occurs, the cluster member will consult with other members to determine who is at fault for the communication failure. If the witness members agree that the suspect member is at fault, the suspect is removed from the cluster. If the witnesses unanimously disagree, the accuser is removed. This process is known as the witness protocolSince Member 2 cannot communicate with Member 5, it selects two witnesses (Members 1 and 4) to determine if the communication issue is with Member 5 or with itself (Member 2). However, Member 4 is on the switch that is no longer accessible by Members 1, 2 and 3; thus a packet timeout for member 4 is recorded as well:

2010-02-26 15:30:35.648/199.938 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=PacketPublisher, member=2): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)
by MemberSet(Size=2, BitSetCount=2
  Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
  Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)
  )

Member 1 has the ability to confirm the departure of member 4, however Member 6 cannot as it is also inaccessible. At the same time, Member 3 sends a request to remove Member 6, which is followed by a report from Member 3 indicating that Member 6 has departed the cluster:

2010-02-26 15:30:35.706/199.996 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=2): MemberLeft request for Member 6 received from Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer)
2010-02-26 15:30:35.709/199.999 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=2): MemberLeft notification for Member 6 received from Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer)

The log for Member 3 determines how Member 6 departed the cluster:

2010-02-26 15:30:35.161/191.694 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=PacketPublisher, member=3): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)
by MemberSet(Size=2, BitSetCount=2
  Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
  Member(Id=2, Timestamp=2010-02-26 15:27:17.847, Address=10.149.155.77:8088, MachineId=1101, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:296, Role=CoherenceServer)
  )
2010-02-26 15:30:35.165/191.698 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=Cluster, member=3): Member departure confirmed by MemberSet(Size=2, BitSetCount=2
  Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
  Member(Id=2, Timestamp=2010-02-26 15:27:17.847, Address=10.149.155.77:8088, MachineId=1101, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:296, Role=CoherenceServer)
  ); removing Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)

In this case, Member 3 happened to select two witnesses that it still had connectivity with (Members 1 and 2) thus resulting in a simple decision to remove Member 6.


Given the departure of Member 6, Member 2 is left with a single witness to confirm the departure of Member 4:

2010-02-26 15:30:35.713/200.003 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=Cluster, member=2): Member departure confirmed by MemberSet(Size=1, BitSetCount=2
  Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer)
  ); removing Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)

In the meantime, Member 4 logs a missing heartbeat from the senior member. This message is also logged on Members 5 and 6.

2010-02-26 15:30:07.906/150.453 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=PacketListenerN, member=4): Scheduled senior member heartbeat is overdue; rejoining multicast group.

Next, Member 4 logs a TcpRing failure with Member 2, thus resulting in the termination of Member 2:

2010-02-26 15:30:21.421/163.968 Oracle Coherence GE 12.1.2.0.0 <D4> (thread=Cluster, member=4): TcpRing: Number of socket exceptions exceeded maximum; last was "java.net.SocketTimeoutException: connect timed out"; removing the member: 2

For quick process termination detection, Oracle Coherence utilizes a feature called TcpRing which is a sparse collection of TCP/IP-based connections between different members in the cluster. Each member in the cluster is connected to at least one other member, which (if at all possible) is running on a different physical box. This connection is not used for any data transfer, only heartbeat communications are sent once a second per each link. If a certain number of exceptions are thrown while trying to re-establish a connection, the member throwing the exceptions is removed from the cluster. Member 5 logs a packet timeout with Member 3 and cites witnesses Members 4 and 6:

2010-02-26 15:30:29.791/165.037 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=PacketPublisher, member=5): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer)
by MemberSet(Size=2, BitSetCount=2
  Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)
  Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)
  )
2010-02-26 15:30:29.798/165.044 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=Cluster, member=5): Member departure confirmed by MemberSet(Size=2, BitSetCount=2
  Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)
  Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer)
  ); removing Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer)

Eventually we are left with two distinct clusters consisting of Members 1, 2, 3 and Members 4, 5, 6, respectively. In the latter cluster, Member 4 is promoted to senior member.


The connection between the two switches is restored at 15:33. Upon the restoration of the connection, the cluster members immediately receive cluster heartbeats from the two senior members. In the case of Members 1, 2, and 3, the following is logged:

2010-02-26 15:33:14.970/369.066 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=Cluster, member=1): The member formerly known as Member(Id=4, Timestamp=2010-02-26 15:30:35.341, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) has been forcefully evicted from the cluster, but continues to emit a cluster heartbeat; henceforth, the member will be shunned and its messages will be ignored.

Likewise for Members 4, 5, and 6:

2010-02-26 15:33:14.343/336.890 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=Cluster, member=4): The member formerly known as Member(Id=1, Timestamp=2010-02-26 15:30:31.64, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) has been forcefully evicted from the cluster, but continues to emit a cluster heartbeat; henceforth, the member will be shunned and its messages will be ignored.

This message indicates that a senior heartbeat is being received from members that were previously removed from the cluster, in other words, something that should not be possible. For this reason, the recipients of these messages will initially ignore them. After several iterations of these messages, the existence of multiple clusters is acknowledged, thus triggering the panic protocol to reconcile this situation. When the presence of more than one cluster (i.e. Split-Brain) is detected by a Coherence member, the panic protocol is invoked in order to resolve the conflicting clusters and consolidate into a single cluster. The protocol consists of the removal of smaller clusters until there is one cluster remaining. In the case of equal size clusters, the one with the older Senior Member will survive. Member 1, being the oldest member, initiates the protocol:

2010-02-26 15:33:45.970/400.066 Oracle Coherence GE 12.1.2.0.0 <Warning> (thread=Cluster, member=1): An existence of a cluster island with senior Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) containing 3 nodes have been detected. Since this Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) is the senior of an older cluster island, the panic protocol is being activated to stop the other island's senior and all junior nodes that belong to it.

Member 3 receives the panic:

2010-02-26 15:33:45.803/382.336 Oracle Coherence GE 12.1.2.0.0 <Error> (thread=Cluster, member=3): Received panic from senior Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) caused by Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer)

Member 4, the senior member of the younger cluster, receives the kill message from Member 3:

2010-02-26 15:33:44.921/367.468 Oracle Coherence GE 12.1.2.0.0 <Error> (thread=Cluster, member=4): Received a Kill message from a valid Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer); stopping cluster service.

In turn, Member 4 requests the departure of its junior members 5 and 6:

2010-02-26 15:33:44.921/367.468 Oracle Coherence GE 12.1.2.0.0 <Error> (thread=Cluster, member=4): Received a Kill message from a valid Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer); stopping cluster service.

2010-02-26 15:33:43.343/349.015 Oracle Coherence GE 12.1.2.0.0 <Error> (thread=Cluster, member=6): Received a Kill message from a valid Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer); stopping cluster service.

Once Members 4, 5, and 6 restart, they rejoin the original cluster with senior member 1. The log below is from Member 4. Note that it receives a different member id when it rejoins the cluster.

2010-02-26 15:33:44.921/367.468 Oracle Coherence GE 12.1.2.0.0 <Error> (thread=Cluster, member=4): Received a Kill message from a valid Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer); stopping cluster service.
2010-02-26 15:33:46.921/369.468 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Service Cluster left the cluster
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Invocation:InvocationService, member=4): Service InvocationService left the cluster
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=OptimisticCache, member=4): Service OptimisticCache left the cluster
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=ReplicatedCache, member=4): Service ReplicatedCache left the cluster
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=DistributedCache, member=4): Service DistributedCache left the cluster
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Invocation:Management, member=4): Service Management left the cluster
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Member 6 left service Management with senior member 5
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Member 6 left service DistributedCache with senior member 5
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Member 6 left service ReplicatedCache with senior member 5
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Member 6 left service OptimisticCache with senior member 5
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Member 6 left service InvocationService with senior member 5
2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=4): Member(Id=6, Timestamp=2010-02-26 15:33:47.046, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) left Cluster with senior member 4
2010-02-26 15:33:49.218/371.765 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=main, member=n/a): Restarting cluster
2010-02-26 15:33:49.421/371.968 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2010-02-26 15:33:49.625/372.172 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=Cluster, member=n/a): This Member(Id=5, Timestamp=2010-02-26 15:33:50.499, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) joined cluster "cluster:0xDDEB" with senior Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)

Cool isn't it?

About

Ricardo Ferreira is just a regular person passionate for technology, traveling, movies and his family. Currently he is working at Oracle U.S in the FMW Architects Team, otherwise known as the A-Team.

Search

Categories
Archives
« February 2015
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
       
       
Today