Tuesday Sep 10, 2013

One Client Two Clusters

Sometimes its desirable to have a client connect to multiple clusters, either because the data is dispersed or for instance the clusters are in different locations for high availability. For an extend or external client, which connects to a Coherence cluster via a proxy service over TCP, this is very straight forward. This post outlines how to do this and comes with a simple Java example you can try out yourself.[Read More]

Thursday Jul 19, 2012

Dynamic Authorised Hosts

A simple way to help secure a Coherence cluster is to configure the authorised hosts that can be part of the cluster. If a Coherence application tries to join a cluster and its not running on a server in the authorised host it will be rejected. To setup authorised hosts the cluster host names or IP addresses can be explicitly added to the authorised hosts section of the tangosol-coherence-override.xml file, as shown below:

  <authorized-hosts>
    <host-address id="1">192.168.56.101</host-address>
    <host-address id="2">192.168.56.1</host-address>
  </authorized-hosts>

But what happens when your cluster needs to grow and you need to add another server that is not in the authorised host list or your cluster topology needs to change that and this includes a hostnames or IP address of the server not in the list?

Sometimes a full cluster restart with a different configuration is an option but in other cases this is not. Fortunately there are a couple of solutions to this problem - apart from a complete cluster re-start. 

The first is to perform a rolling-restart (shutting down each node in turn, changing its configuration and restarting it). But this operation will impact cluster performance as cache data will need to be recovered and re-balanced. Another option is to specify a range of authorised hosts rather than explicitly name them. This approach balances the need to secure your cluster with the need to change the configuration at some point in the future. A range of hosts that con join a cluster can be specified as follows:

  <authorized-hosts>
    <host-range>
      <from-address>192.168.56.101</from-address>
      <to-address>192.168.56.190</to-address>
    </host-range>
  </authorized-hosts>

A third option is to use a Coherence filter to determine which hosts are authorised to join the cluster. This allows for a more dynamic configuration as a custom filter can access a separate file with a list of hosts, LDAP server or database. By using a filter its possible to specify exactly which hosts can run cluster members while at the same time allowing this lit to be changed dynamically.

A filter to determine authorised hosts can be added to a cluster configuration (tangosol-coherence-override.xml file) as shown below:

  <authorized-hosts>
    <host-filter>
      <!-- Custom Filter to perform authorised host check -->
      <class-name>com.oracle.coherence.test.AuthroizedHostsFilter</class-name>
      <init-params>
        <!-- URL for authorised host list -->
        <init-param>
          <param-type>String</param-type>
          <param-value>

            file:/Users/Dave/workspace/CoherenceAuthorizedHostsTest/hosts.txt

          </param-value>
        </init-param>
        <!-- Time in ms between re-reading authorised hosts -->
        <init-param>
          <param-type>Int</param-type>
          <param-value>10000</param-value>
        </init-param>
      </init-params>
    </host-filter>
  </authorized-hosts>

And an example custom filter that periodically checks a list of hosts at a URL could look like this:

/**
 * File: AuthorizedHostFilter.java
 * 
 * Copyright (c) 2012. All Rights Reserved. Oracle Corporation.
 * 
 * Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
 * 
 * This software is the confidential and proprietary information of Oracle
 * Corporation. You shall not disclose such confidential and proprietary
 * information and shall use it only in accordance with the terms of the license
 * agreement you entered into with Oracle Corporation.
 * 
 * Oracle Corporation makes no representations or warranties about the
 * suitability of the software, either express or implied, including but not
 * limited to the implied warranties of merchantability, fitness for a
 * particular purpose, or non-infringement. Oracle Corporation shall not be
 * liable for any damages suffered by licensee as a result of using, modifying
 * or distributing this software or its derivatives.
 * 
 * This notice may not be removed or altered.
 */
package com.oracle.coherence.test;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Timer;
import java.util.TimerTask;

import com.tangosol.net.CacheFactory;
import com.tangosol.util.Filter;

/**
 * Simple filter to check if a host is in an authoirsed host list
 * Note: this implementation checks the IP address of a host not the hostname
 * 
 * @author Dave Felcey
 */
public class AuthroizedHostsFilter implements Filter {
  /**
   * List of authorised hosts. Thi list is synchronised in case an update and check are being 
   * perform at the same time
   */
  private List hosts = Collections.synchronizedList(new ArrayList());
  /**
   * URL where authorised host list is located
   */
  private String hostsFileUrl;
  /**
   * Timer use to re-read authorised hosts
   */
  private Timer timer = new Timer();
  
  /**
   * Constructor for AuthroizedHostsFilter
   * @param hostsFileUrl the URL where authorised hosts list is located
   * @param reLoadInterval interval in ms at which authorised hosts list is re-read
   */
  public AuthroizedHostsFilter(String hostsFileUrl, int reLoadInterval) {
    this.hostsFileUrl = hostsFileUrl;
    
    // Load values
    load();
    
    // Schedule periodic reload
    timer.scheduleAtFixedRate(new TimerTask() {
        public void run() {
          CacheFactory.log("About to refresh host list");
          load();
        }
    }, reLoadInterval, reLoadInterval);
  }

  /**
   * Loads authorised host list
   */
  private void load() {
    try {
      CacheFactory.log("Curent dir: " + System.getProperty("user.dir"), CacheFactory.LOG_DEBUG);
      CacheFactory.log("Loading hosts file from URL: " + hostsFileUrl, CacheFactory.LOG_DEBUG);
      URL url = new URL(hostsFileUrl);
      BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));

      String inputLine;
      while ((inputLine = in.readLine()) != null) {
          CacheFactory.log("Host IP address: " + inputLine, CacheFactory.LOG_DEBUG);
          hosts.add(inputLine);
      }
      in.close();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
  
  @Override
  public boolean evaluate(Object host) {
    String h = host.toString();   
    h = h.substring(h.indexOf('/') + 1);
    
    CacheFactory.log("Validating host IP address: " + host + "(" + h + "), " + hosts.contains(h), CacheFactory.LOG_DEBUG);
    return hosts.contains(h);
  }
}

The dynamic lookup in the example above could be secured further by requiring some kind of credentials for the lookup or instead of polling the resource some kind of change notification could be used to tell the filter when to re-read the authorised hosts. However, in many cases the above approach should be adequate. A full example of the above filter is available from here.

Friday Jun 01, 2012

Throttling Cache Events

The real-time eventing feature in Coherence is great for relaying state changes to other systems or to users. However, sometimes not all changes need to or can be sent to consumers. For instance;

  • If rapid changes cannot be consumed or interpreted as fast as they are being sent. A user looking at changing Stock prices may only be able to interpret and react to 1 change per second.
  • A client may be using low bandwidth connection, so rapidly sending events will only result in them being queued and delayed
  • A large number of clients may need to be notified of state changes and sending 100 events p/s to 1000 clients cannot be supported with the available hardware, but 10 events p/s to 1000 clients can. Note this example assumes that many of the state changes are to the same value.

One simple approach to throttling Coherence cache events is to use a cache store to capture changes to one cache (data cache) and insert those changes periodically in another cache (events cache). Consumers interested in state changes to entires in the first cache register an interest (event listener) against the second event cache. By using the cache store write-behind feature rapid updates to the same cache entry are coalesced so that updates are merged and written at the interval configured to the event cache. The time interval at which changes are written to the events cache can easily be configured using the write-behind delay time in the cache configuration, as shown below.

  <caching-schemes>
    <distributed-scheme>
      <scheme-name>CustomDistributedCacheScheme</scheme-name>
      <service-name>CustomDistributedCacheService</service-name>
      <thread-count>5</thread-count>
      <backing-map-scheme>
        <read-write-backing-map-scheme>
          <scheme-name>CustomRWBackingMapScheme</scheme-name>
          <internal-cache-scheme>
            <local-scheme />
          </internal-cache-scheme>
          <cachestore-scheme>
            <class-scheme>
              <scheme-name>CustomCacheStoreScheme</scheme-name>
              <class-name>com.oracle.coherence.test.CustomCacheStore</class-name>
              <init-params>
                <init-param>
                  <param-type>java.lang.String</param-type>
                  <param-value>{cache-name}</param-value>
                </init-param>
                <init-param>
                  <param-type>java.lang.String</param-type>
                  <!-- The name of the cache to write events to -->
                  <param-value>cqc-test</param-value>
                </init-param>
              </init-params>
            </class-scheme>
          </cachestore-scheme>
          <write-delay>1s</write-delay>
          <write-batch-factor>1</write-batch-factor>
        </read-write-backing-map-scheme>
      </backing-map-scheme>
      <autostart>true</autostart>
    </distributed-scheme>
  </caching-schemes>

The cache store implementation to perform this throttling is trivial and only involves overriding the basic cache store functions.

public class CustomCacheStore implements CacheStore {
	private String publishingCacheName;
	private String sourceCacheName;

	public CustomCacheStore(String sourceCacheStore, String publishingCacheName) {
		this.publishingCacheName = publishingCacheName;
		this.sourceCacheName = sourceCacheName;
	}

	@Override
	public Object load(Object key) {
		return null;
	}

	@Override
	public Map loadAll(Collection keyCollection) {
		return null;
	}

	@Override
	public void erase(Object key) {
		if (sourceCacheName != publishingCacheName) {
			CacheFactory.getCache(publishingCacheName).remove(key);
			CacheFactory.log("Erasing entry: " + key, CacheFactory.LOG_DEBUG);
		}
	}

	@Override
	public void eraseAll(Collection keyCollection) {
		if (sourceCacheName != publishingCacheName) {
			for (Object key : keyCollection) {
				CacheFactory.getCache(publishingCacheName).remove(key);
				CacheFactory.log("Erasing collection entry: " + key,
						CacheFactory.LOG_DEBUG);
			}
		}
	}

	@Override
	public void store(Object key, Object value) {
		if (sourceCacheName != publishingCacheName) {
			CacheFactory.getCache(publishingCacheName).put(key, value);
			CacheFactory.log("Storing entry (key=value): " + key + "=" + value,
					CacheFactory.LOG_DEBUG);
		}
	}

	@Override
	public void storeAll(Map entryMap) {
		if (sourceCacheName != publishingCacheName) {
			CacheFactory.getCache(publishingCacheName).putAll(entryMap);
			CacheFactory.log("Storing entries: " + entryMap,
					CacheFactory.LOG_DEBUG);
		}
	}
} 

As you can see each cache store operation on the data cache results in a similar operation on event cache. This is a very simple pattern which has a lot of additional possibilities, but it also has a few drawbacks you should be aware of:

  • This event throttling implementation will use additional memory as a duplicate copy of entries held in the data cache need to be held in the events cache too - 2 if the event cache has backups
  • A data cache may already use a cache store, so a "multiplexing cache store pattern" must also be used to send changes to the existing and throttling cache store. 
If you would like to try out this throttling example you can download it here. I hope its useful and let me know if you spot any further optimizations.

Tuesday May 15, 2012

Testing the Coherence Simple Partition Assignment Strategy

The Simple Partition Assignment Strategy introduced with Coherence 3.7.1 now allows Coherence to re-balance cache data partitions for primary and backup data using a centralised partitioning strategy. Previously (and the current default) each cluster member determined how cache data partitions were distributed autonomously (by itself), for instance by taking the total number of partitions and dividing it by the number of cluster members.

A centralised strategy allows the complete topology of the cluster to be taken into account. Like the default autonomous strategy the simple partition assignment strategy tries to fairly distribute cache data partitions, but as far as is possible it also tries to ensure "data safety", by placing primary and backup copies of data on different sites, racks and machines. So if there are multiple sites then primary and backup data will be put on different sites.  If a cluster spans multiple racks then primary and backup partitions will be put on different racks so a complete rack failure will not result in data loss. Lastly if all the machines are on the same rack then primay and backup partitions will be placed on different machines.

It should be noted that in order for the the simple partition assignment strategy to determine where cache data partitions should be located each member in the cluster should specify its full identity, i.e. the name of the machine it is on, its rack and site using member identity parameters. These are usually set as system properties (as shown below) but can also be placed in the cluster override file.

-Dtangosol.coherence.cluster=ProdAppCluster

-Dtangosol.coherence.site=MainDataCentre

-Dtangosol.coherence.rack=Rack07

-Dtangosol.coherence.machine=Blade11

Normally to test that the simple partition assignment strategy works would involve quite a bit of setup. However, a small test framework called "littlegrid" (written by Jon Hall) enables the whole test to be run in a single JVM and written as a simple JUnit test. Below is the cache configuration file that introduces the new simple assignment strategy class (to the distributed service CustomDistributedCacheService) and is highlighted in bold.

<?xml version="1.0"?>

<cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config 
    http://xmlns.oracle.com/coherence/coherence-cache-config/1.0/coherence-cache-config.xsd">
  <defaults>
    <serializer>pof</serializer>
  </defaults>
  <caching-scheme-mapping>
    <cache-mapping>
      <cache-name>*</cache-name>
      <scheme-name>CustomDistributedCacheScheme</scheme-name>
    </cache-mapping>
  </caching-scheme-mapping>
  <caching-schemes>
    <distributed-scheme>
      <scheme-name>CustomDistributedCacheScheme</scheme-name>
      <service-name>CustomDistributedCacheService</service-name>
      <thread-count>5</thread-count>
      <partition-assignment-strategy>
        <instance>
          <class-name>com.tangosol.net.partition.SimpleAssignmentStrategy
          </class-name>
        </instance>
      </partition-assignment-strategy>
      <partition-listener>
        <class-name>com.oracle.coherence.test.PartitionLostListener</class-name>
      </partition-listener>
      <backing-map-scheme>
         "><local-scheme>
        </local-scheme>
      </backing-map-scheme>
      <autostart>true</autostart>
    </distributed-scheme>
  </caching-schemes>
</cache-config> 

Now here is the JUnit test setup using the "littlegrid" test framework to simulate a rack failure. 3 racks, each containing 2 machines with 2 nodes in each machine is created, along with a management node and obviously a storage disabled test client. The last 2 nodes are in a separate rack - "default rack". The test involves the following steps:

  1. Some data is added to a test cache so that data is held by each partition
  2. A random rack (from the 3 created) is selected and all the nodes in that rack shutdown at the same time
  3. After a short pause the JMX MBean "Partition Lost" - shown below - is checked on all remaining storage members to ensure that no partitions were lost.

This last step is possible because a Partition Event Listener is registered with the partitioned cache service (CustomDistributedCacheService) in the cache configuration file. This Partition Event Listener also exposes itself as  JMX MBean, making it possible to check for this kind of event.

The code for the test is shown below:


package com.oracle.coherence.test;

import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

import java.lang.management.ManagementFactory;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;
import java.util.Set;
import java.util.concurrent.TimeUnit;

import javax.management.MBeanServerConnection;
import javax.management.ObjectName;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.littlegrid.ClusterMemberGroup;
import org.littlegrid.ClusterMemberGroupUtils;

import com.tangosol.net.CacheFactory;
import com.tangosol.net.Member;
import com.tangosol.net.NamedCache;
import com.tangosol.net.PartitionedService;

/**
 * Coherence simple assignment strategy tests.
 * 
 * @author Dave Felcey
 */
public class TestCase {
  private ClusterMemberGroup memberGroup;
  private NamedCache cache = null;
  private int[][] racks = null;

  @Before
  public void setUp() {
    // Create member group now, so later code simply merges into this group
    memberGroup = ClusterMemberGroupUtils.newBuilder()
        .buildAndConfigureForNoClient();

    final int numberOfRacks = 3;
    final int numberOfMachines = 2;
    final int numberOfStorageEnabledMembersPerMachine = 2;
    final int expectedClusterSize = (numberOfRacks
        * numberOfMachines * numberOfStorageEnabledMembersPerMachine);

    racks = new int[numberOfRacks][numberOfMachines
        * numberOfStorageEnabledMembersPerMachine];

    // Start up the storage enabled members on different racks and machines
    for (int rack = 0; rack < numberOfRacks; rack++) {
      for (int machine = 0; machine < numberOfMachines; machine++) {
        // Build using the identity parameters
        memberGroup.merge(ClusterMemberGroupUtils
            .newBuilder()
            .setFastStartJoinTimeoutMilliseconds(100)
            .setSiteName("site1")
            .setMachineName("r-" + rack + "-m-" + machine)
            .setRackName("r-" + rack)
            .setStorageEnabledCount(
                numberOfStorageEnabledMembersPerMachine)
            .buildAndConfigureForNoClient());
      }

      // Save member id's for rack
      System.arraycopy(memberGroup.getStartedMemberIds(),
          rack * numberOfMachines
              * numberOfStorageEnabledMembersPerMachine,
          racks[rack], 0, numberOfMachines
              * numberOfStorageEnabledMembersPerMachine);
    }

    // Create Management and client members with default rack and machine
    // identities
    memberGroup.merge(ClusterMemberGroupUtils.newBuilder()
        .setJmxMonitorCount(1).setLogLevel(9)
        .buildAndConfigureForStorageDisabledClient());

    assertThat(
        "Cluster size check - includes storage disabled client and JMX monitor",
        CacheFactory.ensureCluster().getMemberSet().size(),
        is(expectedClusterSize + 2));

    assertThat(
        "Member group check size is as expected - includes JMX monitor, but not storage disabled client",
        memberGroup.getStartedMemberIds().length,
        is(expectedClusterSize + 1));

    cache = CacheFactory.getCache("test");
  }

  /**
   * Demonstrate SimpleAssignementStrategy.
   */
  @Test
  public void testSimpleAssignmentStrategy()
      throws Exception {

    final Map entries = new HashMap();
    final int totalEntries = 1000;

    // Load test data
    for (int i = 0; i < totalEntries; i++) {
      entries.put(i, "entry " + i);
    }

    cache.putAll(entries);
    assertThat(cache.size(), is(totalEntries));

    // Kill rack - if partition lost then will exit
    Random random = new Random();
    int rack = Math.abs(random.nextInt() % racks.length);
    System.out.println("Stopping rack: " + rack);

    memberGroup.stopMember(racks[rack]);

    System.out
        .println("Pausing to allow data to be recovered");
    TimeUnit.SECONDS.sleep(memberGroup
        .getSuggestedSleepAfterStopDuration()
        * racks[rack].length * 10);

    assertThat("Partition lost",
        getPartitionLostEventCount(), is(0));
    assertThat("Cache size", cache.size(), is(totalEntries));
  }

  @After
  public void tearDown() {
    // Quick stop all - members *don't* leave the cluster politely - done for
    // this test so it shuts down quicker
    memberGroup.stopAll();

    ClusterMemberGroupUtils
        .shutdownCacheFactoryThenClusterMemberGroups(memberGroup);
  }

  /**
   * Get the number of partitions lost.
   * 
   * @return partitions lost
   */
  private int getPartitionLostEventCount() throws Exception {

    // Create an MBeanServerConnection
    final MBeanServerConnection connection = ManagementFactory
        .getPlatformMBeanServer();

    @SuppressWarnings("unchecked")
    final Set members = ((PartitionedService) cache
        .getCacheService()).getOwnershipEnabledMembers();
    final String serviceName = cache.getCacheService()
        .getInfo().getServiceName();

    int lostPartitions = 0;

    // Get any partition lost event information from cluster members
    for (Member member : members) {
      String path = "Coherence:type=PartitionListener,name=PartitionLostCount,service="
          + serviceName
          + ",id=PartitionLost,nodeId="
          + member.getId();

      lostPartitions += (Integer) connection.getAttribute(
          new ObjectName(path), "PartitionLostCount");
    }

    return lostPartitions;
  }
}

By extending the pause time after the rack has been stopped you can view the PartitionLost MBean information in JConsole, as shown below:

JConsole

The PartitionLostListerner (based on an example by Andrew Wilson) looks like this: 

package com.oracle.coherence.test;

import java.util.concurrent.atomic.AtomicInteger;

import com.tangosol.net.CacheFactory;
import com.tangosol.net.management.Registry;
import com.tangosol.net.partition.PartitionEvent;
import com.tangosol.net.partition.PartitionListener;

/**
 * Partition lost listener
 */
public class PartitionLostListener implements
    PartitionListener, PartitionLostListenerMBean {
  private final AtomicInteger lostCount = new AtomicInteger();
  private String serviceName;

  public PartitionLostListener() {
  }

  @Override
  public synchronized void onPartitionEvent(
      PartitionEvent partitionEvent) {
    // Initialize if necessary
    if (serviceName == null) {
      serviceName = partitionEvent.getService().getInfo()
          .getServiceName();
      System.out.println("Registering JMX");
      Registry reg = CacheFactory.getCluster()
          .getManagement();
      reg.register(
          reg.ensureGlobalName("Coherence:type=PartitionListener,name=PartitionLostCount,service="
              + serviceName + ",id=PartitionLost"), this);
      System.out.println("Registered JMX");
    }

    // Handle the event
    if (partitionEvent.getId() == PartitionEvent.PARTITION_LOST) {
      System.out.println("Partition lost: "
          + partitionEvent);
      lostCount.addAndGet(partitionEvent.getPartitionSet()
          .cardinality());
    }
  }

  // Returns any partitions lost and resets
  public synchronized Integer getPartitionLostCount() {
    int temp = lostCount.get();
    lostCount.set(0);
    return temp;
  }
}

If you would like to try this for yourself then you can download the complete example form here and find the "littlegrid" test tool here. Please note that to properly test the Simple Partition Strategy you should get the latest release of Coherence (which is available from Oracle Support and at the time of writting is 3.7.1.4). Unfortunately patch releases are not available from OTN. Another tip for using the simple partition assignment strategy with large clusters is to use the distribution quorum feature during cluster startup. It stops partition re-balancing taking place until a cluster membership threshold has been reached eliminating unneccessary network traffic and processing. A good starting point would be to set the threshold for an n rack cluster to be at n - 1 * number of machines per rack - 1. ⁞The restore and read quorum features can also be used to mitigate the the effects of a "split brain" scenario by making data read-only and enabling a cluster to reform should this occur.

Thursday Feb 16, 2012

Coherence Clustering Principles

Overview 

A Coherence environment consists of a number of components. Below I’ll describe how they release to each other and what the terms mean. But just to give you a flavor, here they are displayed as a hierarchy.

Coherence Cluster

Distributed cache services with the same name will cluster together to manage their cache data. So you can have multiple clustered services across Coherence nodes. This article will explain how these components work together and how applications interacts with them.

What do we mean by a Coherence Cluster?

It’s a set of configuration parameters that control the operational and run-time settings for clustering, communication, and data management services. These are defined in the Coherence operational override file and include such things as:
  • Multi-cast or unicast addresses for locating cluster members
  • Cluster identity information
  • Management settings
  • Networking parameters
  • Security information
  • Logging information
  • Service configuration parameters

These settings are used by Coherence services to communicate with other nodes, to determine cluster membership, for logging and other operational parameters and are similar to the Domain configuration used by Weblogic Server. They also apply to the entire Coherence cluster node, which usually means the whole JVM. Although a cluster node will usually equate to a JVM, it is possible to have more than one node per JVM by loading the Coherence libraries multiple times in different isolated class loaders, e.g. a "child first" class loader. However, this is only usually done within the contaxt of an application server or a test framework.

The coherence.jar file contains a default operational override file – tangosol-coherence-override.xml – that will be used if another is not detected in the CLASSPATH before the coherence.jar library is loaded. In actual fact there are 3 versions of this file;
  • tangosol-coherence-override-eval.xml
  • tangosol-coherence-override-dev.xml
  • tangosol-coherence-override-prod.xml

Which is selected will depend on the mode that Coherence is running in (the default is Developer mode) and can be set using the system property -Dtangosol.coherence.mode=prod.

Note: Its important to ensure that the production mode is selected for production usage, as in production mode certain communication timeouts etc will be extended so that Coherence will wait longer for services to recover – amongst other things.

What is a Coherence Service?

A Coherence service is a thread (and sometimes pool of worker threads) that has a specific function. This can be:
  • Connectivity Services
    • Clustering Service – manage cluster membership communications. There is exactly one of these services per JVM (or within a class-loader). This service tracks which other nodes are in the cluster, node failures etc.
    • Proxy Services – manage external connections into the cluster from “extend” clients
  • Processing Services
    • Invocation Service – executes processing requests/tasks on the node in the cluster it is running on.
  • Data Services
    • Distributed Cache Service – manages cache data for caches created from the scheme its defined in
    • Replicated Cache Service – provides synchronous replication cache services, where all managed cache data exists on every cluster node 
This document will focus on Distributed Cache Services that manage distributed caches defined from a distributed schema definition. 

When a schema definition (as shown below) is parsed Coherence instantiates a service thread with the name specified. This service thread will manage the data from caches created using the schema definition. So how does this all happen then?

If an application using Coherence calls:

NamedCache tradeCache = CacheFactory.getCache(“trade-cache”);

A couple of things happen:

  • When the Coherence classes are loaded they will by default search the CLASSPATH for the coherence-cache-config.xml file – which is actually the name specified in the default operational override file. The first instance that is found wild be used. However, if one is specified using the system property –Dtangosol.coherence.cacheconfig=<cache config file> it will use that cache configuration file instead. Also a cache configuration can be explicitly loaded from the CLASSPATH, as follows:

// Load a applicaton X cache configuration file
ClassLoader loader  = getClass().getClassLoader();
// Note: it’s a URI that is specified for the location
// of the cache configuration file
ConfigurableCacheFactory factory = CacheFactory.getCacheFactoryBuilder.getConfigurableCacheFactory(“applicationx-cache-config”, loader);

NamedCache localCacheInstance = factory.ensureCache("trade-cache", loader);

  • When the cache configuration file is parsed distributed cache service threads are started for all the cache “schemes” that are defined if the autostart parameter is set to true (by default its false).

<caching-schemes>
  <distributed-scheme>
    <scheme-name>DefaultDistributedCacheScheme</scheme-name>
    <service-name>DefaultDistributedCacheService</service-name>
    <autostart>true</autostart>
  </distributed-scheme>
</caching-schemes>

Note: For data services the autostart flag is only observed for distributed caches. So a replicated cache service would automatically be started.
  • Each cache service started is given a name - or one is created if none is specified.  This service thread then attempts to join with other services that have the same name in the Coherence cluster. If none are found, i.e. it’s the first to start, it will become the “senior member” of the service. To illustrate this take a look at a sample log statement when Coherence starts a service.
2012-02-02 11:00:04.666/16.462 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DefaultDistributedCacheService, member=1): Service DefaultDistributedCacheService joined the cluster with senior service member 1

In this case a distributed cache service called DefaultDistributedCacheService has started up on Member 1 of the cluster (the first JVM). As it’s the first service with this name to start it becomes the senior member – which means it has a couple of extra responsibilities, like sending out heartbeats etc.
  • Once the cache services have been started Coherence will try and match the cache name that has been passed, in this case “trade-cache”, with the appropriate cache scheme (and service) that will manage this cache. It uses the cache scheme mappings part of the cache configuration file to do this and wild card matching (if necessary) to identify the right cache scheme. 

Note: Regular expression parsing is not used and wild cards cannot be used at the start of the cache name.

<caching-scheme-mapping>
  <cache-mapping>
    <cache-name>trade-*</cache-name>
    <scheme-name>DefaultDistributedCacheScheme</scheme-name>
  </cache-mapping>
</caching-scheme-mapping>
  • Once the correct cache scheme has been matched a reference to an existing cache managed by the cache service for this scheme will be returned or a new cache created, using  the parameters of the cache scheme.
Cache services that use the same cluster name should try and scope their names to prevent name clashes. This is so that multiple applications sharing the same cluster don’t inadvertently use the same service name - and unintentionally shares data. 

Each application that wishes to cache data in isolation from others can load its own cache configuration file (as shown above) and specify a unique scope name. This will then be used as a prefix to cluster service names to prevent service name collision. Below is an example scope definition in a cache configuration file:

<?xml version="1.0"?>
<cache-config>
<scope-name>com.oracle.custom</scope-name>
...
</cache-config>

And in JConsole you can see the scope prefix being used in conjunction with service names – circled in red. 

JConsole

Note: Its not used as a prefix to cache names.

By scoping service names multiple services can co-exist in the same Coherence cluster without interacting, thus allowing individual applications to create and manage their own caches in isolation.

Below we have 3 nodes (JVM’s) each running 2 partitioned cache services. Each cache service manages 2 caches and the data (key/value pairs) stored in these caches is shared evenly between the nodes.

Coherence Nodes

Key: S=Service, C=Cache

If you want to find out more information about caches, how they relate to maps and how they are partitioned across cache services please see this (http://blackbeanbag.net/wp/2010/07/01/partitions-backing-maps-and-services-oh-my/) excellent blog article covering these topics. 

Monday Feb 13, 2012

Oracle Coherence SIG Meeting on Thursday 1st March at the Oracle City Office in London

If you are interested in finding out more about what customers are doing with Coherence or how it can be used, then why not register for the up-coming London Coherence Special Interest Group (SIG) meeting. Its in the Oracle City office on Thursday 1st March and FREE.  There will be talks from customers, partners and our engineers - amongst others - covering a wide range of topics.

Thursday Oct 20, 2011

Oracle Coherence SIG Meeting on 4th November at the Oracle City Office in London

If you are interested in finding out more about what customers are doing with Coherence or how it can be used, then why not register for the up-coming London Coherence Special Interest Group (SIG) meeting. Its in the Oracle City office on 4th November and FREE.  There will be talks from customers, partners and our engineers - amongst others - covering a wide range of topics.

Tuesday Aug 09, 2011

Using Weblogic Server ActiveCache for Coherence

As web applications are architected/re-architected caching is often added to reduce data access bottlenecks that often hinder scalability and performance. Weblogic Server (WLS) now has a range of features, collectively called "ActiveCache", that provides Coherence cache integration in a Java Servlet context, a JPA context (Toplink Grid) and an HTTP session context (Coherence*Web).Ê This post will concentrate on the first of these integration points and show how a J2EE web application cab be packaged, deployed and managed in WLS to seamlessly take advantage of a data caching tier.

Deployment Architecture

So how do all the parts work together? With WLS 10.3.4+ you can now use its NodeManager (a process monitoring and management service) to manage components other than WLS, like Coherence. There are also additional administration pages in the WLS Administration Console (http://localhost:7001/console) to configure Coherence clusters (via the Environment > Coherence Clusters menu):

And Coherence cluster nodes (via the Environment > Coherence Servers menu):

WLS Admin Console Coherence Nodes

These allow a cluster to be defined and all the cluster nodes to be started and stopped from the WLS Admin console.

In this example the Servlet and all the Coherence cache nodes are all part of the same Coherence cluster, but the Servlet does not hold any data, i.e. its storage attribute is disabled. This means that when it is deployed or un-deployed no cache data needs to be moved between the cluster nodes - because it will not hold any data. This topology is shown below:

In-cluster topology

Note: the diagram above shows how HTTP session data can be cached in Coherence but the topology is identical for this use case, where web application data is being cached.

Setup

These instructions make a few assumptions, namely that:

To setup the environment to deploy the sample application the following steps are required:

  1. Configure and startup WLS NodeManager that will start and stop the Coherence cluster nodes.

    This step may be obvious for readers who are familiar with WLS, but just for completeness these are the steps I needed to do on a Mac.

    FirstÊ I set the environment parameters used to start-up WLS:


    export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
    # Dir where WLS installed to
    export MW_HOME=/Users/Dave/apps/wls1035_dev/
    export USER_MEM_ARGS="-Xmx1024m -XX:PermSize=1024m"


    Then I set the NodeManager parameters in the nodemanager.properties file ($MW_HOME/wlserver/common/nodemanager/nodemanager.properties) so that it did not try to use the native libraries (this step is platform specific) and that SSL was not used - to make the setup simpler;

    NativeVersionEnabled=false
    PropertiesVersion=10.3
    ListenAddress=localhost
    SecureListener=false
    DomainRegistrationEnabled=true


    I also added the WLS domain that my application will be deployed to the nodemamanger.domain ($MW_HOME/wlserver/common/nodemanager/nodemanager.domain) properties file;

    base_domain=/Users/Dave/apps/mywls/user_projects/domains/base_domain

    Finally I started the NodeManager for my laptop (note a NodeManager is associated with a machine and not a domain);

    cd $MW_HOME/wlserver/server/bin
    ./startNodeManager.sh
  2. Configure a new Machine definition using the WLS Admin console

    Next you need to define a Machine definition that will enable a NodeManager to be used to start the Coherence cluster. As was mentioned above, a NodeManager is associated with a server not a domain and "A machine is the logical representation of the computer that hosts one or more Weblogic Server instances (servers)."

    To configure a Machine navigate to Machines screen in the WLS Admin console via the Environment > Machines menu and click on the New button and specify:

    Screen 1
    Name: MyMachine - the name of your machine
    Machine OS: Other - I did not specify one here as there are no native libraries for Mac OSX

    WLS create machine - screen 1

    Screen 2
    Type: Plain - so as to avoid additional security configuration

    Note: I left all other parameters to the defaults.

    WLS create machine - screen 2

    As the NodeManager has been started up using the default parameters it should be found without specifying any additional setup. The monitoring page for the machine should then show that the NodeManager is reachable.

    WLS Machine status
    If the status is not reachable the take a look at the output for the NodeManager if you have started it from a console or at the NodeManager log file ($MW_HOME/wlserver/common/nodemanager/nodemanager.log).
  3. Configure and startup the Coherence cluster

    To startup the Coherence cluster that will be used to cache the web application data you need to create a Coherence cluster configuration and then define the and startup the Coherence servers. To configure the Coherence cluster navigate to the Coherence cluster definition screen via the Environment > Coherence Clusters menu. The configuration parameters I used were:

    Name: MyCluster - this is the name of the cluster and will help prevent accidental clustering between different environments
    Use a Custom Cluster Configuration File: /Users/Dave/Documents/workspace/WLSCohWebAppEAR/EarContent/APP-INF/classes/tangosol-coherence-override.xml - this is the Coherence operational override file for the cluster, NOT the cache configuration file

    To configure the Coherence cluster servers navigate to the Coherence cluster definition screen via the Environment > Coherence Servers menu. The
    General configuration parameters I used were:

    Name: CacheServer1 - this can be any name
    Machine: MyMachine - select the name of the Machine definition you have just created
    Cluster: MyCluster - select the name of the cluster you have just created
    Unicast Listen Port: 9999 - the default is 8888 but its worth selecting something different and from that other clusters to prevent your servers accidentally joining another cluster

    All other default general configuration parameters were accepted.

    The Server Start configuration parameters entered were:


    Java Home: /Library/Java/Home - the base dir of your Java installation
    Java Vendor: Apple
    BEA Home: /Users/Dave/apps/wls1035_dev/wlserver - the installation dir for WLS
    Root Directory: /Users/Dave/apps/mywls/user_projects/domains/base_domain - the domain base dir
    Class Path: /Users/Dave/apps/wls1035_dev/modules/features/weblogic.server.modules.coherence.server_10.3.4.1.jar:/Users/Dave/coherence/3.7/coherence-java/coherence/lib/coherence.jar - note the order of these Jar files seems to be important and you should specify the correct path separator, in this case a ':' char
    Arguments: -Xms1024m -Xmx1024m -Dtangosol.coherence.cacheconfig=/Users/Dave/Documents/workspace/WLSCohWebAppEAR/EarContent/APP-INF/classes/coherence-cache-config.xml -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.role=CacheServer - the arguments to the JVM that will run the Coherence cache server. Note: the last parameter as in the Coherence override file the default storage setting has been set to false, so for storage enabled nodes it has to be explicitly enable. This is because its not possible to override the storage enabled default setting by using system properties just for the TestServlet other than setting this property in the WLS startup script.

    Obviously you will need to modify these paths to reflect thoseÊ in your environment.

    To create additional Coherence Servers you can just clone this one on the Coherence Servers admin page and change the name of the new Coherence Server.


    The Coherence Cluster is started from the Coherence Servers page which you can navigate to by selecting the Environment > Coherence Servers menu. On this page select the Control tab, select all the Coherence Servers that you have just defined and click on the StartÊ button.

    Start Cohernce Servers
  4. Deploy the Coherence and ActiveCache shared libraries

    The Coherence and ActiveCache Jar files can be deployed in a number of ways, added to the CLASSPATH of WLS, deployed as a shared library or as part of the web application. In this case the Jar files have been deployed as a shared library, which provides both usage isolation (as only applications that import the libraries will have them in scope) and a minimal resource overhead (as only 1 copy of the classes will be loaded). Shared libraries can be deployed through the WLS Admin Console via the Environment > Deployments menu, as shown below (accepting the default settings should be fine to get started):

    Deploy WLS Shared Libraty

  5. Package and deploy the web application that will utilize a Coherence cache

    This can be done as either a WAR or EAR file. I choose an EAR file here. You can do this either by dropping the archive into the autodeploy directory of WLS, through the Admin console, using the Weblogic Scripting Tool (WLST) or through and IDE like Eclipse (that uses WLST). I choose the later for convenience. An easy way to do so is through the wizards provided by the Oracle Enterprise Pack for Eclipse (OEPE) plugin - as shown below:

    Deploy EAR to WLS from Eclipse

    To do this you can just import the example EAR web application projects (and its dependent projects),Ê update the library paths for Coherence etc and deploy to the target WLS.

Testing of the sample web application

To access the TestServlet sample web application go to the URL http://<your hostname>:7001//WLSCohWebApp/TestServlet , for instance http://192.168.56.1:7001/WLSCohWebApp/TestServlet.. It outputs the Coherence cluster members and also puts an entry in a cache called "MyCache" whose key is the HTTP session Id and the value is the last modified value of the HTTP session. Its output is shown below:

Web app output

This output tells you how many clients have an active session and how many cluster members are in the cluster. All this cache data is visible across all invocations of this Servlet, so application data like reference data or values in drop-down-lists etc, that doesn't change frequently are ideal candidates.

Description of the sample web application

The ActiveCache functionality allows cache references to be configured as resources. In the TestServlet the Resournce annotation is used to dynamically inject a reference to the NamedCache "MyCache", as shown below:

mycache resource ref

A Coherence cache reference (NamedCache) resource and also be looked up using a JNDI lookup.

The shared libraries that the Servlet references are specified in the weblogic-application.xml file:

Shared libs config

The Coherence cluster can be monitored through JMX. To see the metrics for the cluster open JConsole on the machine you are testing on and select one of the weblogic.server.nodemanager... process (a Coherence management server could also be started to provide remote access). If you expand the node tree you will see the following metrics:

JConsole view

For explicit startup and shutdown operations a Servlet listener class has been registered that will shutdown the web application as a cluster node if the application is un-deployed.

I hope this post has been useful and gives you an overview of how the WLS ActiveCache functionality can be used in a web application to provide fast access to frequently used data. Although the cache is used as a "side cache" here - where the data is added to the cache explicitly - it could easily be configured to load the required data into the cache on-demand from a database or some other data source. Other options not explored here, but worth considering, are scripting the cluster setup and startup using WLST.

The sample application can be downloaded from here.

Other useful resources

Tuesday Aug 03, 2010

Coherence Rolling Upgrades

Oracle Coherence Rolling Upgrades

Oracle Coherence provides a highly available resilient in-memory cache that provides fast access to frequently used data. It is increasingly becoming a key component of systems that need to be online 24x7 with little or no opportunity to make changes offline. This document outlines a number of techniques that can be used to upgrade a Coherence cluster online without the need to stop the consumers of its data.

How can Coherence be upgraded online?

Typically this is done in a rolling upgrade. This is where each node or JVM in a cluster is stopped, its configuration or CLASSPATH changed and then it is re-started. By cycling through all nodes or JVM's in the cluster they can all be migrated to a new version of the Coherence binaries, a new configuration or version of custom classes.

Although Coherence has been designed to handle nodes or JVM's dynamically leaving and re-joining a cluster, the internal data recovery and re-balancing that Coherence performs can take a few seconds. Coherence can usually detect the failure of a node or JVM within a second, but recovering and re-balancing the data can take a few seconds – depending on the amount of data it held, the speed of the clusters network and how busy the cluster is.

So a rolling re-start must be managed. Every Coherence distributed service exposes a JMX attribute called StatusHA. It indicates the status of the data managed by that service, i.e. the data in the caches managed by the service. It can have 3 values. MACHINE-SAFE, when data in caches managed by the service has been copied to more than one machine, meaning no data will be lost if a machine fails. NODE-SAFE, when multiple copies of data exist on more than one node or JVM, and finally ENDANGERED, when some of the data it manages is only held on one node or JVM.

In a rolling re-start of a cluster another node cannot be re-started if the StatusHA of any distributed service is ENDANGERED or data may be lost if the next node or JVM to be re-started contains the only copy of a piece of un-recoverd data.

Some changes can be also be made dynamically, with no node or JVM re-start, for instance JMX changes. However, it should be noted that these will not be persisted and so will not survive node re-starts. The following change can be made dynamically through JMX (as of release 3.5.2):

Cluster Node Settings

JMX Parameter

Setting

Comments

BufferPublishSize

Integer

The buffer size of the unicast datagram socket used by the Publisher, measured in the number of packets. Changing this value at runtime is an inherently unsafe operation that will pause all network communications and may result in the termination of all cluster services.

BufferReceiveSize

Integer

The buffer size of the unicast datagram socket used by the Receiver, measured in the number of packets. Changing this value at runtime is an inherently unsafe operation that will pause all network communications and may result in the termination of all cluster services.

BurstCount

Integer

The maximum number of packets to send without pausing. Anything less than one (e.g. zero) means no limit.

BurstDelay

Integer

The number of milliseconds to pause between bursts. Anything less than one (e.g. zero) is treated as one millisecond.

LoggingFormat

String

Specifies how messages will be formatted before being passed to the log destination

LoggingLevel

Integer

Specifies which logged messages will be output to the log destination. Valid values are non-negative integers or -1 to disable all logger output.

LoggingLimit

Integer

The maximum number of characters that the logger daemon will process from the message queue before discarding all remaining messages in the queue. Valid values are integers in the range [0...]. Zero implies no limit.

MulticastThreshold

Integer

The percentage (0 to 100) of the servers in the cluster that a packet will be sent to, above which the packet will be multicasted and below which it will be unicasted.

ResendDelay

Integer

The minimum number of milliseconds that a packet will remain queued in the Publisher`s re-send queue before it is resent to the recipient(s) if the packet has not been acknowledged. Setting this value too low can overflow the network with unnecessary repetitions. Setting the value too high can increase the overall latency by delaying the re-sends of dropped packets. Additionally, change of this value may need to be accompanied by a change in SendAckDelay value.

SendAckDelay

Integer

The minimum number of milliseconds between the queueing of an Ack packet and the sending of the same. This value should be not more then a half of the ResendDelay value.

TrafficJamCount

Integer

The maximum total number of packets in the send and resend queues that forces the publisher to pause client threads. Zero means no limit.

TrafficJamDelay

Integer

The number of milliseconds to pause client threads when a traffic jam condition has been reached. Anything less than one (e.g. zero) is treated as one millisecond.

Point-to-point Settings

JMX Parameter

Setting

Comments

ViewedMemberId

Integer

The Id of the member being viewed.

Service Settings

JMX Parameter

Setting

Comments

RequestTimeoutMillis

Long

The default timeout value in milliseconds for requests that can be timed-out (e.g. implement the com.tangosol.net.PriorityTask interface), but do not explicitly specify the request timeout value.

TaskHungThresholdMillis

Long

The amount of time in milliseconds that a task can execute before it is considered hung. Note that a posted task that has not yet started is never considered as hung.

TaskTimeoutMillis

Long

The default timeout value in milliseconds for tasks that can be timed-out (e.g. implement the com.tangosol.net.PriorityTask interface), but do not explicitly specify the task execution timeout value.

ThreadCount

Integer

The number of threads in the service thread pool.

Cache Settings

JMX Parameter

Setting

Comments

BatchFactor

Double

The BatchFactor attribute is used to calculate the `soft-ripe` time for write-behind queue entries. A queue entry is considered to be `ripe` for a write operation if it has been in the write-behind queue for no less than the QueueDelay interval. The `soft-ripe` time is the point in time prior to the actual `ripe` time after which an entry will be included in a batched asynchronous write operation to the CacheStore (along with all other `ripe` and `soft-ripe` entries). This attribute is only applicable if asynchronous writes are enabled (i.e. the value of the QueueDelay attribute is greater than zero) and the CacheStore implements the storeAll() method. The value of the element is expressed as a percentage of the QueueDelay interval. Valid values are doubles in the interval [0.0, 1.0].

ExpiryDelay

Integer

The time-to-live for cache entries in milliseconds. Value of zero indicates that the automatic expiry is disabled. Change of this attribute will not affect already-scheduled expiry of existing entries.

FlushDelay

Integer

The number of milliseconds between cache flushes. Value of zero indicates that the cache will never flush.

HighUnits

Integer

The limit of the cache size measured in units. The cache will prune itself automatically once it reaches its maximum unit level. This is often referred to as the `high water mark` of the cache.

LowUnits

Integer

The number of units to which the cache will shrink when it prunes. This is often referred to as a `low water mark` of the cache.

QueueDelay

Integer

The number of seconds that an entry added to a write-behind queue will sit in the queue before being stored via a CacheStore. Applicable only for WRITE-BEHIND persistence type.

RefreshFactor

Double

The RefreshFactor attribute is used to calculate the `soft-expiration` time for cache entries. Soft-expiration is the point in time prior to the actual expiration after which any access request for an entry will schedule an asynchronous load request for the entry. This attribute is only applicable for a ReadWriteBackingMap which has an internal LocalCache with scheduled automatic expiration. The value of this element is expressed as a percentage of the internal LocalCache expiration interval. Valid values are doubles in the interval[0.0, 1.0]. If zero, refresh-ahead scheduling will be disabled.

RequeueThreshold

Integer

The maximum size of the write-behind queue for which failed CacheStore write operations are requeued. If zero, the write-behind requeueing will be disabled. Applicable only for WRITE-BEHIND persistence type.

Management Settings

JMX Parameter

Setting

Comments

ExpiryDelay

Long

The number of milliseconds that the MBeanServer will keep a remote model snapshot before refreshing.

RefreshPolicy

String

The policy used to determine the behavior when refreshing remote models. Valid values are: refresh-ahead, refresh-behind, refresh-expired, refresh-onquery. Invalid values will convert to `refresh-expired`.

Others may need a parallel cluster to be available so that clients can be failed over to it and failed back when the original cluster has been completely shutdown and re-started.

The process for upgrading an application that uses Coherence will start at the cache store (usually a database) level and then at subsequent dependent application tiers, cluster level and client level, until everything has been upgraded.

This document will outline what types of changes can be made to Coherence online and how they can be made.

What types of online changes can be made to a Coherence cluster?

The Coherence binaries/JAR files

Minor upgrades to the Coherence binaries can be done online in a rolling fashion. This also applies to patches to the Coherence binaries. So a patch release upgrade from release 3.6.1.2 to 3.6.1.6 (Major.Major.Pack:Patch) can be done online. For major and pack release upgrades a complete cluster shutdown will be required. This is because a major or pack release will likely not be compatible on the TCMP protocol boundary (there are plans to document these).

To perform a major upgrade to the Coherence binaries, e.g. from 3.4 to 3.5, a parallel and identical cluster must be available. Usually this will be a Disaster Recovery (DR) site or another cluster setup in an active-active configuration using the Push Replication feature so that the data on each is synchronised.

For Extend Clients the procedure will first require clients using the primary site to be re-directed to the DR or another active site. Any data that has not yet been replicated can be pulled through from the primary site if necessary. Once all the active clients have been re-directed and changes on the primary site fully replicated it can then be shut-down, the major release of the binaries placed in the CLASSPATH and the cluster re-started. Finally, it can be re-synchronized with the other active-active site and clients re-directed back.

For data clients, i.e. storage disabled TCMP clients, it is not possible to re-direct them to another cluster- as described above.

Extend clients

Forward compatibility is from client to server, i.e. from Extend Client to the cluster. For example a 3.4 client can connect to a 3.5 and 3.6 proxy. However a 3.6 client cannot connect to a 3.4 proxy. In addition, this requires the use of POF all around, since POF can support backward compatible serialization change.

It should be noted that during an online upgrade new clients must not access a Coherence cluster until all the nodes or JVM's have been upgraded first.

Coherence Configuration Changes

The cache configuration can be made either in the XML configuration file or through JMX. JMX enables a limited set of changes to be made online, such as changing the logging level (see above for the full list of online changes that can be made). These are made at a node or JVM level. Management tools, like Oracle Enterprise Manager, can JMX changes to be propagated to all nodes in a cluster, but they are not persisted. Where JMX does not expose the configuration setting as an attribute that can be modified then it must be made to the cache or override configuration file and the cluster re-started in a rolling fashion. It should be noted that not all changes can be made in a rolling fashion, for instance changes to the cluster ports and addresses.

Portable Object Format (POF) changes must be synchronised with any CLASSPATH changes on both the cluster and clients using the cluster. Changes to the POF configuration can be made in a rolling fashion, but classes listed must also be added to the CLASSPATH at the same time. Furthermore, the whole cluster must be upgraded first before clients using any new classes access the cluster. Changes must also be both backwardly and forwardly compatible. So for instance a POF entry can be removed if it is for a class which represents a parameter to an entry processor which will no longer be used by any client. If older clients are still active though it must be retained.

Database Schema Changes

Many Coherence clusters use a database to persist cache data. For the most part database changes must be “additive”, i.e. a new table column may be added but one cannot be dropped while it still contains data. Some databases do provide sophisticated features to enable more dynamic changes to be made online, but they will not be discussed here. Nor will upgrading alternative cache store technologies.




Database changes must be made before any cache data can be modified, so that any dependencies new cache classes will have on an underlying database will be in place when they are used.

Changing Cache Data

Here we are going to focus on how cache data classes can be changed online, rather than code that implements custom eviction policies, entry processors etc. These types of classes usually have no dependency on a database and can simply be made to the CLASSPATH and POF configuration (if necessary) and the cluster re-started in a rolling fashion.

To enable cache classes to be upgraded online they must be able to contain different versions of the same data. The Coherence Evolvable interface facilitates this. The Evolvable interface is implemented by classes that require forwards and backward compatibility of their serialized form. Although evolvable classes are able to support the evolution of classes by the addition of data, changes that involve the removal of data or modifications cannot be safely accomplished as long as a previous version of the class exists that relies on that data.

When an Evolvable object is de-serialized, it retains any unknown data that has been added to newer versions of the class, and the version identifier for that data format. When the Evolvable object is subsequently serialized, it includes both that version identifier and the unknown future data.

For instance in the rolling upgrade example version 1 of the class DummyObject looks like this:




The implementation of the Evolvable interface looks like this:




Version 2 of DummyObject has a new field, newStr:


In the readExternal() method there is a check to see what version of the object it is. This is because if a version 1 stream of DummyObject were written to the dummy cache and it was then read by a version 2 client there would be no newStr field value as the object data has been created from a version 1 not version 2 client.


Changing the Environment

Changing a Coherence environment, like modifying the system properties for a JVM or even changing the physical resources of each server can also be performed in a rolling fashion. If cluster resources are modified then ensure that they are still sufficient and balanced after the change.

Rolling Upgrade Script

Below is an example script to perform a rolling upgrade of a Coherence cluster to introduce a new version of a cache client while an existing version of a client is still running.

Upgrade database schema

Update Coherence nodes CLASSPATH

For each server

For each JVM

While JMX value of StatusHA not MACHINE_SAFE

for all nodes for all services

Pause for 5s

End while

Stop / Start JVM

End For

End for

Introduce new version of client

Potential Issues

Online upgrades of a Coherence cluster are not always necessary. Many systems using Coherence have maintenance windows where changes can be made off-line or are not online 24x7. Where this is the case and the cache data is persisted an alternative data store, e.g. a database, an offline upgrade will be much simpler.

When considering an online upgrade of Coherence you should:

  • Script the upgrade process and thoroughly test it beforehand.

  • Ensure that the overhead of a rolling upgrade on your cluster is acceptable. When a cluster contains a lot of data you are effectivly going to move all the data around – twice!. This can significantly degrade cluster performance if the cluster is heavily loaded with normal activity.

Rolling Upgrade Example

To accompany this document there is an example application with scripts to upgrade it online. The application consists of a simple class, DummyObject, that is used to hold cache values, containing an int and a String attribute

The example has been tested against Coherence 3.5.3 for Java and a MySQL database. The scripts are simple bash scripts that can be run standalone on a laptop. The example is packaged as an Eclipse Galileo project, but the classes have been pre-compiles so an the Eclipse IDE is not essential.

The sequence for running the example is:

  • Download the necessary drivers for MySQL

  • Setup. Change the paths, IP addresses and ports in the scriipts/env.sh file to reflect those of your system.

  • Use the script create.sql to create the base tables in MySQL

  • Start the cache servers. Change to the scripts directory and run the ./start.sh script. This will start 3 cache servers and 1 JMX node in the background.

  • Open another terminal and in the scripts directory run the ./client1.sh to start version 1 of the client. This will run version 1 of the client that will connect to the cluster, put some objects in a cache called dummy and then continually update the string attribute to “dummy 1”

  • Update the database schema by running the alter-table.sql script in the sql directory to add an additional column to the Dummy table.

  • Copy the new DummyObject (version 2) JAR file over the old one (version 1) – that adds a new field (newString) to the DummyObject class.

  • In a new terminal go to the scripts directory and run the script ./restart.sh to perform a rolling cluster re-start. If will return and its output is written to the file logs/rserver.log...

  • Open another terminal and in the scripts directory run the ./client2.sh script. This will run a version 2 client that will update version 2 of the DummyObject values in the dummy cache. It will update the string attribute to “Dummy 2” and also print out the value of the new string attribute – which will always be “”

Summary

Many applications using Coherence have maintenance windows or are not required to run continuously, 24x7. However, some are. For these, the dynamic features of Coherence can be used to make online changes using rolling upgrades and side-by-side clusters. Even where downtimes are available these features allow for critical changes to be made when needed. These techniques complement the existing high availability features of Coherence and make it a truly unique data caching platform.

Examples

There are 2 example Eclipse projects to support this document, which allow you to try out a simple rolling upgrade. The examples can be downloaded here.

Tuesday Jun 15, 2010

A Simple Coherence C++ Client

[Read More]

Monday Nov 23, 2009

Caching PHP HTTP Sessions in Coherence

[Read More]

Thursday Oct 15, 2009

Look, no Java!

 

With release 3.5 of Oracle Coherence it is now possible to query, aggregate and modify serialized POF (Portable Object Format) values stored in a cache natively, that is without writing a Java representation of the object. So  .NET and C++ developers can just write C# etc. and C++.

Note: With release 3.7.1 there is now no requirement to create corresponding Java classes if you choose to use data affinity and annotations can be be used with Java, .NET and C++ clients instead of explicitly adding POF serialisation code. See the end of this article for further details.

This is achieved with the introduction of POF extractors and POF updaters, that fetch and update native POF objects without de-serializing the value to Java. As well as allowing .NET and C++ developers to just work in the language they are most comfortable and productive in, this new feature also has a number of other benefits:

  • It dramatically improves performance, as no de-serialization needs to take place, so no new objects need to be created – or garbage collected.
  • Less memory is required as a result.
  • The development and deployment process is simpler, as no corresponding Java classes need to be created, managed and deployed.

However, there are some occasions where you do still need to create complementary Java objects to match your .NET or C++ objects. These are:

  • When you want to use Key Association, as Coherence will always de-serializes keys to determine whether they implement KeyAssociation.
  • If you use a Cache Stores - Coherence passes the de-serialized version of the key and value to the cache store to write to the back end so that ORM (Object Relational Mapping) tools, like Hibernate and EclipseLink (JPA) have access to the Java objects.

To update a value in a cache from C# you would use code similar to that shown below:

    // Parameters: key, (extractor, new value)

    cache.Invoke(0, new UpdaterProcessor(new PofUpdater(BetSerializer.ODDS), 1)); 

 

 

Here a cache value identified by the key ‘0’ is being updated using an UpdateProcessor. The ValueUpdator being used is a C# PofUpdater which will access the attribute at offset BetSerializer.ODD in the serialized POF value. Values held in a cache are held in a serialized format. When the POF serialization mechanism is used the values of an object are written and read from the POF stream in the same order. So here a constant  (BetSerializer.ODDS) is being used to provide a more readable representation for an offset like 3, i.e. the 3rd object value to be written and read from the POF stream.

 

 

For reading values from a cache in C# a similar approach is used, as shown below:

 

 

    // Query cache for all entries whose event name is 'FA Cup'

    EqualsFilter equalsFilter =

        new EqualsFilter(new PofExtractor(BetSerializer.EVENT_NAME), "FA Cup");

 

 

    Object[] results = cache.GetValues(equalsFilter);

    Console.WriteLine("Filter results:");

 

 



    for (int i = 0; i < results.Length; i++)

    {

        Console.WriteLine("Value = " + results[i]);

    }

 

 

Here all cache values that have an event name or “FA Cup” will be returned by the EqualsFilter. The C# PofExtractor is being used to extract the attribute at the offset BetSerializer.EVENT_NAME, which will be a number , like 4 indicating it was the 4th value to be written and read from the POF stream when the object it was part of was serialized.

 

 

This is a great new feature of Coherence and very easy to use. If you would like to see the full example the above extracts were taken from the you can download it from here. Happy coding - in C# and C++ ;).

Release 3.7.1 Update

There is now no longer any need to provide Java classes for classes used as keys in many circumstances.  Prior to 3.7.1, if you had a custom key class (for example a Name class that might have multiple strings like first and last name), then you needed to provide a corresponding Java class for that key class.  That is no longer the case. With Coherence 3.7.1, keys are not deserialized by the cluster. Now they don't even need to be deserialized for key assocation.  As of 3.7.1 key association checks are done at the Extend client (in this case .NET).  This is covered in the Coherence documentation in Oracle Coherence Release Notes, Release 3.7.1 under 1 Technical Changes and Enhancements -> Coherence*Extend Framework Enhancements and Fixes (here).

POF Annotations have also been added in release 3.7.1.  With POF Annotations, you can annotate classes which need to be POF serializable and no longer need to write serialization methods or separate serialization classes.  POF Annotations for .NET classes is covered in the Oracle Coherence Client Guide, Release 3.7.1 under 18.9 Using POF Annotations to Serialize Objects (here).

 

About

Views and ideas about Oracle Coherence and other software

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today