Break New Ground

Coherence Clusters on Docker Swarm

Java in memory data grids share a common feature in that they are typically made up of a number of JVMs that form a cluster and distribute data among the themselves. A common approach to cluster discovery when the JVMs start is to use multicast but this does not work inside Docker containers so IMDG products have to fall back on other discovery mechanisms, typically this is either a manually injected list of member addresses or a lookup to some other discovery mechanism.

Oracle Coherence is available on the Docker Store!

Running Oracle Coherence clusters on Docker Swarm has made cluster discovery much easier than it was in the past. Having to inject an address list at start time relies on knowing all or some of the cluster member container IP addresses before the cluster is started, which can be done with Docker as you can specify a fixed IP address for a container when it is run, but it is always more flexible to not have to care about address allocation.

Having to look up addresses from inside the cluster member once it has started relies on having some external data source holding the cluster IP addresses. There are applications available for Docker service discovery that can do this but it is another moving piece to add to your infrastructure and the simple an applications infrastructure is the less there is to go wrong and the easier life will be.

In a previous blog about Oracle Coherence on Docker I mentioned that to enable Coherence members to form a cluster across multiple Docker hosts then you either need to use host networking or overlay networking. Host networking that allows multicast is the simplest solution but not always available; certainly in modern cloud environments. This leaves us with overlay networks and the issues of service discovery that they bring. But, with the introduction of Swarm Mode in Docker forming clusters of containers using Oracle Coherence has become much easier.

Coherence Cluster Discovery

As we’ve already touched on, Coherence has a couple of ways for JVMs to discover the members of a cluster on start-up. The easiest is multicast but this is not always available and certainly isn’t inside Docker containers. The second way is called well-know-addressing where a list of one or more IP addresses is provided to a JVM and this is the method we need to use inside containers.

Well know addresses can be provided in two ways and is controlled by using the Coherence operational configuration file (typically called the overrides file).

The contents of the well-known-addresses element depends on how the addresses are going to be provided; there are two choices.
First is a custom address provider, which is a custom class that implements com.tangosol.net.AddressProvider and typically retrieves the address list from some third party source and returns them as required. For example a project I was am involved with is using Marathon/Mesos for container management and this project uses a custom address provider that queries the Marathon API to obtain the IP addresses of the containers in the Coherence cluster. This address provider would be configured a bit like this:

The other option for providing a well known address list is to just put the addresses directly into the overrides file.

In fact, the out of the box the WKA configuration in Oracle Coherence is set to allow a single address to be passed in via a System property; the configuration looks like this:

This means that you can set a single WKA value at runtime and that value will be used to locate members of the cluster, so the first cluster member needs to be started on that server; for example:

$ java -cp ${COHERENCE_HOME}/lib/coherence.jar \
  -Dcoherence.wka=foo.oracle.com \


In this case each Coherence member will look for a member on foo.oracle.com to use to discover the remaining cluster members.
This means that without adding any configuration to Coherence we can start an out of the box cluster and pass in a single address and the Coherence members on the server with that address will be contacted by new JVM cluster members when they start in order to discover the other members. This is all fine except when there is an issue with that server; for example, if it has crashed you cannot start any new cluster members as the WKA list will point to an invalid IP address so to work around this you would typically provide more than one WKA address in the configuration file.

Coherence Clusters on Docker Swarm

With the introduction of Swarm Mode and services in Docker one of the features that goes with this is to be able to use DNS from inside a container to lookup the members of a service. One of the features of how Oracle Coherence uses the WKA names is that if those names resolve to multiple addresses via DNS then all of those addresses are used by Coherence for the WKA list. So in the case above where we can only specify a single host name in the default Coherence configuration this is no longer such a limitation if we can use DNS to resolve that host name to multiple servers. This is exactly what Docker Swarm Mode allows us to do for the containers that make up a service. If we start the Coherence cluster as a service in Docker we can use a special host name to look up all of the IP addresses of the containers that make up that service. The Docker documentation explains how Swarm Mode Networking works.

The important point for Oracle Coherence from that link is that every Service has a DNS name that can be used to look up the IP addresses. If we repeat the example from the Docker documentation, we can see how easy it is.

First create an overlay network for the service to use.

$ docker network create \
  --driver overlay \
  --subnet \


Now we will start a simple service with a basic container.

$ docker service create \
  --name my-busybox \
  --network my-network \
  busybox \
  sleep 3000


If we list out the services we should see our my-busybox service running:

$ docker service ls
ID            NAME        MODE        REPLICAS  IMAGE
tpsqpgwh423c  my-busybox  replicated  1/1       busy box


We can now scale up the service to let’s say three containers:

$ docker service scale my-busybox=3
my-busybox scaled to 3


If we now list out the containers that are running we should see the containers that are part of the my-busybox service:

$ docker ps --format "{{.ID}}: {{.Names}}"
443d3ea90e62: my-busybox.2.mx5nq93n8w7z80ndfkqd1zmhj
81893dfcdaf8: my-busybox.3.nd4lp1z7iliys8wsplusxkq0a
19d4fdded3dd: my-busybox.1.xewffdq29h7pomx8jhw9xib14


Now let’s run nslookup in one of the containers and lookup the tasks.my-busybox address

You will see that we have three addresses back, one for each of our containers. We can now use the exact same concept to create an Oracle Coherence cluster using nothing but the out of the box Coherence Docker image available on the Docker Store. By the way, this image is built on top of Oracle Linux and the official Oracle server JRE image. The shell script in the image that runs as the container entry point will by default start a storage enabled instance of com.tangosol.net.DefaultCacheServer. The shell script also allows various parameters to be passed in by setting environment variables, one of these is COH_WKA that sets the well known address value.

$ docker pull store/oracle/coherence:
$ docker service create  --name coh-cluster \
  --network my-network \  
  -e COH_WKA=tasks.coh-cluster \


The coh-cluster service should now be running with a single member, as we did not specify a size when we started the service; and running docker ps will show the single container.

$ docker ps --format "{{.ID}}: {{.Names}}"
c4917b7c9b9b: coh-cluster.1.tpz4kby4nwpxi0n89herpfsyp


If we look at the logs for the container we should see that there is a new cluster of one member.

$ docker logs coh-cluster.1.tpz4kby4nwpxi0n89herpfsyp

The logs will show something like this:

We can see that we have a WKA list with a single member as the host name tasks.coh-service only resolved to the single container.

Now let’s scale the service up to more members:

$ docker service scale coh-cluster=3
coh-cluster scaled to 3

...and we should have three containers:

$ docker ps --format "{{.ID}}: {{.Names}}"
50db2c5b76ed: coh-cluster.2.r79i1q03q0umup2b8geovc6m0
8f4159035dd1: coh-cluster.3.i8wdao099zbwa5tdeslx9y2bz
c4917b7c9b9b: coh-cluster.1.tpz4kby4nwpxi0n89herpfsyp

If we look at the logs of the third container:

$ docker logs coh-cluster.3.i8wdao099zbwa5tdeslx9y2bz

...we now have three members in the cluster.

...and we can see that the WKA list has all three addresses in it. So we can use Swam to scale up and down the Coherence cluster and thanks to Swam's built in DNS Coherence WKA will just work. There are more details on running the Oracle Coherence image with various settings and options in the setup instruction on the Docker Store

You can also try Coherence and Docker on Oracle Cloud Platform, using the Container Cloud Service. Experiment by going to cloud.oracle.com/tryit, get an account and $300 in free credits.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.