Creating a WebLogic domain with a Coherence cluster and deploying it in Kubernetes, then managing the domain life cycle is a difficult task made easier with a combination of the WebLogic Kubernetes Operator and WebLogic Deployment Tooling (WDT). You don’t need to write any WebLogic Scripting Language (WLST) scripts or bash scripts to configure the domain with Coherence, or to activate lifecycle operations. All of this is done for you by the Operator and WDT. To build a domain image, you just provide a short, declarative domain configuration YAML file, then WDT will create and configure the domain for you. WDT will also deploy your artifacts into any domain target. After you have the image, it is easy to create the WebLogic domain in Kubernetes using the Operator. Furthermore, the Operator now supports rolling restart of Coherence managed servers to prevent the cluster from losing data during a domain restart. This article shows you exactly how to create the image, create the domain, and test a rolling restart.
Note: This article uses a Coherence cluster that runs as part of a WebLogic domain. You can also use the Coherence Operator to create and manage a stand-alone Coherence cluster in Kubernetes. See https://github.com/oracle/coherence-operator for more information.
You must complete the WebLogic Operator Quick Start Guide to prepare the environment for this sample. Most of the prerequisites are fulfilled in that guide.
Note: This article uses Docker Desktop for Mac for the Docker and Kubernetes environment.
Let’s get started by building the domain image using WDT. First, clone the Oracle Docker Image project as shown below. This project has the scripts and files needed to build the image used for this exercise.
Create a work directory and clone the Oracle Docker image git repository:
The WDT tool was not included in the git repository you just cloned, so use the command below to get the weblogic-deploy.zip file from https://github.com/oracle/weblogic-deploy-tooling/releases.
Execute the following command to download WDT version 1.3:
curl -v -Lo ./weblogic-deploy.zip https://github.com/oracle/weblogic-deploy-tooling/releases/download/weblogic-deploy-tooling-1.3.0/weblogic-deploy.zip
Before running WDT, let's view the model YAML file that it needs. The YAML file specifies the domain configuration and the deployment artifacts. The YAML shown below also specifies a CoherenceClusterSystemResource, which is required for a Coherence cluster. Notice that the Coherence cluster is scoped to WebLogic cluster-1. This means that each WebLogic Server in cluster-1 will be a member of the Coherence cluster. This includes new servers that are created during scale out. The domain will be able to scale out to five managed servers, but you can control the actual number of servers that are started when you create the domain in Kubernetes, as you will see later. The appDeployments section specifies the applications being deployed to the domain, as described in the next section. Following is the WDT model file, cohModel.yaml, you do not need to modify it:
domainInfo:
AdminUserName: '@@FILE:/u01/oracle/properties/adminuser.properties@@'
AdminPassword: '@@FILE:/u01/oracle/properties/adminpass.properties@@'
topology:
AdminServerName: 'admin-server'
ProductionModeEnabled: true
Log:
FileName: domain1.log
NMProperties:
JavaHome: /usr/java/jdk1.8.0_211
LogFile: '@@DOMAIN_HOME@@/nodemanager/nodemanager.log'
DomainsFile: '@@DOMAIN_HOME@@/nodemanager/nodemanager.domains'
NodeManagerHome: '@@DOMAIN_HOME@@/nodemanager'
weblogic.StartScriptName: startWebLogic.sh
Cluster:
'cluster-1':
CoherenceClusterSystemResource: CoherenceCluster
DynamicServers:
ServerNamePrefix: 'managed-server-'
MaxDynamicClusterSize: 5
CalculatedListenPorts: false
MaximumDynamicServerCount: 5
ServerTemplate: 'cluster-1-template'
DynamicClusterSize: 5
Server:
'admin-server':
NetworkAccessPoint:
T3Channel:
PublicPort: 30012
ListenPort: 30012
PublicAddress: kubernetes
ServerTemplate:
'cluster-1-template':
ListenPort: 8001
Cluster: 'cluster-1'
JTAMigratableTarget:
Cluster: 'cluster-1'
SSL:
ListenPort: 8100
resources:
CoherenceClusterSystemResource:
CoherenceCluster:
CoherenceResource:
CoherenceClusterParams:
ClusteringMode: unicast
ClusterListenPort: 7574
appDeployments:
Application:
'coh-proxy':
SourcePath: 'wlsdeploy/applications/coh-proxy-server.gar'
ModuleType: gar
Target: 'cluster-1'
Now, you can build the WDT archive.zip file which includes the coh-proxy-server.gar to be deployed to WebLogic Server. The WDT archive can contain multiple artifacts, such as WAR and EAR files. The Coherence GAR file is the only artifact deployed here, as shown in in the YAML above, because we are using a Coherence proxy running in the domain. The application accessing the cache will be running on a development machine, so the GAR file has only the proxy configuration needed by Coherence.
Build the archive as shown below:
./build-archive.sh
At this point, everything is in place to build the image that will be used to create the WebLogic domain in Kubernetes. WDT will be executed inside of a Docker container to build the domain; it is never executed on your development machine. The Docker build file will copy the WDT zip file to the Docker container, unzip it, and then run WDT to build the domain and deploy the Coherence GAR file. After the docker build is done, you will have an image the contains both the WebLogic Server binary home and the domain home. Let’s run the exact Docker build command shown below.
Build the WebLogic image:
docker build -f Dockerfile --no-cache \
--build-arg CUSTOM_DOMAIN_NAME=sample-domain1 \
--build-arg WDT_MODEL=cohModel.yaml \
--build-arg WDT_ARCHIVE=archive.zip \
--build-arg WDT_VARIABLE=properties/docker-build/domain.properties \
--force-rm=true \
-t coherence-12213-domain-home-in-image-wdt .
You can now use the WebLogic Kubernetes Operator to create a domain, using your custom image. The Operator’s main responsibility is lifecycle management of the domain, including the following:
If you don’t have an Operator environment set up, then follow the Quick Start instructions. The remaining steps in this article require that the Operator is deployed to Kubernetes and the account and secret used for the domain creation exist.
The Operator domain YAML file below was generated using the Quick Start instructions. We have made a few changes:
Create a file named domain.yaml with the following contents:
apiVersion: "weblogic.oracle/v4"
kind: Domain
metadata:
name: sample-domain1
namespace: sample-domain1-ns
labels:
weblogic.resourceVersion: domain-v2
weblogic.domainUID: sample-domain1
spec:
domainHome: /u01/oracle/user_projects/domains/sample-domain1
domainHomeInImage: true
image: "coherence-12213-domain-home-in-image-wdt:latest"
imagePullPolicy: "Never"
webLogicCredentialsSecret:
name: sample-domain1-weblogic-credentials
includeServerOutInPodLog: true
serverStartPolicy: "IF_NEEDED"
serverPod:
shutdown:
timeoutSeconds: 120
env:
- name: JAVA_OPTIONS
value: "-Dtestval=1 "
- name: USER_MEM_ARGS
value: "-XX:+UseContainerSupport -Djava.security.egd=file:/dev/./urandom "
- name: SHUTDOWN_TYPE_ARG
value: "forced"
adminServer:
serverStartState: "RUNNING"
adminService:
channels:
- channelName: default
nodePort: 30701
clusters:
- clusterName: cluster-1
serverStartState: "RUNNING"
replicas: 2
Run kubectl apply to create the domain and Kubernetes will notify the Operator that a domain resource is being configured. Notice the kind: Domain field in the YAML file. This identifies the resource as a domain Kubernetes Custom Resource, which is managed by the Operator. The Operator will create the Kubernetes resources needed to provision the WebLogic domain, including a pod for the administration server and a pod for each managed server. The replicas: 2 field in the YAML file tells the Operator to create two managed servers. Let’s execute the following command to create the domain.
Create the WebLogic domain in Kubernetes:
kubectl apply -f ./domain.yaml
You can monitor the progress of the domain creation by checking the status of the pods. When the READY field is 1/1 and the STATUS is Running, then the WebLogic Server on that pod is ready. The Operator uses a Kubernetes readiness probe to determine when the pod is ready, and a liveness probe to determine if the server is healthy. See Kubernetes Probes for more information. It takes a few minutes to fully start the domain; then we can move to the next section.
Check the status of the domain pods:
kubectl get pod -n sample-domain1-ns
...
NAME READY STATUS RESTARTS AGE
sample-domain1-admin-server 1/1 Running 0 5h22m
sample-domain1-managed-server-1 1/1 Running 0 5h20m
sample-domain1-managed-server-2 1/1 Running 0 5h18m
You may notice that you didn’t need to do any network configuration for Coherence. That’s because the WebLogic Managed Coherence Server automatically generates a Well Known Address (WKA) list and configures Coherence to use it along with UNICAST networking. Even when you scale out the cluster, the new server will automatically be configured with the WKA list.
At this point, you have a WebLogic domain deployed with a Coherence cluster running. The Coherence coh-proxy-server.gar file that was deployed configures Coherence to accept a proxy connection and provide access to a distributed service where a cache can be created. Because the service is configured with autostart=true, it is up and running and ready to be used. As mentioned previously, the cache service is restricted to the servers in WebLogic cluster-1.
Build the Coherence proxy client program which boths load the cache and and verifies the contents.
cd coh-proxy-client
mvn package -DskipTests=true
Because we are running the application from a development machine outside the Kubernetes cluster, a service is required to provide access to the Coherence cluster. The kubectl expose command can be used for this.
Create a service to expose the Coherence proxy, which is listening on port 9000:
kubectl expose pod sample-domain1-managed-server-1 -n sample-domain1-ns --port=9000 --target-port=9000 --type=LoadBalancer --name=coh-proxy-ms1
Now we are ready to access the Coherence cluster. Let’s load 10,000 entries, where each entry has a 1k value and an integer key.
Run the proxy client application to load the cache:
java -jar target/proxy-client-1.0.jar load
...
Loading cache
Cache size = 10000
SUCCESS Load Test - elapsed Time = 2 seconds
Next, we will validate the contents of the cache by reading every cache entry and checking the value.
Run the proxy client application to validate the cache:
java -jar target/proxy-client-1.0.jar validate
...
Cache size = 10000
SUCCESS Validate Test - elapsed Time = 3 seconds
So far, we have created a domain image using WDT, provisioned a WebLogic domain in Kubernetes, and tested the Coherence cluster by loading and validating cache data. Now let’s see how easy it is to scale out and perform a rolling restart.
Modify domain.yaml and increase the replicas count to three:
...
replicas: 3
Apply the new configuration and get the pods:
kubectl apply -f ./domain.yaml
kubectl get pod -n sample-domain1-ns
...
NAME READY STATUS RESTARTS AGE
sample-domain1-admin-server 1/1 Running 0 5h22m
sample-domain1-managed-server-1 1/1 Running 0 5h20m
sample-domain1-managed-server-2 1/1 Running 0 5h18m
sample-domain1-managed-server-3 0/1 Running 0 8s
You can see that sample-domain1-managed-server-3 was created and is starting up.
The WebLogic Kubernetes Operator provides a rolling-restart feature that automatically recycles servers in the domain, one at a time. This is a critical feature in general, because it minimizes service interruptions. It is especially important when using Coherence, to prevent data loss. Coherence is typically configured to store backup data partitions on a different cluster member than the member that owns the primary data. That way, if a cluster member goes down, the data is still safe on the backup partition. However, if both the primary and backup partitions are lost at the same time, then you will lose the data. Furthermore, even if you are only shutting down a single server, you must wait until it is safe to do so. For example, all the cluster members could be running, but Coherence might be shuffling partitions around the cluster. Coherence provides an HAStatus MBean that will let you know if it is safe to shut down a server. If the status is ENDANGERED. then the server must not be shut down.
The Operator includes a set of scripts that run in the pod and perform both startup and shutdown of the domain servers. If the domain has a Coherence cluster configured, then the shutdown script will check the HAStatus and wait until it is safe to shut down the server. This script is configured as a Kubernetes pre-stop hook and is invoked by Kubernetes during pod deletion. Kubernetes will wait a certain amount of time for the script to complete, then it will destroy the pod if that time expires. If the shutdown script returns before the timeout, then Kubernetes will proceed to delete the pod. This timeout is configured by the field, shutdown.timeoutSeconds in the Operator domain YAML file. If you have a Coherence cluster, you should set this timeout to a high value, like fifteen minutes, to provide ample time for the cluster to be safe in the worst-case scenario where the shutdown scripts cannot get the Coherence MBEAN. This is extremely unlikely to ever happen, and the long timeout value won’t affect performance if the MBean is accessible and the script can detect that Coherence is safe to shut down.
Let’s activate a rolling-restart by changing the JAVA_OPTIONS testval value to 2. After the new configuration is applied, the Operator will detect the change and begin a rolling restart.
Modify domain.yaml and change the testval system property to two:
...
value: "-Dtestval=2"
Now apply the YAML file to begin the rolling-restart:
kubectl apply -f ./domain.yaml
You can see in the three output sections below, that the rolling restart begins with the administration server, then moves on to managed-server-1. The pod is deleted (Terminating STATUS), then recreated (ContainerCreating STATUS). After the administration server is running again, one of the managed servers will be recycled. Note that the order of managed server recycling is indeterminate.
Get the pods:
kubectl get pod -n sample-domain1-ns
NAME READY STATUS RESTARTS AGE
sample-domain1-admin-server 1/1 Terminating 0 5h33m
sample-domain1-managed-server-1 1/1 Running 0 5h30m
sample-domain1-managed-server-2 1/1 Running 0 5h28m
sample-domain1-managed-server-3 1/1 Running 0 6m5s
Get the pods:
kubectl get pod -n sample-domain1-ns
NAME READY STATUS RESTARTS AGE
sample-domain1-admin-server 0/1 ContainerCreating 0 1s
sample-domain1-managed-server-1 1/1 Running 0 5h31m
sample-domain1-managed-server-2 1/1 Running 0 5h29m
sample-domain1-managed-server-3 1/1 Running 0 7m13s
Get the pods:
kubectl get pod -n sample-domain1-ns
NAME READY STATUS RESTARTS AGE
sample-domain1-admin-server 1/1 Running 0 79s
sample-domain1-managed-server-1 1/1 Terminating 0 5h33m
sample-domain1-managed-server-2 1/1 Running 0 5h30m
sample-domain1-managed-server-3 1/1 Running 0 8m31s
As part of this exercise, we ran the cache validate client after the rolling restart to validate that no data was lost.
This wraps up our discussion of creating a WebLogic domain with Coherence in Kubernetes. The powerful features of both the WebLogic Operator and WebLogic Deployment Tooling show you how easy it is to create and deploy a complex domain. In addition, the Operator makes it very simple to do post-provisioning lifecycle operations. We hope this article is helpful and we look forward to your feedback. See the references below for more information.
The way I was thinking that it should work is like this:
Step 1. Coherence cluster is deployed in domain D-coherence, and is used by all versions of application ApplicationX for session replication.
Step 2. ApplicationX-v1 is deployed to domain D-blue -> uses coherence cluster deployed in domain D-coherence
Step 3. ApplicationX-v2 is deployed to domain D-green -> uses coherence cluster deployed in domain D-coherence
This would allow us to easily test the new version v2 and rollback if necessary.
My questions are:
1. Can I deploy a coherence cluster in K8S using the weblogic operator?
2. Can I connect to a coherence cluster from a different domain?
3. Is it possible to have two apps, running in different domains, that share the same cache? We need this so that we don't loose sessions while we switch to the new version.