X

Proactive insights, news and tips from Oracle WebLogic Server Support. Learn Oracle from Oracle.

Build, test and deploy apps on Oracle Cloud. Start Now

Creating a WebLogic Domain with Coherence in Kubernetes

Paul Mackin
Software Development Director

Creating a WebLogic domain with a Coherence cluster and deploying it in Kubernetes, then managing the domain life cycle is a difficult task made easier with a combination of the WebLogic Kubernetes Operator and WebLogic Deployment Tooling (WDT). You don’t need to write any WebLogic Scripting Language (WLST) scripts or bash scripts to configure the domain with Coherence, or to activate lifecycle operations. All of this is done for you by the Operator and WDT. To build a domain image, you just provide a short, declarative domain configuration YAML file, then WDT will create and configure the domain for you. WDT will also deploy your artifacts into any domain target. After you have the image, it is easy to create the WebLogic domain in Kubernetes using the Operator. Furthermore, the Operator now supports rolling restart of Coherence managed servers to prevent the cluster from losing data during a domain restart. This article shows you exactly how to create the image, create the domain, and test a rolling restart.

Note: This article uses a Coherence cluster that runs as part of a WebLogic domain.  You can also use the Coherence Operator to create and manage  a stand-alone Coherence cluster in Kubernetes.  See https://github.com/oracle/coherence-operator for more information.

Before you start

You must complete the WebLogic Operator Quick Start Guide to prepare the environment for this sample. Most of the prerequisites are fulfilled in that guide.  

Note: This article uses Docker Desktop for Mac for the Docker and Kubernetes environment.

Prerequisites:

  • Complete QuickStart Guide
  • WebLogic Operator 2.3.0
  • WebLogic image at container-registry.oracle.com/middleware/weblogic:12.2.1.3-dev (tagged as oracle/weblogic:12.2.1.3-developer)
  • Kubernetes 1.11.5+, 1.12.3+, 1.13.0+, and 1.14.0+
  • Docker 18.03.1.ce+
  • Helm 2.8.2+
Building a WebLogic Domain Docker Image using WDT

Let’s get started by building the domain image using WDT. First, clone the Oracle Docker Image project as shown below.  This project has the scripts and files needed to build the image used for this exercise.

Prepare your environment to use WDT

Create a work directory and clone the Oracle Docker image git repository:

mkdir ~/coh-demo
cd ~/coh-demo
git clone https://github.com/oracle/docker-images.git
cd  ~/coh-demo/docker-images/OracleWebLogic/samples/12213-coherence-domain-in-image-wdt

The WDT tool was not included in the git repository you just cloned, so use the command below to get the weblogic-deploy.zip file from https://github.com/oracle/weblogic-deploy-tooling/releases.

Execute the following command to download WDT version 1.3:

curl -v  -Lo ./weblogic-deploy.zip https://github.com/oracle/weblogic-deploy-tooling/releases/download/weblogic-deploy-tooling-1.3.0/weblogic-deploy.zip

Inputs needed by WDT

Before running WDT, let's view the model YAML file that it needs. The YAML file specifies the domain configuration and the deployment artifacts. The YAML shown below also specifies a CoherenceClusterSystemResource, which is required for a Coherence cluster. Notice that the Coherence cluster is scoped to WebLogic cluster-1. This means that each WebLogic Server in cluster-1 will be a member of the Coherence cluster. This includes new servers that are created during scale out. The domain will be able to scale out to five managed servers, but you can control the actual number of servers that are started when you create the domain in Kubernetes, as you will see later. The appDeployments section specifies the applications being deployed to the domain, as described in the next section.  Following is the WDT model file, cohModel.yaml, you do not need to modify it:

domainInfo:
    AdminUserName: '@@FILE:/u01/oracle/properties/adminuser.properties@@'
    AdminPassword: '@@FILE:/u01/oracle/properties/adminpass.properties@@'
topology:
    AdminServerName: 'admin-server'
    ProductionModeEnabled: true
    Log:
        FileName: domain1.log
    NMProperties:
        JavaHome: /usr/java/jdk1.8.0_211
        LogFile: '@@DOMAIN_HOME@@/nodemanager/nodemanager.log'
        DomainsFile: '@@DOMAIN_HOME@@/nodemanager/nodemanager.domains'
        NodeManagerHome: '@@DOMAIN_HOME@@/nodemanager'
        weblogic.StartScriptName: startWebLogic.sh
    Cluster:
        'cluster-1':
            CoherenceClusterSystemResource: CoherenceCluster
            DynamicServers:
                ServerNamePrefix: 'managed-server-'
                MaxDynamicClusterSize: 5
                CalculatedListenPorts: false
                MaximumDynamicServerCount: 5
                ServerTemplate: 'cluster-1-template'
                DynamicClusterSize: 5
    Server:
        'admin-server':
            NetworkAccessPoint:
                T3Channel:
                    PublicPort: 30012
                    ListenPort: 30012
                    PublicAddress: kubernetes
    ServerTemplate:
        'cluster-1-template':
            ListenPort: 8001
            Cluster: 'cluster-1'
            JTAMigratableTarget:
                Cluster: 'cluster-1'
            SSL:
                ListenPort: 8100
resources:
    CoherenceClusterSystemResource:
        CoherenceCluster:
            CoherenceResource:
                CoherenceClusterParams:
                    ClusteringMode: unicast
                    ClusterListenPort: 7574
appDeployments:
    Application:
        'coh-proxy':
            SourcePath: 'wlsdeploy/applications/coh-proxy-server.gar'
            ModuleType: gar
            Target: 'cluster-1'

Now, you can build the WDT archive.zip file which includes the coh-proxy-server.gar to be deployed to WebLogic Server. The WDT archive can contain multiple artifacts, such as WAR and EAR files. The Coherence GAR file is the only artifact deployed here, as shown in in the YAML above, because we are using a Coherence proxy running in the domain. The application accessing the cache will be running on a development machine, so the GAR file has only the proxy configuration needed by Coherence. 

Build the archive as shown below:

./build-archive.sh

Build the image

At this point, everything is in place to build the image that will be used to create the WebLogic domain in Kubernetes. WDT will be executed inside of a Docker container to build the domain; it is never executed on your development machine. The Docker build file will copy the WDT zip file to the Docker container, unzip it, and then run WDT to build the domain and deploy the Coherence GAR file. After the docker build is done, you will have an image the contains both the WebLogic Server binary home and the domain home. Let’s run the exact Docker build command shown below.

Build the WebLogic  image:

docker build -f Dockerfile --no-cache  \
  --build-arg CUSTOM_DOMAIN_NAME=sample-domain1 \
  --build-arg WDT_MODEL=cohModel.yaml \
  --build-arg WDT_ARCHIVE=archive.zip \
  --build-arg WDT_VARIABLE=properties/docker-build/domain.properties  \
  --force-rm=true   \
   -t coherence-12213-domain-home-in-image-wdt .

Creating the domain in Kubernetes

You can now use the WebLogic Kubernetes Operator to create a domain, using your custom image.  The Operator’s main responsibility is lifecycle management of the domain, including the following:

  • Provisioning (domain creation)
  • Termination (domain deletion)
  • Scale out
  • Scale down
  • Rolling restart
  • Health check and pod auto-restart

If you don’t have an Operator environment set up, then follow the Quick Start instructions. The remaining steps in this article require that the Operator is deployed to Kubernetes and the account and secret used for the domain creation exist.

Preparing to create a domain

The Operator domain YAML file below was generated using the Quick Start instructions. We have made a few changes:

  • Image name is coherence-12213-domain-home-in-image-wdt:latest
  • JAVA_OPTIONS is "-Dtestval=1". This is just a placeholder for the subsequent rolling restart exercise.
  • timeoutSeconds: 300. This will be discussed in later in the article.

Create a file named domain.yaml with the following contents:

apiVersion: "weblogic.oracle/v4"
kind: Domain
metadata:
  name: sample-domain1
  namespace: sample-domain1-ns
  labels:
    weblogic.resourceVersion: domain-v2
    weblogic.domainUID: sample-domain1
spec:
  domainHome: /u01/oracle/user_projects/domains/sample-domain1
  domainHomeInImage: true
  image: "coherence-12213-domain-home-in-image-wdt:latest"
  imagePullPolicy: "Never"
  webLogicCredentialsSecret:
    name: sample-domain1-weblogic-credentials
  includeServerOutInPodLog: true
  serverStartPolicy: "IF_NEEDED"
  serverPod:
    shutdown:
      timeoutSeconds: 120
    env:
      - name: JAVA_OPTIONS
        value: "-Dtestval=1 "
      - name: USER_MEM_ARGS
        value: "-XX:+UseContainerSupport -Djava.security.egd=file:/dev/./urandom "
      - name: SHUTDOWN_TYPE_ARG
        value: "forced"
  adminServer:
    serverStartState: "RUNNING"
    adminService:
      channels:
        - channelName: default
          nodePort: 30701
  clusters:
    - clusterName: cluster-1
      serverStartState: "RUNNING"
      replicas: 2

Create the domain

Run kubectl apply to create the domain and Kubernetes will notify the Operator that a domain resource is being configured. Notice the kind: Domain field in the YAML file. This identifies the resource as a domain Kubernetes Custom Resource, which is managed by the Operator. The Operator will create the Kubernetes resources needed to provision the WebLogic domain, including a pod for the administration server and a pod for each managed server. The replicas: 2 field in the YAML file tells the Operator to create two managed servers. Let’s execute the following command to create the domain.

Create the WebLogic domain in Kubernetes:

kubectl apply -f ./domain.yaml

You can monitor the progress of the domain creation by checking the status of the pods. When the READY field is 1/1 and the STATUS is Running, then the WebLogic Server on that pod is ready. The Operator uses a Kubernetes readiness probe to determine when the pod is ready, and a liveness probe to determine if the server is healthy. See Kubernetes Probes for more information. It takes a few minutes to fully start the domain; then we can move to the next section.

Check the status of the domain pods:

kubectl get pod -n sample-domain1-ns
...
NAME                              READY   STATUS    RESTARTS   AGE
sample-domain1-admin-server       1/1     Running   0          5h22m
sample-domain1-managed-server-1   1/1     Running   0          5h20m
sample-domain1-managed-server-2   1/1     Running   0          5h18m

You may notice that you didn’t need to do any network configuration for Coherence. That’s because the WebLogic Managed Coherence Server automatically generates a Well Known Address (WKA) list and configures Coherence to use it along with UNICAST networking. Even when you scale out the cluster, the new server will automatically be configured with the WKA list.

Running the Coherence proxy client

At this point, you have a WebLogic domain deployed with a Coherence cluster running. The Coherence coh-proxy-server.gar file that was deployed configures Coherence to accept a proxy connection and provide access to a distributed service where a cache can be created. Because the service is configured with autostart=true, it is up and running and ready to be used. As mentioned previously, the cache service is restricted to the servers in WebLogic cluster-1.

Loading and validating cache

Build the Coherence proxy client program which boths load the cache and and verifies the contents.

cd coh-proxy-client
mvn package -DskipTests=true

Because we are running the application from a development machine outside the Kubernetes cluster, a service is required to provide access to the Coherence cluster. The kubectl expose command can be used for this.

Create a service to expose the Coherence proxy, which is listening on port 9000:

kubectl expose pod sample-domain1-managed-server-1 -n  sample-domain1-ns --port=9000 --target-port=9000 --type=LoadBalancer --name=coh-proxy-ms1

Now we are ready to access the Coherence cluster. Let’s load 10,000 entries, where each entry has a 1k value and an integer key.

Run the proxy client application to load the cache:

java -jar target/proxy-client-1.0.jar load
...
Loading cache
Cache size = 10000
SUCCESS Load Test - elapsed Time = 2 seconds

Next, we will validate the contents of the cache by reading every cache entry and checking the value.

Run the proxy client application to validate the cache:

java -jar target/proxy-client-1.0.jar validate
...
Cache size = 10000
SUCCESS Validate Test - elapsed Time = 3 seconds

More lifecycle operations

So far, we have created a domain image using WDT, provisioned a WebLogic domain in Kubernetes, and tested the Coherence cluster by loading and validating cache data. Now let’s see how easy it is to scale out and perform a rolling restart.

Scale-out the domain

Creating a new managed server is called scale out and removing a server is called scale down. To scale out or scale down, just modify the domain YAML file, change the replicas count, and then apply the configuration. The Operator will get notified and see that the desired replicas count is different than the current count, and then either add or remove a pod. For scale out, the Operator will create a new pod and the WebLogic managed server on that pod will be started. The new server will join the Coherence cluster and Coherence will move the primary and backup data partitions between cluster members to balance the cluster. All this happens automatically and seamlessly, with no service interruption. Let’s scale out the cluster now.

Modify domain.yaml and increase the replicas count to three:

...
replicas: 3

Apply the new configuration and get the pods:

kubectl apply -f ./domain.yaml
kubectl get pod -n sample-domain1-ns
...
NAME                              READY   STATUS    RESTARTS   AGE
sample-domain1-admin-server       1/1     Running   0          5h22m
sample-domain1-managed-server-1   1/1     Running   0          5h20m
sample-domain1-managed-server-2   1/1     Running   0          5h18m
sample-domain1-managed-server-3   0/1     Running   0          8s

You can see that sample-domain1-managed-server-3 was created and is starting up.

Domain rolling restart

The WebLogic Kubernetes Operator provides a rolling-restart feature that automatically recycles servers in the domain, one at a time. This is a critical feature in general, because it minimizes service interruptions. It is especially important when using Coherence, to prevent data loss. Coherence is typically configured to store backup data partitions on a different cluster member than the member that owns the primary data. That way, if a cluster member goes down, the data is still safe on the backup partition. However, if both the primary and backup partitions are lost at the same time, then you will lose the data. Furthermore, even if you are only shutting down a single server, you must wait until it is safe to do so. For example, all the cluster members could be running, but Coherence might be shuffling partitions around the cluster. Coherence provides an HAStatus MBean that will let you know if it is safe to shut down a server. If the status is ENDANGERED. then the server must not be shut down.

Operator shutdown script

The Operator includes a set of scripts that run in the pod and perform both startup and shutdown of the domain servers. If the domain has a Coherence cluster configured, then the shutdown script will check the HAStatus and wait until it is safe to shut down the server. This script is configured as a Kubernetes pre-stop hook and is invoked by Kubernetes during pod deletion. Kubernetes will wait a certain amount of time for the script to complete, then it will destroy the pod if that time expires. If the shutdown script returns before the timeout, then Kubernetes will proceed to delete the pod. This timeout is configured by the field, shutdown.timeoutSeconds in the Operator domain YAML file. If you have a Coherence cluster, you should set this timeout to a high value, like fifteen minutes, to provide ample time for the cluster to be safe in the worst-case scenario where the shutdown scripts cannot get the Coherence MBEAN. This is extremely unlikely to ever happen, and the long timeout value won’t affect performance if the MBean is accessible and the script can detect that Coherence is safe to shut down.

Let’s activate a rolling-restart by changing the JAVA_OPTIONS testval value to 2. After the new configuration is applied, the Operator will detect the change and begin a rolling restart.

Modify domain.yaml and change the testval system property to two:

...
value: "-Dtestval=2"

Now apply the YAML file to begin the rolling-restart:

kubectl apply -f ./domain.yaml

Checking the restart status

You can see in the three output sections below, that the rolling restart begins with the administration server, then moves on to managed-server-1. The pod is deleted (Terminating STATUS), then recreated (ContainerCreating STATUS). After the administration server is running again, one of the managed servers will be recycled. Note that the order of managed server recycling is indeterminate.

Get the pods:

kubectl get pod -n sample-domain1-ns

NAME                              READY   STATUS        RESTARTS   AGE
sample-domain1-admin-server       1/1     Terminating   0          5h33m
sample-domain1-managed-server-1   1/1     Running       0          5h30m
sample-domain1-managed-server-2   1/1     Running       0          5h28m
sample-domain1-managed-server-3   1/1     Running       0          6m5s

Get the pods:

kubectl get pod -n sample-domain1-ns

NAME                              READY   STATUS              RESTARTS   AGE
sample-domain1-admin-server       0/1     ContainerCreating   0          1s
sample-domain1-managed-server-1   1/1     Running             0          5h31m
sample-domain1-managed-server-2   1/1     Running             0          5h29m
sample-domain1-managed-server-3   1/1     Running             0          7m13s

Get the pods:

kubectl get pod -n sample-domain1-ns

NAME                              READY   STATUS        RESTARTS   AGE
sample-domain1-admin-server       1/1     Running       0          79s
sample-domain1-managed-server-1   1/1     Terminating   0          5h33m
sample-domain1-managed-server-2   1/1     Running       0          5h30m
sample-domain1-managed-server-3   1/1     Running       0          8m31s

As part of this exercise, we ran the cache validate client after the rolling restart to validate that no data was lost.

Summary

This wraps up our discussion of creating a WebLogic domain with Coherence in Kubernetes. The powerful features of both the WebLogic Operator and WebLogic Deployment Tooling show you how easy it is to create and deploy a complex domain. In addition, the Operator makes it very simple to do post-provisioning lifecycle operations. We hope this article is helpful and we look forward to your feedback. See the references below for more information.

References

 

 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.

Recent Content