As part of certifying WebLogic Server on Kubernetes, we have identified best practices for sharing file data among WebLogic Server pods that are running in a Kubernetes environment. In this blog, I review the WebLogic Server services and files that are typically configured to leverage shared storage, and I provide full end-to-end samples, which you can download and run, that show mounting shared storage for a WebLogic domain that is orchestrated by Kubernetes.

WebLogic Server Persistence in Volumes

When running WebLogic Server on Kubernetes, refer to the blog Docker Volumes in WebLogic for information about the advantages of using data volumes. This blog also identifies the WebLogic Server artifacts that are good candidates for being persisted in those data volumes.

Kubernetes Solutions

In a Kubernetes environment, pods are ephemeral. To persist data, Kubernetes provides the Volume abstraction, and the PersistentVolume (PV) and PersistentVolumeClaim (PVC) API resources.

Based on the official Kubernetes definitions [Kubernetes Volumes and Kubernetes Persistent Volumes and Claims], a PV is a piece of storage in the cluster that has been provisioned by an administrator, and a PVC is a request for storage by a user. Therefore, PVs and PVCs are independent entities outside of pods. They can be easily referenced by a pod for file persistence and file sharing among pods inside a Kubernetes cluster.

When running WebLogic Server on Kubernetes, using PVs and PVCs to handle shared storage is recommended for the following reasons:

  • Usually WebLogic Server instances run in pods on multiple nodes that require access to a shared PV. The life cycle of a WebLogic Server instance is not limited to a single pod.
  • PVs and PVCs can provide more control. For example, the ability to specify: access modes for concurrent read/write management, mount options provided by volume plugins, storage capacity requirements, reclaim policies for resources, and more.

Use Cases of Kubernetes Volumes for WebLogic Server

To see the details about the samples, or to run them locally, please download the examples and follow the steps provided below.

Software Versions

Host machine: Oracle Linux 7u3 UEK4 (x86-64)

Kubernetes v1.7.8

Docker 17.03 CE

Prepare Dependencies

  1. Build the oracle/weblogic:12.2.1.3-developer image locally based on the Dockerfile and scripts at https://github.com/oracle/docker-images/tree/master/OracleWebLogic/dockerfiles/12.2.1.3/.
  2. Download the WebLogic Kubernets domain sample source code from https://github.com/oracle/docker-images/tree/master/OracleWebLogic/samples/wls-k8s-domain.

    Put the sample source code to a local folder named wls-k8s-domain.

  3. Build the WebLogic domain image locally based on the Dockerfile and scripts. 
    $ cd wls-k8s-domain
$ docker build -t was-k8s-domain .
  4. For Use Case 2, below, prepare a NFS server and a shared directory by entering the following commands (in this example I use machine 10.232.128.232). Note that Use Case 1 uses a host path instead of NFS and does not require this step. 
    # systemctl start rpcbind.service
# systemctl start nfs.service
# systemctl start nfslock.service
$ mkdir -p /scratch/nfsdata
$ chmod o+rw /scratch/nfadata
# echo /scratch/nfsdata *(rw,fsid=root,no_root_squash,no_subtree_check) >> /etc/exports

    By default, in the WebLogic domain wls-k8s-domain, all processes in pods that contain WebLogic Server instances run with user ID 1000 and group ID 1000. Proper permissions need to be set to the external NFS shared directory to make sure that user ID 1000 and group ID 1000 have read and write permission to the NFS volume. To simplify the permissions management in the examples, we grant read and write permission to others to the shared directory as well.

Use Case 1: Host Path Mapping at Individual Machine with a Kubernetes Volume

The WebLogic domain consists of an Administration Server and multiple Managed Servers, each running inside its own pod. All pods have volumes directly mounted to a folder on the physical machine. The domain home is created in a shared folder when the Administration Server pod is first started.

At runtime, all WebLogic Server instances, including the Administration Server, share the same domain home directory via a mounted volume.

Note: This example runs on a single machine, or node, but this approach also works when running the WebLogic domain across multiple machines. When running on multiple machines, each WebLogic Server instance must share the same directory. In turn, the host path can refer to this directory, thus access to the volume is controlled by the underlying shared directory. Given a set of machines that are already set up with a shared directory, this approach is simpler than setting up an NFS client (although maybe not as portable).

To run this example, complete the following steps:

  1. Prepare the yml file for the WebLogic Administration Server.

    Edit wls-admin.yml to mount the host folder /scratch/data to /u01/wlsdomain in the Administration Server pod:

    apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: admin-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: admin-server
    spec:
      containers:
      - name: admin-server
        image: wls-k8s-domain
        imagePullPolicy: Never
        command: ["sh"]
        args: ["/u01/oracle/startadmin.sh"]
        readinessProbe:
            httpGet:
              path: /weblogic/ready
              port: 8001
            initialDelaySeconds: 15
            timeoutSeconds: 5
        ports:
        - containerPort: 8001

        env:
        - name: WLUSER
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: username
        - name: WLPASSWORD
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: password
        volumeMounts:
          # name must match the volume name below
          - name: domain-home
            mountPath: "/u01/wlsdomain"

      volumes:
        - name: domain-home
          hostPath:
            path: /scratch/data
            type: Directory
  2. Prepare the yml file for the Managed Servers.

    Edit wls-stateful.yml to mount the host folder /scratch/data to /u01/wlsdomain in the Managed Server pods:

    apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: StatefulSet
metadata:
  name: managed-server
spec:
  serviceName: wls-subdomain
  replicas: 2
  template:
    metadata:
      name: ms
      labels:
        app: managed-server
    spec:
      subdomain: wls-subdomain
      containers:
      - name: managed-server
        image: wls-k8s-domain
        imagePullPolicy: Never
        command: ["sh"]
        args: ["/u01/oracle/startms.sh"]
        readinessProbe:
            httpGet:
              path: /weblogic/ready
              port: 8011
            initialDelaySeconds: 15
            timeoutSeconds: 5
        ports:
        - containerPort: 8011
        env:
        - name: JAVA_OPTIONS
          value: "-Dweblogic.StdoutDebugEnabled=true"
        - name: USER_MEM_ARGS
          value: "-Xms64m -Xmx256m "
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: DNS_DOMAIN_NAME
          value: "wls-subdomain"
        - name: WLUSER
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: username
        - name: WLPASSWORD
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: password

        volumeMounts:
          # name must match the volume name below
          - name: domain-home
            mountPath: "/u01/wlsdomain"

      volumes:
        - name: domain-home
          hostPath:
            path: /scratch/data
            type: Directory
  3. Create the Administration Server and Managed Server pods with the shared volume. These WebLogic Server instances will start from the mounted domain location. 
    $ kubectl create -f wls-admin.yml
$ kubectl create -f wls-stateful.yml

Use Case 2: NFS Sharing with Kubernetes PV and PVC

This example shows a WebLogic Server cluster with one Administration Server and several Managed Server instances, each server residing in a dedicated pod. All the pods have volumes mounted to a central NFS server that is located in a physical machine that the pods can reach. The first time the Administration Server pod is started, the WebLogic domain is created in the shared NFS folder. At runtime, all WebLogic Server instances, including the Administration Server, share the same domain home directory via a mounted volume by PV and PVC.

  1. In this sample we have the NFS server on host 10.232.128.232, which has a read/write export to all external hosts on /scratch/nfsdata.
  2. Prepare the PV.

    Edit pv.yml to make sure each WebLogic Server instance has read and write access to the NFS shared folder:

    kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv1
  labels:
    app: wls-domain
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle  # Retain, Recycle, Delete
  nfs:
    # Please use the correct NFS server host name or IP address
    server: 10.232.128.232
    path: "/scratch/nfsdata"
  3. Prepare the PVC.

    Edit pvc.yml:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wlserver-pvc-1
  labels:
    app: wls-server
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

    Kubernetes will find the matching PV for the PVC, and bind them together [Kubernetes Persistent Volumes and Claims].

  4. Create the PV and PVC: 
    $ kubectl create -f pv.yml
$ kubectl create -f pvc.yml

    Then check the PVC status to make sure it binds to the PV:

    $ kubectl get pvc
NAME            STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
wlserver-pvc-1  Bound     pv1       10Gi       RWX           manual         7s
  5. Prepare the yml file for the Administration Server. It has a reference to the PVC wlserver-pvc-1.

    Edit wls-admin.yml to mount the NFS shared folder to /u01/wlsdomain in the WebLogic Server Administration Server pod:

    apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: admin-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: admin-server
    spec:
      containers:
      - name: admin-server
        image: wls-k8s-domain
        imagePullPolicy: Never
        command: ["sh"]
        args: ["/u01/oracle/startadmin.sh"]
        readinessProbe:
            httpGet:
              path: /weblogic/ready
              port: 8001
            initialDelaySeconds: 15
            timeoutSeconds: 5
        ports:
        - containerPort: 8001

        env:
        - name: WLUSER
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: username
        - name: WLPASSWORD
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: password
        volumeMounts:
          # name must match the volume name below
          - name: domain-home
            mountPath: "/u01/wlsdomain"

      volumes:
        - name: domain-home
          persistentVolumeClaim:
            claimName: wlserver-pvc-1
  6. Prepare the yml file for the Managed Servers. It has a reference to the PVC wlserver-pvc-1.

    Edit wls-stateful.yml to mount the NFS shared folder to /u01/wlsdomain in each Managed Server pod:

    apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: StatefulSet
metadata:
  name: managed-server
spec:
  serviceName: wls-subdomain
  replicas: 2
  template:
    metadata:
      name: ms
      labels:
        app: managed-server
    spec:
      subdomain: wls-subdomain
      containers:
      - name: managed-server
        image: wls-k8s-domain
        imagePullPolicy: Never
        command: ["sh"]
        args: ["/u01/oracle/startms.sh"]
        readinessProbe:
            httpGet:
              path: /weblogic/ready
              port: 8011
            initialDelaySeconds: 15
            timeoutSeconds: 5
        ports:
        - containerPort: 8011
        env:
        - name: JAVA_OPTIONS
          value: "-Dweblogic.StdoutDebugEnabled=true"
        - name: USER_MEM_ARGS
          value: "-Xms64m -Xmx256m "
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: DNS_DOMAIN_NAME
          value: "wls-subdomain"
        - name: WLUSER
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: username
        - name: WLPASSWORD
          valueFrom:
            secretKeyRef:
              name: wlsecret
              key: password

        volumeMounts:
          - name: domain-home
            mountPath: "/u01/wlsdomain"

      volumes:
      - name: domain-home
        persistentVolumeClaim:
          claimName: wlserver-pvc-1
  7. Create the Administration Server and Managed Server pods with the NFS shared volume. Each WebLogic Server instance will start from the mounted domain location: 
    $ kubectl create -f wls-admin.yml
$ kubectl create -f wls-stateful.yml

Summary

This blog describes the best practices of setting Kubernetes data volumes when running a WebLogic domain in a Kubernetes environment. Because Kubernetes pods are ephemeral, it is a best practice to persist the WebLogic domain to volumes, as well as files such as logs, stores, and so on. Kubernetes provides persistent volumes and persistent volume claims to simplify externalizing state and persisting important data to volumes. We provide two use cases: the first describes how to map the volume to a host machine where the Kubernetes nodes are running; and the second describes how to use an NFS shared volume. In both use cases, all WebLogic Server instances must have access to the files that are mapped to these volumes.