Update (March 23, 2022): Oracle Container Engine for Kubernetes (OKE) supports provisioning PVCs by binding them to PVs backed by the File Storage Service for clusters running Kubernetes version 1.18 or later. To use File Storage Service with your OKE cluster, please follow the steps in Provisioning PVCs on the File Storage Service.

Oracle Container Engine for Kubernetes is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud.

One of the best practices for containerized applications is to use stateless containers. However, many real-world applications require stateful behavior for some of their containers. For example, a classic three-tier application might have three containers:

  • One for the presentation layer, stateless

  • One for the application layer, stateless

  • One for persistence ( such as database), stateful

In Kubernetes, each container can read and write to its own file system. But when a container is restarted, all data is lost. Therefore, containers that need to maintain state would store data in a persistent storage such as Network File System (NFS). What’s already stored in NFS isn’t deleted when a pod, which might contain one or more containers, is destroyed. Also, an NFS can be accessed from multiple pods at the same time, so an NFS can be used to share data between pods. This behavior is really useful when containers or applications need to read configuration data from a single shared file system or when multiple containers need to read from and write data to a single shared file system.

Oracle Cloud Infrastructure File Storage provides a durable, scalable, and distributed enterprise-grade network file system that supports NFS version 3 along with Network Lock Manager (NLM) for a locking mechanism. You can connect to File Storage from any bare metal, virtual machine, or container instance in your virtual cloud network (VCN). You can also access a file system from outside the VCN by using Oracle Cloud Infrastructure FastConnect or an Internet Protocol Security (IPSec) virtual private network (VPN). File Storage is a fully managed service so you don’t have to worry about hardware installations and maintenance, capacity planning, software upgrades, security patches,  and so on. You can start with a file system that contains only a few kilobytes of data and grows to handle 8 exabytes of data.

This post explains how to use File Storage (sometimes referred to as FSS) with Container Engine for Kubernetes (sometimes referred to as OKE). We’ll create two pods. One pod runs on Worker Node 1, the other pod runs on Worker Node 2, and they share the same File Storage file system:
 

Then we’ll look inside the pod and see how to configure it with File Storage.

Prerequisites

High-Level Steps

  1. Create storage class.
  2. Create a persistent volume (PV).
  3. Create a persistent volume claim (PVC).
  4. Create a pod to consume the PVC.

Create a Storage Class

Create a storage class that references the mount target ID from file system that you created:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: oci-fss
provisioner: oracle.com/oci-fss
parameters:
  # Insert mount target from the FSS here
  mntTargetId: ocid1.mounttarget.oc1.iad.aaaaaaaaaaaaaaaaaaaaaaaaaa

 

Create a Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolume
metadata:
 name: oke-fsspv
spec:
 storageClassName: oci-fss
 capacity:
  storage: 100Gi
 accessModes:
  - ReadWriteMany
 mountOptions:
  - nosuid
 nfs:
# Replace this with the IP of your FSS file system in OCI
  server: 10.0.32.8
# Replace this with the Path of your FSS file system in OCI
  path: "/okefss"
  readOnly: false

 

Create a Persistent Volume Claim (PVC)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: oke-fsspvc
spec:
 storageClassName: oci-fss
 accessModes:
 - ReadWriteMany
 resources:
  requests:
 # Although storage is provided here it is not used for FSS file systems
    storage: 100Gi
 volumeName: oke-fsspv

 

Verify That the PVC Is Bound

raghpras-Mac:fss raghpras$ kubectl get pvc oke-fsspvc
NAME         STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
oke-fsspvc   Bound    oke-fsspv   100Gi      RWX            oci-fss        1h
raghpras-Mac:fss raghpras$

 

Label the Worker Nodes

Label two worker nodes so that a pod can be assigned to each of them:

kubectl label node 129.213.110.23 nodeName=node1

kubectl label node 129.213.137.236  nodeName=node2

 

Use the PVC in a Pod

The following pod (oke-fsspod) on Worker Node 1 (node1) consumes the file system PVC (oke-fsspvc).

#okefsspod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: oke-fsspod
spec:
  containers:
  - name: web
    image: nginx
    volumeMounts:
     - name: nfs
       mountPath: "/usr/share/nginx/html/"
    ports:
      - containerPort: 80
        name: http
  volumes:
  - name: nfs
    persistentVolumeClaim:
      claimName: oke-fsspvc
      readOnly: false
  nodeSelector:
    nodeName: node1

 

Create the Pod

kubectl apply -f okefsspod.yaml

 

Test

After creating the pod, use kubectl exec to test that you can write to the file share:

raghpras-Mac:fss raghpras$ kubectl get pods oke-fsspod -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP            NODE             NOMINATED NODE
oke-fsspod   1/1     Running   0          33m   10.244.2.11   129.213.110.23   <none>
raghpras-Mac:fss raghpras$

 

Write to the File System by Using kubectl exec

raghpras-Mac:fss raghpras$ kubectl exec -it oke-fsspod bash

root@oke-fsspod:/# echo "Hello from POD1" >> /usr/share/nginx/html/hello_world.txt

root@oke-fsspod:/# cat /usr/share/nginx/html/hello_world.txt

Hello from POD1

root@oke-fsspod:/#

 

Repeat the Process with the Other Pod

Ensure that this file system can be mounted into the other pod (oke-fsspod2), which is on Worker Node 2 (node2):

apiVersion: v1
#okefsspod2.yaml
kind: Pod
metadata:
  name: oke-fsspod2
spec:
  containers:
  - name: web
    image: nginx
    volumeMounts:
     - name: nfs
       mountPath: "/usr/share/nginx/html/"
    ports:
      - containerPort: 80
        name: http
  volumes:
  - name: nfs
    persistentVolumeClaim:
      claimName: oke-fsspvc
      readOnly: false
  nodeSelector:
   nodeName: node2

 

raghpras-Mac:fss raghpras$ kubectl apply -f okefsspod2.yaml 

pod/oke-fsspod2 created

raghpras-Mac:fss raghpras$ kubectl get pods oke-fsspod oke-fsspod2 -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP            NODE              NOMINATED NODE
oke-fsspod    1/1     Running   0          12m   10.244.2.17   129.213.110.23    <none>
NAME          READY   STATUS    RESTARTS   AGE   IP            NODE              NOMINATED NODE
oke-fsspod2   1/1     Running   0          12m   10.244.1.9    129.213.137.236   <none>

raghpras-Mac:fss raghpras$ kubectl exec -it oke-fsspod2 -- cat /usr/share/nginx/html/hello_world.txt

Hello from POD1

raghpras-Mac:fss raghpras$

 

Test Again

You can also test that the newly created pods can write to the share:

raghpras-Mac:fss raghpras$ kubectl exec -it oke-fsspod2 bash

root@oke-fsspod2:/# echo "Hello from POD2" >> /usr/share/nginx/html/hello_world.txt

root@oke-fsspod2:/# cat /usr/share/nginx/html/hello_world.txt

Hello from POD1

Hello from POD2

root@oke-fsspod2:/# exit

exit

 

Conclusion

Both File Storage and Container Engine for Kubernetes are fully managed services that are highly available and highly scalable. File Storage also provides highly persistent and durable storage for your data on Oracle Cloud Infrastructure. File Storage is built on distributed architecture to provide scale for your data and for accessing your data. Leveraging both services will simplify your workflows in the cloud and give you flexibility and options on how you store your container data.

What’s Next

Dynamic volume provisioning for File Storage Service., which is in development, creates file systems and mount targets when a customer requests file storage inside the Kubernetes cluster.

If you want to learn more about Oracle Cloud Infrastructure, Container Engine for Kubernetes, or File Storage, our cloud landing page is a great place to start.

Resources