Oracle Cloud Infrastructure Container Engine for Kubernetes is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud.
One of the best practices for containerized applications is to use stateless containers. However, many real-world applications require stateful behavior for some of their containers. For example, a classic three-tier application might have three containers:
One for the presentation layer, stateless
One for the application layer, stateless
One for persistence ( such as database), stateful
In Kubernetes, each container can read and write to its own file system. But when a container is restarted, all data is lost. Therefore, containers that need to maintain state would store data in a persistent storage such as Network File System (NFS). What’s already stored in NFS isn't deleted when a pod, which might contain one or more containers, is destroyed. Also, an NFS can be accessed from multiple pods at the same time, so an NFS can be used to share data between pods. This behavior is really useful when containers or applications need to read configuration data from a single shared file system or when multiple containers need to read from and write data to a single shared file system.
Oracle Cloud Infrastructure File Storage provides a durable, scalable, and distributed enterprise-grade network file system that supports NFS version 3 along with Network Lock Manager (NLM) for a locking mechanism. You can connect to File Storage from any bare metal, virtual machine, or container instance in your virtual cloud network (VCN). You can also access a file system from outside the VCN by using Oracle Cloud Infrastructure FastConnect or an Internet Protocol Security (IPSec) virtual private network (VPN). File Storage is a fully managed service so you don't have to worry about hardware installations and maintenance, capacity planning, software upgrades, security patches, and so on. You can start with a file system that contains only a few kilobytes of data and grows to handle 8 exabytes of data.This post explains how to use File Storage (sometimes referred to as FSS) with Container Engine for Kubernetes (sometimes referred to as OKE). We'll create two pods. One pod runs on Worker Node 1, the other pod runs on Worker Node 2, and they share the same File Storage file system:
Then we'll look inside the pod and see how to configure it with File Storage.
Oracle Cloud Infrastructure account credentials for the tenancy.
A Container Engine for Kubernetes cluster created in your tenancy. An example of that is shown in the Container Engine for Kubernetes documentation.
Security lists configured to support File Storage as explained in the File Storage documentation. The following image shows a sample security list configuration:
A file system and a mount target created according to the instructions in Announcing File Storage Service UI 2.0.
Create a storage class that references the mount target ID from file system that you created:
kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: oci-fss provisioner: oracle.com/oci-fss parameters: # Insert mount target from the FSS here mntTargetId: ocid1.mounttarget.oc1.iad.aaaaaaaaaaaaaaaaaaaaaaaaaa
apiVersion: v1 kind: PersistentVolume metadata: name: oke-fsspv spec: storageClassName: oci-fss capacity: storage: 100Gi accessModes: - ReadWriteMany mountOptions: - nosuid nfs: # Replace this with the IP of your FSS file system in OCI server: 10.0.32.8 # Replace this with the Path of your FSS file system in OCI path: "/okefss" readOnly: false
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: oke-fsspvc spec: storageClassName: oci-fss accessModes: - ReadWriteMany resources: requests: # Although storage is provided here it is not used for FSS file systems storage: 100Gi volumeName: oke-fsspv
raghpras-Mac:fss raghpras$ kubectl get pvc oke-fsspvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE oke-fsspvc Bound oke-fsspv 100Gi RWX oci-fss 1h raghpras-Mac:fss raghpras$
Label two worker nodes so that a pod can be assigned to each of them:
kubectl label node 22.214.171.124 nodeName=node1
kubectl label node 126.96.36.199 nodeName=node2
The following pod (oke-fsspod) on Worker Node 1 (node1) consumes the file system PVC (oke-fsspvc).
#okefsspod.yaml apiVersion: v1 kind: Pod metadata: name: oke-fsspod spec: containers: - name: web image: nginx volumeMounts: - name: nfs mountPath: "/usr/share/nginx/html/" ports: - containerPort: 80 name: http volumes: - name: nfs persistentVolumeClaim: claimName: oke-fsspvc readOnly: false nodeSelector: nodeName: node1
kubectl apply -f okefsspod.yaml
After creating the pod, use kubectl exec to test that you can write to the file share:
raghpras-Mac:fss raghpras$ kubectl get pods oke-fsspod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE oke-fsspod 1/1 Running 0 33m 10.244.2.11 188.8.131.52 <none> raghpras-Mac:fss raghpras$
raghpras-Mac:fss raghpras$ kubectl exec -it oke-fsspod bash root@oke-fsspod:/# echo "Hello from POD1" >> /usr/share/nginx/html/hello_world.txt root@oke-fsspod:/# cat /usr/share/nginx/html/hello_world.txt Hello from POD1 root@oke-fsspod:/#
Ensure that this file system can be mounted into the other pod (oke-fsspod2), which is on Worker Node 2 (node2):
apiVersion: v1 #okefsspod2.yaml kind: Pod metadata: name: oke-fsspod2 spec: containers: - name: web image: nginx volumeMounts: - name: nfs mountPath: "/usr/share/nginx/html/" ports: - containerPort: 80 name: http volumes: - name: nfs persistentVolumeClaim: claimName: oke-fsspvc readOnly: false nodeSelector: nodeName: node2
raghpras-Mac:fss raghpras$ kubectl apply -f okefsspod2.yaml pod/oke-fsspod2 created raghpras-Mac:fss raghpras$ kubectl get pods oke-fsspod oke-fsspod2 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE oke-fsspod 1/1 Running 0 12m 10.244.2.17 184.108.40.206 <none> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE oke-fsspod2 1/1 Running 0 12m 10.244.1.9 220.127.116.11 <none> raghpras-Mac:fss raghpras$ kubectl exec -it oke-fsspod2 -- cat /usr/share/nginx/html/hello_world.txt Hello from POD1 raghpras-Mac:fss raghpras$
You can also test that the newly created pods can write to the share:
raghpras-Mac:fss raghpras$ kubectl exec -it oke-fsspod2 bash root@oke-fsspod2:/# echo "Hello from POD2" >> /usr/share/nginx/html/hello_world.txt root@oke-fsspod2:/# cat /usr/share/nginx/html/hello_world.txt Hello from POD1 Hello from POD2 root@oke-fsspod2:/# exit exit
Both File Storage and Container Engine for Kubernetes are fully managed services that are highly available and highly scalable. File Storage also provides highly persistent and durable storage for your data on Oracle Cloud Infrastructure. File Storage is built on distributed architecture to provide scale for your data and for accessing your data. Leveraging both services will simplify your workflows in the cloud and give you flexibility and options on how you store your container data.
Dynamic volume provisioning for File Storage Service., which is in development, creates file systems and mount targets when a customer requests file storage inside the Kubernetes cluster.
If you want to learn more about Oracle Cloud Infrastructure, Container Engine for Kubernetes, or File Storage, our cloud landing page is a great place to start.
Just a quick note that FSS requires public subnets which means that the OKE worker nodes need to be on the public subnet in order for this solution to work.