Kubernetes has gained rapid traction over the last three years and is being deployed in production by many companies. While in general, Kubernetes does follow the core software security principles, some ownership of security falls on the shoulders of the end users. Just like a shared security responsibility model exists between all cloud providers and the customers, there is a shared security responsibility for managed Kubernetes services being offered by cloud providers. Managed Kubernetes Services Cloud providers like Oracle Cloud Infrastructure Container Engine for Kubernetes (also known as Oracle Kubernetes Engine or OKE), Azure Kubernetes Service (AKS), and others are typically responsible for managing and securing the control plane (API Server, scheduler, etcd, controllers) of the Kubernetes cluster and customers of the managed service are responsible for the securing the data plane (node pools, ingress, networking, service mesh etc).
I started working on Kubernetes about three years ago using Kubernetes Oracle Linux Vagrant boxes. Here are 5 Kubernetes Security practices based on my experience:
Let's look at each of these areas.
Use Role Based Access Control (RBAC)
Role Based Access Control (RBAC) lets the customer control who can access the Kubernetes API and what permissions they have. RBAC is typically enabled by default in Kubernetes. However, if one has upgraded from a very old Kubernetes release and had not enabled it earlier, RBAC settings should be checked to make sure they are enabled. Another thing to keep in mind is that simply enabling RBAC is not enough, one should manage the authorization policies and use them properly. Use RBAC to limit users and groups to just the actions and tasks they may need. Always follow the principle of least privilege to ensure users and Kubernetes Service Accounts have the minimal set of privileges required. Make sure to not give cluster-wide permissions and do not give anyone cluster admin privileges unless absolutely necessary. Refer to the official Kubernetes RBAC documentation for more information. For operations on Kubernetes clusters created and managed using Oracle Container Engine for Kubernetes, Oracle Cloud Infrastructure Identity and Access Management (OCI IAM) provides complete access control. The documentation here provides more details.
Secrets should be secrets
Secrets contain sensitive data such as a password, a token, or a ssh key. Kubernetes secrets help securely initialize pods with artifacts like keys, passwords, tokens, etc. When a pod starts up, it will generally need to access its secrets. Whenever a Service Account is created, a Kubernetes secret storing its auth token is automatically generated. Kubernetes supports encryption at rest. This will encrypt Secret resources in etcd, preventing access to your etcd backups from viewing the content of those secrets. Encryption offers an additional level of defense when backups are not encrypted or an attacker gains read access to etcd. Ensure that the communication between users and the API server and from the API server to the kubelets is protected using SSL/TLS as explained here. A recommended practice is to have a short lifetime for a secret or credential to make it harder for an attacker to use them. Setting short lifetimes on certificates and automating their rotation is a good practice. Another thing to keep in mind is being aware of third-party integrations that request access to secrets of your Kubernetes cluster. In such cases, carefully review the RBAC permissions and access being requested or you may compromise the security profile of your cluster. If you are using Oracle Kubernetes Engine, refer to Encrypting Kubernetes Secrets at Rest in Etcd for more information.
Secure nodes and pods
Nodes: A Kubernetes node is a worker node that can be a VM or physical machine that typically runs Linux Operating System (OS). The services running on a node include the container runtime, kubelet and kube-proxy. Hardening and securing the OS running on the nodes is important and this is the responsibility of the cloud provider and the Kubernetes administrator. Oracle Kubernetes Engine (OKE) nodes come with a hardened Oracle Linux image. Security patches should be regularly applied on the Oracle Linux image that runs on OKE nodes by the Kubernetes administrator once the nodes have been provisioned by a customer. With managed OKE, Center for Internet Security (CIS) Kubernetes benchmark is also used for the nodes. In addition to OS security, it is recommended that nodes are on a private network and not accessible from the Internet. A gateway may be configured for access to other services outside of the cluster network if needed. Network ports access on nodes should controlled via network access lists. It is also recommend limiting Secure Shell (SSH) access to the Nodes. The Oracle Kubernetes Engine node pool security documentation provides some more guidance.
Pods: A pod is a group of one or more containers that run on nodes and use shared storage/network. By default, there are no restrictions on which nodes may run a pod. Use Network Policies to define rules of communication for pods within a cluster. Network policies are implemented by the network plugin and their usage may require a network driver that supports policies. Oracle Kubernetes Engine offers multiple options to secure communication to and from the workloads in your cluster. For the best network security posture, one should evaluate using a combination of network policies (to secure pod-level network communication) and security lists (to secure host-level network communication). Pod Security Policies allow a customer to control runtime execution properties of the pods, such as ability to run containers as privileged containers, usage of the host file system, network and ports. By default, a pod may be scheduled on any node in the cluster. Kubernetes offers multiple ways to control pod assignment to nodes such as policies for controlling placement of pods onto nodes and taint based pod placement and eviction. If using Oracle Kubernetes Engine (OKE), one can set up pod security policies for the cluster as explained in the documentation.
Eliminate container security risks
Applications are packaged as container images (commonly Docker images). Container images (Docker images) are stored and pulled from a container registry and instantiated as runtime containers inside pods. Security has to be a design principle right at the beginning of the development process when you are working on the source code and libraries to build container images for your applications. Implement security practices in your CI/CD tool chain and during the entire build, store, and deploy process of container images. These include securely storing the container images, scanning those images for security vulnerabilities, and managing the run time security of the containers. As part of your DevSecOps cycle, it is a good idea to automate vulnerability scanning of third-party libraries that you may be using to build applications. If you are using Oracle Kubernetes Engine, you can also look at some of our partner solutions like NeuVector and Twistlock. When building Docker images and containers, use hardened slim OS images and ensure that the users running the application have the least level of OS privileges necessary to run the processes inside the container. Another important thing to remember is to regularly apply security updates on the source image and then redeploy them as updated containers. It is also important to use private Docker registries like Oracle Cloud Infrastructure Registry with proper access control and policies in place and having governance for the management of container images. Signing container images and maintaining a system of trust for the content of containers is suggested.
Auditing, Logging, and Monitoring is essential
Auditing, Logging and Monitoring are important security aspects that can help improve the security posture of your cluster and should not be overlooked. Kubernetes audit logs are detailed descriptions of each call made to the Kubernetes API-Server. These audit logs provide useful information about what is happening in a cluster and can even be used for auditing, compliance and security analysis. Kubernetes audit records includes security records that capture the complete sequence of activities and can help detecting anomalous behavior and access to sensitive resources. It is recommended to enable audit logging and save the audit logs on a secure repository for analysis in the event of a compromise. Kubernetes also provides cluster-based logging to record container activity into a central logging subsystem. The standard output and standard error output of each container in a Kubernetes cluster can be ingested using an agent like Fluentd running on each node into tools like Elasticsearch and viewed with Kibana. And finally, monitor containers, pods, applications, services, and other components of your cluster. One can use tools such as Prometheus, Grafana, Jaeger for monitoring, visibility, and tracing the cluster.
If using Oracle Cloud Infrastructure Container Engine for Kubernetes (also known as Oracle Kubernetes Engine or OKE), review the OCI Security Guide and some additional recommendations for securing Oracle Kubernetes Engine. OKE is also integrated with Oracle Cloud Infrastructure (OCI) Identity and Access Management (IAM), which provides additional integrated security for authentication using native OCI identity functionality.
Secure your Kubernetes cluster today!
Manish Kapur is a technical product leader with a strong background in product management, developer relations, marketing, and strategy. He currently leads the developer relations platform team at Oracle.