Kubernetes provides the following autoscalers to improve the overall performance of your Kubernetes cluster:
-
VerticalPodAutoscaler
-
HorizontalPodAutoscaler
-
ClusterAutoscaler
In this article, I cover the vertical pod auto scaler. The other auto scalers are covered in subsequent articles. A fair amount of documentation regarding these auto scalers exists. However, the documentation covers the material at a coarse-grain viewpoint. I find it much easier to understand the technology if I can get my hands dirty and begin to turn a few knobs in the configuration.
The VerticalPodAutoscaler (VPA) frees the users from the necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it sets the requests automatically based on usage and allows proper scheduling onto nodes that the appropriate resource amount is available for each pod. It also maintains ratios between limits and requests that were specified in the initial container’s configuration.
The VPA can scale down pods that are over-requesting resources and scale up pods that are under-requesting resources based on their usage over time.
VPA Components
The VPA consists of the following components:
-
Recommender
-
Updater
-
Admission Plugin
Recommender
The recommender is the core of the VerticalPodAutoscaler system. It computes the recommended resource requests for pods based on the historical and current usage of the resources. The current recommendations are put in a status of the VPA resources, where they can be inspected.
Updater
Updater runs in Kubernetes cluster and decides which pods to restart, based on resources allocation recommendation calculated by the recommender. If a pod needs updating, the Updater tries to evict the pod. It respects the pod disruption budget by using the eviction API to evict pods. The updater doesn’t perform the actual resources update but relies on the VPA admission plugin to update pod resources when the pod is created after eviction.
Admission plugin (controller)
The admission controller registers itself as a mutating admission webhook and is therefore on the path of creating all pods. For each pod creation, it receives a request from the APIServer, and it either finds the corresponding VPA configuration (or decides there isn’t one) and uses the current recommendation to set resource requests in the pod.
Policies
The VPA specification identifies two policies: Update and resource. These policies request how the recommender and updater behave. I cover these policies later in the document.
VPA in action
Now that we have a basic understanding of the components in the VPA, we can work with the VPA. Look at our sample VerticalPodAutoScaler and the deployment it’s intended to monitor. You can find the example for this demonstration on GitHub.
The image deployed is a slim ubuntu image. A simple endless loop is executed to increase the CPU and memory consumption. To verify the container is running successfully, access the logs for the pod. You can see multiple “OK” strings being written to the stdout.
| VerticalPodAutoscaler | Deployment |
|---|---|
| apiVersion: “autoscaling.k8s.io/v1beta2” kind: VerticalPodAutoscaler metadata: name: hamster-vpa spec: targetRef: apiVersion: “apps/v1” kind: Deployment name: hamster updatePolicy: updateMode: “Off” resourcePolicy: containerPolicies: – containerName: ‘hamster’ mode: “Auto” minAllowed: cpu: 100m memory: 50Mi maxAllowed: cpu: 1 memory: 500Mi controlledResources: [“cpu”, “memory”] |
apiVersion: apps/v1 kind: Deployment metadata: name: hamster spec: selector: matchLabels: app: hamster replicas: 2 template: metadata: labels: app: hamster spec: containers: – name: hamster image: k8s.gcr.io/ubuntu-slim:0.1 resources: requests: cpu: 100m memory: 50Mi command: [“/bin/sh”] args: – “-c” – “while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done” |
The VPA can specify one of many kinds. The options are deployment, statefulset, daemonset, replicaset, job, cronjob, and replicationcontorller. The example specifying a VPA for a kind of deployment. The VPA has been defined for the deployment named “hamster.” A few parameters exist in one of the two policies.
Update policy
The update policy controls how the VPA applies changes. This parameter informs the VPA recommender what action to take with pods that have the incorrect resources. The options can be one of the following values:
-
Off: The VPA never changes the pod resources. The recommender sets the recommended resources in the VPA object. Use this option for a dry run.
-
Auto (default): The VPA assigns resources on pod creation and can update them during the lifetime of the pod, including evicting or rescheduling the pod.
-
Initial: The VPA only assigns resources on pod creation and doesn’t change them during the lifetime of the pod.
Resource policy
The array of policies, containerPolicies, are defined for the containers deployed in the pod. The policy, per container in the pod, controls how the autoscaler computes the recommended resources. Each policy contains the name of the container, scaling mode (auto or off), and the minimum and maximum allowed resources of CPU and memory.
Demonstrations
With the foundation laid, let’s run some tests to demonstrate the behavior of the VPA. The first test is a dry run. The update mode is disabled, but you want the recommender to run so that you see the recommended settings.
The VPA is deployed for the hamster container. When you get a list of the pods, there are only two replicas, which are defined by the deployment for the hamster deployment. Because the update policy is disabled, the autoupdater didn’t eject and create new instances of the container with updated resources. However, run a describe VPA, and you see that the VPA recommender has made some recommendations for the container’s resources.
$kubectl describe vpa hamster-vpa| $ kubectl describe vpa hamster-vpa | $ kubectl describe po hamster |
|---|---|
| Name: hamster-vpa Namespace: default Labels: . . . . Kind: VerticalPodAutoscaler . . . Manager: recommender Operation: Update . . . . Target Ref: API Version: apps/v1 Kind: Deployment Name: hamster Update Policy: Update Mode: Off . . . . Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: hamster Lower Bound: Cpu: 586m Memory: 262144k Target: Cpu: 587m Memory: 262144k Uncapped Target: Cpu: 587m Memory: 262144k Upper Bound: Cpu: 607m Memory: 262144k Events: |
Name: hamster-5f9fdfff5-2prd8 Namespace: default Priority: 0 . . . . Labels: app=hamster pod-template-hash=5f9fdfff5 . . . . Image: k8s.gcr.io/ubuntu-slim:0.1 . . . State: Running Started: Fri, 08 Jan 2021 11:44:55 -0500 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 50Mi Environment: Mounts: . . . . Events: |
Because the VPA has the update mode disabled, the recommender suggested changes to the container resources. However, the pods weren’t evicted and recreated with the updated recommendations. Running a describe on the pod shows that the original resource settings are in place.
The next test execution is the enablement of the update policy. With the update policy enabled, the pods are evicted and rescheduled with the recommended resources for both CPU and memory. If you’re interested in watching the VPA in action, run the following command after deploying the hamster deployment:
$ kubectl get po --watchAs the recommender runs, it captures the status and makes a recommendation, which is stored in the internal recommendation status object. If the recommender says that the resources need updating, the updater evicts the pod and reschedules the pod.
$ kubectl get po --watchNAME READY STATUS RESTARTS AGE
hamster-5f9fdfff5-gmt7r 1/1 Running 0 53s
hamster-5f9fdfff5-xkj76 1/1 Running 0 52s
hamster-5f9fdfff5-xkj76 1/1 Terminating 0 107s
hamster-5f9fdfff5-lgw6z 0/1 Pending 0 1s
hamster-5f9fdfff5-lgw6z 0/1 Pending 0 1s
hamster-5f9fdfff5-lgw6z 0/1 ContainerCreating 0 1s
hamster-5f9fdfff5-lgw6z 1/1 Running 0 3s
Watching the events of the hamster pod shows the pod being deleted, rescheduled, container created, and then set to the running state. Doing another describe of the hamster pod demonstrates that the container is running with updated CPU and memory resources.
$ kubectl describe po hamster-xxxxxxxxxx-xxxxx| Original deployment | After VPA update |
|---|---|
| Name: hamster-5f9fdfff5-2prd8 Namespace: default Priority: 0 . . . . Labels: app=hamster pod-template-hash=5f9fdfff5 . . . . Image: k8s.gcr.io/ubuntu-slim:0.1 . . . State: Running Started: Fri, 08 Jan 2021 11:44:55 -0500 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 50Mi Environment: Mounts: . . . . Events: |
Name: hamster-5f9fdfff5-7c2cl Namespace: default . . . . Labels: app=hamster pod-template-hash=5f9fdfff5 Annotations: vpaObservedContainers: hamster vpaUpdates: Pod resources updated by hamster-vpa: container 0: cpu request, memory request . . . . State: Running Started: Wed, 03 Feb 2021 19:10:23 -0500 Ready: True Restart Count: 0 Requests: cpu: 587m memory: 262144k |
Evictions: What Happens?
When you enable the update policy, what happens to inflight requests for pods that have been scheduled for eviction? A sighup signal is received, and the kubelet waits until the timeout you set on your pod before it sends a sigkill signal. This waiting period allows for the current request to complete, and new requests are handled by the other replicas. The timeout is set in the PodSpecification with the terminationGracePeriodSeconds. Set this value large enough to account for the long-running process.
Obviously, you don’t want all of your pods getting evicted simultaneously, because requests would fail until pods came back online. To control the number of pods that are always available, you can deploy a PodDisruptionBudget (PDB). A PDB specifies the number of replicas that an application can tolerate having, relative to how many it’s intended to have.
The PodDisruptionBudget has the following fields:
-
A label selector to specify the set of pods to which it applies (.spec.selector).
-
A minimum available, which describes the number of pods from the set that must be available after the eviction, even in the absence of the evicted pod (.spec.minAvailable). The minimum available can be an absolute number or a percentage.
-
A maximum unavailable, which describes the number of pods from that set that can be unavailable after the eviction (.spec.maxUnavailable). It can be an absolute number or a percentage.
To better understand the PodDisruptionBudget, visit the following sections of the Kubernetes documentation:
Summary
The VPA is an excellent option for automatically scaling your pods. However, using the VPA does come with the following caveats:
-
If they’re both monitoring CPU and memory resources, don’t deploy the VerticalPodAutoscaler and HorizontalPodAutoscaler (HPA) together.
-
You can deploy the VPA and HPA if the HPA is monitoring custom metrics, not including memory and CPU.
In this article, I only touched on the vertical pod autoscaler. This component helps you gain an understanding of the resources, CPU, and memory needed to run your service efficiently and reduce your overall resource costs. The VPA scales up and down, so you don’t have to worry about scaling to the peak requirements and having to pay those costs 24/7.
In upcoming articles, I discuss the other autoscalers. Taking advantage of these autoscalers improves the resiliency, high availability, and elasticity of your cloud native applications.