X

The latest cloud infrastructure announcements, technical solutions, and enterprise cloud insights.

Using the VerticalPodAutoscaler in your Kubernetes cluster

Sherwood Zern
Product Manager / Strategy

Kubernetes provides the following autoscalers to improve the overall performance of your Kubernetes cluster:

  • VerticalPodAutoscaler

  • HorizontalPodAutoscaler

  • ClusterAutoscaler

In this article, I cover the vertical pod auto scaler. The other auto scalers are covered in subsequent articles. A fair amount of documentation regarding these auto scalers exists. However, the documentation covers the material at a coarse-grain viewpoint. I find it much easier to understand the technology if I can get my hands dirty and begin to turn a few knobs in the configuration.

The VerticalPodAutoscaler (VPA) frees the users from the necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it sets the requests automatically based on usage and allows proper scheduling onto nodes that the appropriate resource amount is available for each pod. It also maintains ratios between limits and requests that were specified in the initial container's configuration.

The VPA can scale down pods that are over-requesting resources and scale up pods that are under-requesting resources based on their usage over time.

VPA Components

The VPA consists of the following components:

  • Recommender

  • Updater

  • Admission Plugin

Recommender

The recommender is the core of the VerticalPodAutoscaler system. It computes the recommended resource requests for pods based on the historical and current usage of the resources. The current recommendations are put in a status of the VPA resources, where they can be inspected.

Updater

Updater runs in Kubernetes cluster and decides which pods to restart, based on resources allocation recommendation calculated by the recommender. If a pod needs updating, the Updater tries to evict the pod. It respects the pod disruption budget by using the eviction API to evict pods. The updater doesn’t perform the actual resources update but relies on the VPA admission plugin to update pod resources when the pod is created after eviction.

Admission plugin (controller)

The admission controller registers itself as a mutating admission webhook and is therefore on the path of creating all pods. For each pod creation, it receives a request from the APIServer, and it either finds the corresponding VPA configuration (or decides there isn’t one) and uses the current recommendation to set resource requests in the pod.

Policies

The VPA specification identifies two policies: Update and resource. These policies request how the recommender and updater behave. I cover these policies later in the document.

VPA in action

Now that we have a basic understanding of the components in the VPA, we can work with the VPA. Look at our sample VerticalPodAutoScaler and the deployment it’s intended to monitor. You can find the example for this demonstration on GitHub.

The image deployed is a slim ubuntu image. A simple endless loop is executed to increase the CPU and memory consumption. To verify the container is running successfully, access the logs for the pod. You can see multiple "OK" strings being written to the stdout.

VerticalPodAutoscaler Deployment
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
   name: hamster-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: hamster
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: 'hamster'
        mode: "Auto"
        minAllowed:
          cpu: 100m
          memory: 50Mi
        maxAllowed:
          cpu: 1
          memory: 500Mi
        controlledResources: ["cpu", "memory"]
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hamster
spec:
  selector:
    matchLabels:
      app: hamster
  replicas: 2
  template:
    metadata:
      labels:
        app: hamster
    spec:
      containers:
        - name: hamster
          image: k8s.gcr.io/ubuntu-slim:0.1
          resources:
            requests:
              cpu: 100m
              memory: 50Mi

          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

The VPA can specify one of many kinds. The options are deployment, statefulset, daemonset, replicaset, job, cronjob, and replicationcontorller. The example specifying a VPA for a kind of deployment. The VPA has been defined for the deployment named "hamster." A few parameters exist in one of the two policies.

Update policy

The update policy controls how the VPA applies changes. This parameter informs the VPA recommender what action to take with pods that have the incorrect resources. The options can be one of the following values:

  • Off: The VPA never changes the pod resources. The recommender sets the recommended resources in the VPA object. Use this option for a dry run.

  • Auto (default): The VPA assigns resources on pod creation and can update them during the lifetime of the pod, including evicting or rescheduling the pod.

  • Initial: The VPA only assigns resources on pod creation and doesn’t change them during the lifetime of the pod.

Resource policy

The array of policies, containerPolicies, are defined for the containers deployed in the pod. The policy, per container in the pod, controls how the autoscaler computes the recommended resources. Each policy contains the name of the container, scaling mode (auto or off), and the minimum and maximum allowed resources of CPU and memory.

Demonstrations

With the foundation laid, let’s run some tests to demonstrate the behavior of the VPA. The first test is a dry run. The update mode is disabled, but you want the recommender to run so that you see the recommended settings.

The VPA is deployed for the hamster container. When you get a list of the pods, there are only two replicas, which are defined by the deployment for the hamster deployment. Because the update policy is disabled, the autoupdater didn’t eject and create new instances of the container with updated resources. However, run a describe VPA, and you see that the VPA recommender has made some recommendations for the container’s resources.

$kubectl describe vpa hamster-vpa
$ kubectl describe vpa hamster-vpa $ kubectl describe po hamster
Name:         hamster-vpa
Namespace:    default
Labels:      
. . . .
Kind:         VerticalPodAutoscaler
. . .
    Manager:         recommender
    Operation:       Update
  . . . .
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         hamster
  Update Policy:
    Update Mode:  Off

. . . .
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     586m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     607m
Memory:  262144k

Events:         
Name:         hamster-5f9fdfff5-2prd8
Namespace:    default
Priority:     0
. . . .
Labels:       app=hamster
              pod-template-hash=5f9fdfff5
. . . .
    Image:         k8s.gcr.io/ubuntu-slim:0.1
    . . .
    State:          Running
      Started:      Fri, 08 Jan 2021 11:44:55 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     50Mi

    Environment: 
    Mounts:
      . . . .
Events:         

Because the VPA has the update mode disabled, the recommender suggested changes to the container resources. However, the pods weren’t evicted and recreated with the updated recommendations. Running a describe on the pod shows that the original resource settings are in place.

The next test execution is the enablement of the update policy. With the update policy enabled, the pods are evicted and rescheduled with the recommended resources for both CPU and memory. If you’re interested in watching the VPA in action, run the following command after deploying the hamster deployment:

$ kubectl get po --watch

As the recommender runs, it captures the status and makes a recommendation, which is stored in the internal recommendation status object. If the recommender says that the resources need updating, the updater evicts the pod and reschedules the pod.

$ kubectl get po --watch

NAME                                        READY       STATUS                     RESTARTS   AGE
hamster-5f9fdfff5-gmt7r                  1/1          Running                     0                    53s
hamster-5f9fdfff5-xkj76                  1/1           Running                     0                    52s
hamster-5f9fdfff5-xkj76                  1/1           Terminating                0                    107s
hamster-5f9fdfff5-lgw6z                 0/1           Pending                      0                    1s
hamster-5f9fdfff5-lgw6z                0/1            Pending                      0                    1s
hamster-5f9fdfff5-lgw6z                0/1           ContainerCreating       0                    1s
hamster-5f9fdfff5-lgw6z               1/1            Running                       0                    3s

Watching the events of the hamster pod shows the pod being deleted, rescheduled, container created, and then set to the running state. Doing another describe of the hamster pod demonstrates that the container is running with updated CPU and memory resources.

$ kubectl describe po hamster-xxxxxxxxxx-xxxxx
Original deployment After VPA update
Name:         hamster-5f9fdfff5-2prd8
Namespace:    default
Priority:     0
. . . .
Labels:       app=hamster
              pod-template-hash=5f9fdfff5
. . . .
    Image:         k8s.gcr.io/ubuntu-slim:0.1
    . . .
    State:          Running
      Started:      Fri, 08 Jan 2021 11:44:55 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     50Mi

    Environment: 
    Mounts:
      . . . .
Events:         
Name:         hamster-5f9fdfff5-7c2cl
Namespace:    default
. . . .
Labels:       app=hamster
              pod-template-hash=5f9fdfff5
Annotations:  vpaObservedContainers: hamster
              vpaUpdates: Pod resources updated by hamster-vpa: container 0: cpu request, memory request

. . . .
    State:          Running
      Started:      Wed, 03 Feb 2021 19:10:23 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        587m
      memory:     262144k

Evictions: What Happens?

When you enable the update policy, what happens to inflight requests for pods that have been scheduled for eviction? A sighup signal is received, and the kubelet waits until the timeout you set on your pod before it sends a sigkill signal. This waiting period allows for the current request to complete, and new requests are handled by the other replicas.  The timeout is set in the PodSpecification with the terminationGracePeriodSeconds. Set this value large enough to account for the long-running process.

Obviously, you don’t want all of your pods getting evicted simultaneously, because requests would fail until pods came back online. To control the number of pods that are always available, you can deploy a PodDisruptionBudget (PDB). A PDB specifies the number of replicas that an application can tolerate having, relative to how many it’s intended to have.

The PodDisruptionBudget has the following fields:

  • A label selector to specify the set of pods to which it applies (.spec.selector).

  • A minimum available, which describes the number of pods from the set that must be available after the eviction, even in the absence of the evicted pod (.spec.minAvailable). The minimum available can be an absolute number or a percentage.

  • A maximum unavailable, which describes the number of pods from that set that can be unavailable after the eviction (.spec.maxUnavailable). It can be an absolute number or a percentage.

To better understand the PodDisruptionBudget, visit the following sections of the Kubernetes documentation:

Summary

The VPA is an excellent option for automatically scaling your pods.  However, using the VPA does come with the following caveats:

  • If they’re both monitoring CPU and memory resources, don’t deploy the VerticalPodAutoscaler and HorizontalPodAutoscaler (HPA) together.

  • You can deploy the VPA and HPA if the HPA is monitoring custom metrics, not including memory and CPU.

In this article, I only touched on the vertical pod autoscaler. This component helps you gain an understanding of the resources, CPU, and memory needed to run your service efficiently and reduce your overall resource costs. The VPA scales up and down, so you don't have to worry about scaling to the peak requirements and having to pay those costs 24/7.

In upcoming articles, I  discuss the other autoscalers. Taking advantage of these autoscalers improves the resiliency, high availability, and elasticity of your cloud native applications.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha