Migrating to Karpenter from Cluster Autoscaler on OKE

With the recent release of Karpenter support on OCI, many organizations are looking to begin using it as soon as possible. Karpenter brings benefits like resilience, cost savings, and scaling performance, all just by changing the node autoscaling solution for a cluster. It is vital to have a smooth migration with no downtime when moving from one autoscaler to another, especially in a live environment. 

When migrating from Cluster Autoscaler to Karpenter, new worker nodes are provisioned differently. For example, Cluster Autoscaler modifies the size of existing managed node pools. Karpenter Provider OCI, also referred to as KPO, provisions OCI compute instances from Karpenter resources that describe both the compute launch configuration and Kubernetes scheduling intent. 

This blog post shows a staged migration path for an Oracle Kubernetes Engine (OKE) cluster that already exists. The goal is to have a deliberate migration approach: prepare Identity and Access Management (IAM), install Karpenter Provider OCI, translate the current managed node pool configuration into OCINodeClass and NodePool resources, validate provisioning, then remove the old nodes gradually.

The changes when you move to Karpenter 

When using Cluster Autoscaler, meaning managed node pools, details like image, node shape, subnets, labels, and taints are tied to the configuration of the node pool and Cluster Autoscaler makes the decision which pool to scale. 

With Karpenter the node configuration is split to provide a clear separation of concerns: 

• OCINodeClass contains the OCI-specific launch details such as boot volume settings, shape configuration, image selection, tags, SSH Keys, kubelet and VNIC configuration. 

• NodePool defines scheduling/provisioning policy, so it contains details such as taints, startup taints, disruption behavior, labels, resource limits, and a reference to an OCINodeClass. 

This split is very important, so the new Karpenter resources can provision capacity to satisfy the same workload requirements as the existing managed node pools. 

Migration prerequisites 

The migration assumes: 

• An OKE cluster running Kubernetes v1.31 or higher (make sure to review the compatibility matrix for KPO and Kubernetes version) 

• OKE Managed Node Pool or Self-Managed nodes that are not managed by Karpenter 

• Existing VCN and subnets 

• Helm installed 

• Critical workloads protected by PDBs (pod disruption budgets) 

If you are using OCI VCN Native CNI, make sure the add-on is version 3.0.0 or later. Version 3.2.0 or later is highly recommended, because versions earlier than 3.2.0 require secondaryVnicConfigs[].ipCount to be 16 or less.

Migration sequence 

The migration approach is: 

1. Create the OCI IAM policies required by the KPO controller. 

2. Create the OCI IAM policies required by the KPO-launched nodes. 

3. Install Karpenter 

4. Inventory the existing OKE managed node pools 

5. Create the OCINodeClass resources that match the scheduling needs of the current workloads 

6. Create the NodePool resources 

7. Validate that the expected worker nodes can be provisioned by Karpenter 

8. Disable Cluster Autoscaler 

9. Remove the old nodes gradually (cordon the old nodes and drain them) 

10. Uninstall Cluster Autoscaler and its IAM policies

The order is important. In the proposed approach, the Cluster Autoscaler is still available in the cluster while Karpenter is installed and validated. The nodes from the managed node pools will be moved only after KPO has proven it can provision replacement nodes where the workloads can be moved.

1. Create IAM policies for Karpenter provider OCI 

The first step in the migration is to create the OCI IAM policies for the controller and for the nodes Karpenter provider OCI will launch. 

Karpenter Provider OCI runs as a Kubernetes deployment, and it uses workload identity to interact with OCI services. The workload identity policy looks like this: 

Allow any-user to <verb> <resource> in <location> where all { 
    request.principal.type = 'workload', 
    request.principal.namespace = '<namespace-name>', 
    request.principal.service_account = '<service-account-name>', 
    request.principal.cluster_id = '<cluster-ocid>' 
} 

The basic IAM statements needed for using Karpenter are the following: 

Allow any-user to manage instance-family in compartment <compartment-name> where all { ... }
Allow any-user to manage volumes in compartment <compartment-name> where all { ... }
Allow any-user to manage volume-attachments in compartment <compartment-name> where all { ... }
Allow any-user to manage virtual-network-family in compartment <compartment-name> where all { ... }
Allow any-user to inspect compartments in compartment <compartment-name> where all { ... }

Optional statements (feature-specific) 

Add feature-specific IAM statements only when the corresponding feature is used. 

• Capacity reservation 

Allow any-user to use compute-capacity-reservations in compartment <compartment-name> where all { ... }

• Compute Clusters 

Allow any-user to use compute-clusters in compartment <compartment-name> where all { ... }

• Cluster Placement Groups 

Allow any-user to use cluster-placement-groups in compartment <compartment-name> where all { ... }

• Defined tags 

Allow any-user to use tag-namespaces in compartment <compartment-name> where all { ... }

Example workload identity policy statement: 

Allow any-user to manage instance-family in compartment mycompartment where all { request.principal.type='workload', request.principal.cluster_id = 'ocid1.cluster...', request.principal.namespace = 'kube-system', request.principal.service_account = 'karpenter' }

Decide which namespace Karpenter Provider OCI will run in. You can use the kube-system namespace or a dedicated namespace, but the service account and namespace in the IAM policies must match the deployment. 

Learn more about the IAM policy setup using the official documentation: Granting IAM Permissions to Karpenter Provider for OCI 

2. Create IAM policies for nodes launched by KPO 

Nodes that are launched by Karpenter also need permission to register with the cluster. To enable this, you need to create a dynamic group that matches all the instances in the compartment(s) where nodes are provisioned: 

ALL {instance.compartment.id = '<compartment-ocid>'}

Then create an IAM policy for instances in the dynamic group to join the cluster: 

Allow dynamic-group <domain-name>/<dynamic-group-name> to {CLUSTER_JOIN} in compartment <compartment-name>

For detailed steps, refer to Oracle’s documentation: Dynamic Group and a Policy for Self-Managed Nodes. 

3. Install Karpenter Provider OCI 

• Add the helm repository by using: 

helm repo add karpenter-provider-oci https://oracle.github.io/karpenter-provider-oci/charts

• Download the latest information about the chart from the repository: 

helm repo update karpenter-provider-oci

• Check the available chart versions: 

helm search repo karpenter-provider-oci/karpenter --versions

• Prepare a Helm values file with the required settings. Ensure it includes all mandatory fields and any additional settings you want to override. A minimal values file looks like this: 

settings:
  # -- [required] Cluster compartment OCID.
  clusterCompartmentId: "<your-cluster-compartment-ocid>"
  # -- [required] Cluster's VCN compartment OCID.
  vcnCompartmentId: "<your-vcn-compartment-ocid>"
  # -- set this to true for a cluster run with OciVcnIpNative 
  ociVcnIpNative: false
  # -- [required] API server endpoint(privateIP) for worker nodes to communicate with the Kubernetes API server.
  apiserverEndpoint: "<api-server-endpoint-ip>"
 
By default, the default node affinity prevents scheduling controller pods on Karpenter-managed nodes, and it spreads the controller pods across multiple nodes. Review the affinity so Karpenter can run on the expected non-Karpenter nodes. 

• Install KPO helm repository: 

helm install karpenter karpenter-provider-oci/karpenter \ 
 --version <chart-version> \ 
 --values <path-to-values.yaml> \ 
 --namespace <karpenter-namespace> \ 
 --create-namespace

• Verify the deployment: 

kubectl -n <namespace> rollout status deploy/karpenter
kubectl -n <namespace> get pods

• Upgrade later if needed: 

helm upgrade karpenter ./chart \
  --namespace karpenter \
  --values values.yaml

If validation fails and Karpenter needs to be removed to perform a rollback, KPO can be removed: 

helm uninstall karpenter --namespace <namespace>

4. Inventory the existing managed node pools 

Before creating the manifests for Karpenter resources, get the current configuration of managed node pools that currently allow workloads to run. 

oci ce node-pool get --node-pool-id <node-pool-ocid>

The response is larger than what is needed to perform the migration. You need to focus on the attributes that affect how nodes are launched and how workloads are scheduled.

{
  "data": {
    "compartment-id": "ocid1.compartment...",
    "id": "ocid1.nodepool...",
    "initial-node-labels": [
      {
        "key": "workload-tier",
        "value": "apps"
      },
      {
        "key": "billing-team",
        "value": "platform"
      }
    ],
    "name": "apps-nodepool",
    "node-config-details": {
      "defined-tags": {
        "Operations": {
          "CostCenter": "42"
        }
      },
      "freeform-tags": {
        "nodepool": "apps"
      },
      "is-pv-encryption-in-transit-enabled": true,
      "kms-key-id": "ocid1.key.oc1...",
      "node-pool-pod-network-option-details": {
        "cni-type": "OCI_VCN_IP_NATIVE",
        "max-pods-per-node": 31,
        "pod-nsg-ids": null,
        "pod-subnet-ids": [
          "ocid1.subnet.oc1..."
        ]
      },
      "nsg-ids": [],
      "placement-configs": [
        {
          "availability-domain": "RiYU:US-ASHBURN-AD-1",
          "fault-domains": [],
          "subnet-id": "ocid1.subnet.oc1..."
        },
        {
          "availability-domain": "RiYU:US-ASHBURN-AD-2",
          "fault-domains": [],
          "subnet-id": "ocid1.subnet..."
        }
      ],
      "size": 3
    },
    "node-image-id": "ocid1.image.oc1.iad...",
    "node-image-name": "Oracle-Linux-8.10-2025.09.16-0-OKE-1.34.1-1330",
    "node-metadata": {
      "user_data": "base64-cloud-init"
    },
    "node-shape": "VM.Standard.E5.Flex",
    "node-shape-config": {
      "memory-in-gbs": 32.0,
      "ocpus": 4.0
    },
    "node-source": {
      "image-id": "ocid1.image.oc1.iad...",
      "source-name": "Oracle-Linux-8.10-2025.09.16-0-OKE-1.34.1-1330",
      "source-type": "IMAGE"
    },
    "node-source-details": {
      "boot-volume-size-in-gbs": 100,
      "image-id": "ocid1.image.oc1.iad...",
      "source-type": "IMAGE"
    },
    "ssh-public-key": "ssh-ed25519 AAAA...",
    "subnet-ids": [
      "ocid1.subnet.oc1.iad..."
    ]
  }
}

Mapping existing node pools to Karpenter resources 

The existing managed node pool is the source of truth. Start with one-to-one migration, then add additional allowed ones, shapes, and disruption behavior once the initial cutover is stable. 

More details about OCINodeClass can be found in the KPO API reference or by running kubectl explain ocinodeclasses.oci.oraclecloud.com.spec. 

Alternatively, details about NodePool resources can be found using the kubectl explain nodepools.karpenter.sh.spec command. 

Use the following mapping rules when you translate that node pool into Karpenter resources: 

Cloud-init needs special attention. In the managed node pools setup, kubelet configuration was embedded in user_data. In Karpenter configuration, these settings need to be moved to Karpenter resources: 

• move --kube-reserved into OCINodeClass.spec.kubeletConfig.kubeReserved

• move --system-reserved into OCINodeClass.spec.kubeletConfig.systemReserved

• move --max-pods into OCINodeClass.spec.kubeletConfig.maxPods

• move --pods-per-core into OCINodeClass.spec.kubeletConfig.podsPerCore

• move --node-labels into NodePool.spec.template.metadata.labels or OCINodeClass.spec.kubeletConfig.nodeLabels

• move --register-with-taints into NodePool.spec.template.spec.taints or startupTaints

• keep package installation, filesystem preparation, OS tuning, or bootstrap hooks in preBootstrapInitScript and postBootstrapInitScript

• use metadata.user_data only when the full control of cloud-init is required. If you do that, make sure it remains compatible with the OKE bootstrap.

Using OCI CLI, the node-pool does not expose taints as a field. If your current nodes rely on taints, most probably the taints were added through kubelet flags, or cloud-init and should be moved into the NodePool resource.

5. Create the OCINodeClass 

The OCINodeClass resource should have the details that were previously defined in the OKE managed node pool and by any custom user-data script. 

apiVersion: oci.oraclecloud.com/v1beta1
kind: OCINodeClass
metadata:
  name: apps-nodeclass
spec:
  # Keep the same flex shape configuration during the initial migration.
  shapeConfigs:
    - ocpus: 4
      memoryInGbs: 32

  # Move node instance tags here. 
  freeformTags:
    nodepool: "apps"
    managed-by: "karpenter"
  definedTags:
    Operations:
      CostCenter: "42"

  # Move SSH access here.
  sshAuthorizedKeys:
    - "ssh-ed25519 AAAA..."
 
  # Move kubelet flags out of cloud-init and into kubeletConfig. 
  kubeletConfig:
    systemReserved:
      cpu: "500m"
      memory: "1Gi"
    kubeReserved:
      cpu: "500m"
      memory: "1Gi"
    evictionHard:
      memory.available: "5%"
      nodefs.available: "10%"
    evictionSoft:
      memory.available: "10%"
    evictionSoftGracePeriod:
      memory.available: "1m"
 
  volumeConfig:
    bootVolumeConfig:
      imageConfig:
        imageType: OKEImage
        imageId: YOUR_OKE_IMAGE_OCID
      volumeAttribute:
        sizeInGBs: 100
        vpusPerGB: 20
        kmsKeyConfig:
          kmsKeyId: YOUR_KMS_KEY_OCID
      pvEncryptionInTransit: true 

  networkConfig:
    primaryVnicConfig:
      subnetConfig:
        subnetId: YOUR_WORKER_SUBNET_OCID
 
  # Prefer these for bootstrap hooks that are not kubelet flags.
  preBootstrapInitScript: "base64-pre-bootstrap-script"
  postBootstrapInitScript: "base64-post-bootstrap-script"

  # Use metadata.user_data only when you need full cloud-init control.
  # If you set it, the payload must remain compatible with OKE bootstrap.
  # metadata:
  #   user_data: "base64-cloud-init-that-still-runs-oke-bootstrap"

If the cluster uses OciIpNativeCni, the OCINodeClass is similar to the one used for Flannel, but it must be extended with a secondary VNIC for pod networking, as shown in the following example.

apiVersion: oci.oraclecloud.com/v1beta1
kind: OCINodeClass
metadata:
  name: apps-nodeclass-ipnative
spec:
  shapeConfigs:
    - ocpus: 4
      memoryInGbs: 32
  volumeConfig:
    bootVolumeConfig:
      imageConfig:
        imageType: OKEImage
        imageId: YOUR_OKE_IMAGE_OCID
  networkConfig:
    primaryVnicConfig:
      subnetConfig:
        subnetId: YOUR_WORKER_SUBNET_OCID
    secondaryVnicConfigs:
      - subnetConfig:
          subnetId: YOUR_POD_SUBNET_OCID
        ipCount: 16

Optional: for automatic OKE image selection use image filters

There is also an alternative to using the explicit imageId. You can use imageFilter with imageType: OKEImage and in this mode Karpenter resolves the OKE image, and you can ensure that you always have the latest version available of an image.

Use kubectl apply –f <file> to create the OCINodeClass.

More info about the configuration of the OCINodeClass: Karpenter Advanced Use Cases Docs.

6. Create the NodePool 

The NodePool contains the scheduling-facing parts: labels, instance shape requirements, taints, fault domains, availablity domains, disruption policy, lifecycle behavior, limits, and weight.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: apps-nodepool
spec:
  template:
    metadata:
      labels:
        workload-tier: apps
    spec:
      expireAfter: Never
      nodeClassRef:
        group: oci.oraclecloud.com
        kind: OCINodeClass
        name: apps-nodeclass
      taints:
        - key: workload-tier
          value: apps
          effect: NoSchedule
      startupTaints:
        - key: bootstrap.oraclecloud.com/in-progress
          value: "true"
          effect: NoSchedule
      requirements:
        - key: kubernetes.io/os
          operator: In
          values:
            - linux
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand
        - key: oci.oraclecloud.com/instance-shape
          operator: In
          values:
            - VM.Standard.E5.Flex
        - key: topology.kubernetes.io/zone
          operator: In
          values:
            - Uocm:PHX-AD-1
            - Uocm:PHX-AD-2
        - key: oci.oraclecloud.com/fault-domain
          operator: In
          values:
            - FAULT-DOMAIN-1
            - FAULT-DOMAIN-2
            - FAULT-DOMAIN-3
      terminationGracePeriod: 120m
  disruption:
    budgets:
      - nodes: 5%
    consolidateAfter: 60m
    consolidationPolicy: WhenEmpty
  limits:
    cpu: 128
    memory: 512Gi
  weight: 20

Use kubectl apply –f <file> to create the NodePool. 

More info about the configuration of the NodePool: Karpenter Advanced Use Cases Docs.

7. Validation and rollout strategy 

After KPO has been deployed and the Karpenter resources are created, verify the logs of the controller:

kubectl logs -f -n <karpenter-namespace> -l app.kubernetes.io/name=karpenter

Then, validate that Karpenter can provision nodes for workloads that were previously running on Cluster Autoscaler-managed nodes.

8. Disable Cluster Autoscaler 

If Cluster Autoscaler is installed through the OKE Cluster Add-on, the configuration numOfReplicas can be updated to 0. 

Otherwise, this command can be used to scale the deployment to 0: 

kubectl scale deploy/cluster-autoscaler -n kube-system --replicas=0

9. Remove Cluster Autoscaler-managed nodes gradually 

The old managed node pools can be scaled down to a minimum size that supports Karpenter and other critical services. As old nodes are cordoned and drained, the pods become unschedulable and Karpenter provisions new nodes for them. 

It would be preferred to use small batches. If there are no PodDisruptionBudgets set or are not enough replicas running, draining the nodes can make the workloads unavailable, so it is recommended to monitor the transition carefully. 

For greater control, cordon and drain the nodes manually, then remove them from the cluster: 

• kubectl cordon <node-name>

• kubectl drain <node-name>

• remove the nodes 

Recommendations during cutover: 

• drain a small number of nodes at a time 

• watch failed placements and pending pods 

• keep the non-Karpenter nodes where KPO controller is running

Rollback and safety considerations 

If at any point, rollback is required, this can be achieved by deleting the OCINodeClass and NodePool resources and uninstalling the helm chart using the helm uninstall karpenter --namespace <namespace> command and reenabling the Cluster Autoscaler by scaling the deployment replica. 

It is recommended to use non-production testing or a blue/green approach when possible.

10. Uninstall Cluster Autoscaler 

Once the migration is successful and the old nodes managed by Cluster Autoscaler have been removed, uninstall Cluster Autoscaler. 

If it is addon-managed, disable the addon. For example, using OCI CLI this can be achieved with oci ce cluster disable-addon --addon-name ClusterAutoscaler --cluster-id <cluster-ocid> --is-remove-existing-add-on <true|false> command.

If Cluster Autoscaler was installed through Helm: 

helm uninstall <release-name> --namespace <namespace-name>

Then the IAM policies for Cluster Autoscaler can be removed: 

• Remove the OCI IAM policies 

• Remove the OCI IAM dynamic group if present, if Cluster Autoscaler used instance principal authentication

Conclusion 

A migration from Cluster Autoscaler to Karpenter Provider OCI is considered successful if the intent was preserved, but it allows greater flexibility in configuration saving capacity and costs. The managed node pools configuration tells you the requirements of your workloads, and these need to be translated to Karpenter resources – OCINodeClass and NodePool. 

Try to migrate from Cluster Autoscaler to Karpenter matching the same node configuration – labels, shapes, taints, and so on. Once the cluster is stable using Karpenter, you have the option to tune disruption behavior or broaden scheduling options. 

Resources

• GitHub repository: https://github.com/oracle/karpenter-provider-oci 

• Documentation: https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/conteng-kpo.htm