We are announcing OCI Kubernetes Engine (OKE) GPU Host Repair, a feature that both makes faults discoverable and coordinates repairs directly through the Kubernetes API. OKE surfaces host health and maintenance information through Kubernetes, prepares affected nodes for repair, coordinates the applicable Compute action, and restores capacity. 

A GPU device can report errors, an RDMA link can degrade, and a bare metal host can need physical repair, all while the worker node representing the host still appears ready in a Kubernetes cluster. The impact of these failures ranges from degraded performance to a host becoming fully unable to run your workloads. For teams running GPU accelerated workloads on OKE, discovering these faults is itself a challenging problem to solve. Additionally, responding to these failures means coordinating steps across Kubernetes and OCI Compute. Operators identify the affected node, move workloads, initiate a host repair action, track the repair, and restore cluster capacity when the repair is complete. Following these steps manually, or investing in automation, takes time and attention away from value driving activities. 

Enable automatic repair of bare metal GPU hosts

Kubernetes-level health signals

The GPU Health Monitoring add-on installs the controller and CRDs to detect and report supported GPU and RDMA health conditions on selected worker nodes. Customers use a GpuHealthMonitoringConfig to select which nodes to monitor. Passive checks are OCI-specific GPU checks used to observe health signals without interrupting normal operation. Active checks are performance-related checks that are customer-scheduled on idle nodes. These checks can expose performance issues and, in conjunction with passive checks, may reveal failures under load that passive monitoring alone might miss. Together, passive and active checks give operators more evidence when deciding whether a node needs attention.

OKE exposes health signals as Kubernetes node conditions and records faults in a per-node NodeHealthReport custom resource. Operators can inspect the NodeHealthReport for faults, recommended actions, related maintenance information, and repair progress. OKE uses the signal, and the applicable repair policy, to determine what happens next. The GPU host repair guide describes the currently documented monitoring and repair resources.Kubernetes-native repair

Kubernetes-native repair

OKE GPU Host Repair takes a Kubernetes native approach to tracking, reporting, and taking action on host issues. The GPU Health Monitoring add-on intakes host-level metrics sources and publishes them as Kubernetes node statuses. Kubernetes custom resource definitions introduce repair policy and host health into the cluster: a NodeRepairConfig custom resource allows you to select nodes for repair and sets guardrails, and a per-node NodeHealthReport records faults, maintenance events, and repair progress. Finally, the OKE-managed Node Repair Controller watches those node conditions and health reports. When a repair qualifies according to the NodeRepairConfig, it coordinates workload eviction through cordoning and draining, and then takes the appropriate OCI Compute action: a reboot or a terminate. Operators can inspect the policy and follow the repair with familiar Kubernetes tools.

Three paths to recovery

The feature supports three flows: user-initiated repair, instance recovery, and maintenance event repair, and includes safety measures to reduce workload disruption. 

User-initiated repair gives an operator a direct way to send a node back for repair based on customer-identified failures. When a customer has evidence of a host problem, even if automated telemetry has not identified an eligible fault, an authorized operator marks the affected node for repair using a simple Kubernetes label. OKE then taints the node to prevents new pods from landing on it, attempts to evict existing workloads, reports the degraded host condition to Compute, and terminates the instance. If the node is part of a managed node pool, the pool will restore its configured capacity with a replacement node when the repair is complete. The operator makes the repair decision, while OKE coordinates the steps that follow. Live user-initiated repair requires tenancy enablement through Oracle Support. Read the configuration steps.

GPU instance recovery offers an automated path to address GPU health signals that call for a reboot. A team selects the managed GPU nodes covered by its policy and enables health monitoring. When a qualifying signal recommends REBOOT and the OKE repair controller determines this it is a persistent failure signal, OKE initiates the repair: it cordons and attempts to drain the node adhering to pod disruption budgets, then reboots its Compute instance. A cooldown period limits repeated reboot attempts if the same fault persists. Signals marked CUSTOMER_ACTION_REQUIRED remain visible for investigation and do not trigger an automated reboot. For more information, read the recovery criteria.

Maintenance event repair recovery offers an automated path to address GPU faults indicated by OCI Compute instance maintenance events. These faults come from the Oracle Cloud Agent (OCA) Compute RDMA GPU Monitoring plugin that enables host level GPU/RDMA metrics. OKE observes these host level faults and tracks related instance maintenance events that signal the need for a repair. When Compute identifies host work that requires action and publishes the corresponding maintenance event, OKE capture and presents the event and its required action alongside the node’s health information. OKE supports two types of maintenance event repairs:

  • Terminating repairs that require nodes to be terminated and replace. For an event with a TERMINATE action, the instance is terminated to allow the host to be repaired. For nodes deployed to a managed node pool, when the repair is complete OKE works to restore the configured capacity.
  • Non-terminating repairs that require nodes to be stopped and restarted. For an event with a STOP action, the instance is stopped, OKE tracks the repair to completion, and the node is returned to a schedulable state. 

The repair controller coordinates workload evacuation with the event, so operators can follow one repair workflow across the node and its underlying instance. The Compute maintenance event documentation explains how those events are tracked.

Control before action

GPU workloads can have long-running jobs, large checkpoints, and limited spare capacity, all of which influence when you want to take action on a node – even an unhealthy one. A NodeRepairConfig is the customer node repair configuration contract used to selects the nodes covered by a repair policy and define guardrails to set limits on how repairs proceed. Teams can limit the number of parallel repairs and stop new repairs when too many nodes are unhealthy. They can also choose how long OKE waits for pod eviction and whether or not to proceed with an action if an eviction fails. Draining uses the Kubernetes Eviction API and respects Kubernetes pod eviction controls, including pod disruption budgets. 

apiVersion: oci.oraclecloud.com/v1beta1
kind: NodeRepairConfig
metadata:
  name: example-node-repair-config
spec:
  nodeSelector:
    matchLabels:
      example.com/gpu-repair: "enabled"
  policy:
    dryRun: false
    evictionGracePeriodMinutes: 30
    forceActionAfterGracePeriod: false
    maxUnhealthyNodeThreshold: 1
    maximumParallelNodeRepairs: 1
    pauseBeforeRepair: true

The dryRun setting lets teams inspect a proposed repair before enabling live action. In dryRun mode, OKE records the proposed workflow in the NodeHealthReport without cordoning or draining the node, or asking Compute to reboot or terminate the instance. This approach enables operators to review node selection, health signals, and policy behavior on a controlled test node before moving to live repair.

The pauseBeforeRepair setting lets customers enable a controlled approval point before the node repair controller begins any disruptive work. This pause allows customers to verify whether the reported condition is still valid, collect diagnostics, or coordinate with workload owners. While the repair is paused, the node remains schedulable. After verification is complete, customers provide a response that informs the controller whether to cancel or proceed with the repair. 

Each flow leaves a visible trail in Kubernetes. Node conditions show current monitoring state, while the NodeHealthReport shows the selected action, whether the run was dry or live, and its result. A Completed result means the requested workflow finished. Operators should still verify that the node is Ready, that current health results no longer report the fault, and that affected applications are healthy. Restoring a replacement node also depends on available GPU capacity.

apiVersion: oci.oraclecloud.com/v1beta1
kind: NodeHealthReport
metadata:
  creationTimestamp: "2026-09-15T07:23:29Z"
  generation: 1
  name: node-4ad8f377-f9bd-4a3a-bb26-f1fc11ab1373
  resourceVersion: "6394911"
  uid: e1d3084e-1046-4914-8d02-246778ff7532
spec:
  instanceRef:
    id: ocid1.instance.oc1.iad.anuwcljrh4gjgpycbnlllbdggqmqlbbgzforp7v2zahslzihonfymmsf5s5a
  nodeRef:
    name: 10.0.10.104
    uid: 4ad8f377-f9bd-4a3a-bb26-f1fc11ab1373
status:
  lastRepairAttempt:
    completedAt: "2026-09-21T17:29:10Z"
    dryRun: true
    repairAction: Reboot
    repairId: noderepair:b5d83e2e-85d4-4aaf-bccc-9a4690fbcd84
    result: Completed
    selectedSignalIds:
    - gpuFaultMonitor-CHC-link-check-2026-09-21T17-19-08.52592286Z
  repair:
    admission:
      admissionDetails:
        admittedNodeSelector:
          matchLabels:
            oci.com/mer-repair: "true"
        admittedPolicy:
          dryRun: true
          evictionGracePeriodMinutes: 1
          forceActionAfterGracePeriod: true
          maxUnhealthyNodeThreshold: 1
          maximumParallelNodeRepairs: 1
          pauseBeforeRepair: false
        admittedPolicyRef:
          generation: 6
          name: mec-e2e-terminate
          uid: 1474bedf-eda5-4591-b584-e05723fb3db7
      state: Admitted
    dryRun: true
    execution:
      currentPhaseStartedAt: "2026-09-21T17:29:10Z"
      repairPhase: Completed
    pausedDurationSeconds: 0
    repairAction: Reboot
    repairId: noderepair:b5d83e2e-85d4-4aaf-bccc-9a4690fbcd84
    selectedSignalIds:
    - gpuFaultMonitor-CHC-link-check-2026-09-21T17-19-08.52592286Z
    startedAt: "2026-09-21T17:29:08Z"
    terminalState: Completed
  signals:
    clusterHealthCheck:
    - action: REBOOT
      faultCode: link_check
      faultDescription: RDMA link check failed - link parameters do not meet expected values
      firstObservedAt: "2026-09-21T17:19:08Z"
      id: gpuFaultMonitor-CHC-link-check-2026-09-21T17-19-08.52592286Z
      lastObservedAt: "2026-09-21T19:53:38.595965324Z"

Getting started

OKE GPU Host Repair supports eligible managed GPU worker nodes in enhanced clusters running Kubernetes 1.32 or later, subject to supported shapes and regional availability. To get started, review the GPU Health Monitoring configuration and repair guide, select a test node, and run its repair policy in dryRun mode.

Further reading