high riskUpgrades

GPU driver upgrade

Upgrade the NVIDIA/AMD driver on GPU nodes, one at a time, via the GPU Operator, validating before moving on.

Change scope

Per node pool / rolling across GPU workers

Maintenance impact

GPU workloads drained per node; capacity temporarily reduced.

Prerequisites
  • ·A validated driver+kernel baseline
  • ·Spare capacity to drain one node at a time
  • ·Change window approved
Pre-checks
  • ·No running training jobs pinned to the target node
  • ·Current driver + Xid/ECC health clean
Procedure
  1. 1

    Cordon and drain the target node.

    bash
    kubectl cordon $NODE
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
  2. 2

    Let the GPU Operator roll the new driver (or update the ClusterPolicy/driver version).

  3. 3

    Validate GPUs on the node, then uncordon.

    bash
    kubectl run cuda-check --rm -i --restart=Never --overrides='{"spec":{"nodeName":"'$NODE'"}}' --image=nvidia/cuda:12.4.1-base-ubuntu22.04 --limits=nvidia.com/gpu=1 -- nvidia-smi
kubectl uncordon $NODE
  4. 4

    Repeat for the next node once healthy.

Validation
  • ✓`nvidia-smi` reports the new driver on every node
  • ✓A GPU test pod schedules and runs
  • ✓No new Xid/ECC errors after soak
Rollback

Pin the previous driver version in the operator and re-roll the affected node; uncordon once validated.

Communication

Announce the window and per-pool capacity reduction; notify teams with jobs on the affected pool.