Operate

Troubleshooting

Start from a symptom. Each workflow gives you scope, diagnostic commands, ranked causes, validation, and escalation — and the flagship symptoms include a guided decision walker that narrows to a likely cause.

Detection & drivers
critical Guided

GPU not detected in Kubernetes

A GPU node reports zero allocatable nvidia.com/gpu (or amd.com/gpu), so nothing can schedule on it.

Open workflow
high

Container can't access the GPU

The pod schedules onto a GPU node but the process can't see or use the device.

Open workflow
high

Driver install failure

The GPU driver (or operator driver pod) fails to build or load.

Open workflow
medium

CUDA / ROCm version mismatch

The app fails with a CUDA/ROCm version or 'no kernel image' error.

Open workflow
Scheduling
high Guided

GPU pod stuck Pending

A pod requesting a GPU won't schedule and sits in Pending.

Open workflow
medium

MIG configuration problem

MIG instances don't appear as allocatable, or the wrong profile is exposed.

Open workflow
Performance
medium

Low GPU utilization

GPUs sit underutilized while work is queued or running — usually the pipeline, not the GPU.

Open workflow
high

GPU out-of-memory (OOM)

The workload is killed with a CUDA/HIP out-of-memory error.

Open workflow
high

Training job hangs

A training run stops advancing steps with no crash — throughput goes to zero.

Open workflow
Hardware & health
medium

Thermal throttling

Clocks drop under load because the GPU is too hot.

Open workflow
critical

ECC / Xid errors

The GPU reports memory (ECC) errors or driver-level Xid faults.

Open workflow
Networking & collectives
critical Guided

NCCL / RCCL errors or timeouts

Distributed training fails or hangs on collective ops (all-reduce), often with an NCCL/RCCL timeout.

Open workflow
Storage
medium

Storage bottleneck

GPUs stall waiting on storage — checkpoints or dataset reads can't keep up.

Open workflow
Serving
high

Inference latency spikes

Serving p99 latency exceeds the SLO.

Open workflow