highScheduling

GPU pod stuck Pending

A pod requesting a GPU won't schedule and sits in Pending.

Scope
Usually a scheduling constraint, not a hardware fault. Start from the scheduler's own explanation.
Check recent changes
  • ·New workload with different requests
  • ·Taints added to GPU nodes
  • ·Quota changes
Guided diagnosis

What does `kubectl describe pod` say under Events?

Diagnostics

Ask the scheduler why

bash
kubectl describe pod <pod> | sed -n '/Events/,$p'

Expect: A message like 'Insufficient nvidia.com/gpu' or a taint mismatch.

Is there any allocatable GPU?

bash
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'

Expect: At least one node with free GPUs ≥ the request.

Likely causes & fixes
  • high likelihoodNo node has enough free GPUs

    Fix · Free capacity, scale out, or reduce the request.

  • high likelihoodToleration missing for the GPU node taint

    Fix · Add the matching toleration to the pod.

  • medium likelihoodRequest exceeds any single node's GPU count

    Fix · Split the job or use nodes with more GPUs.

  • medium likelihoodNamespace GPU quota exhausted

    Fix · Raise the quota or reclaim usage.

Validation
  • ✓Pod transitions to Running and lands on a GPU node
Escalation

If capacity is genuinely exhausted cluster-wide, escalate for more GPUs.

Data to collect first
  • ·Pod scheduling events
  • ·Node allocatable GPUs vs. requests
  • ·Taints/tolerations
  • ·Namespace quota
Related alerts
pending-gpu-pods