criticalHardware & health

ECC / Xid errors

The GPU reports memory (ECC) errors or driver-level Xid faults.

Scope
Hardware health. Distinguish correctable (watch) from uncorrectable/fatal (act now).
Check recent changes
  • ·Thermal/power stress
  • ·Aging hardware
  • ·Firmware change
Diagnostics

Xid codes

bash
dmesg | grep -i xid

Expect: Map codes: 79 (off bus), 48 (DBE), 94 (contained ECC) are fatal.

ECC counts

bash
nvidia-smi -q -d ECC | grep -A2 'Uncorrectable'

Expect: Uncorrectable count of 0 is healthy; any nonzero ⇒ drain.

Likely causes & fixes
  • high likelihoodUncorrectable ECC / fatal Xid ⇒ failing hardware

    Fix · Drain immediately; requeue jobs from checkpoint; RMA.

  • medium likelihoodRising correctable ECC ⇒ deteriorating card

    Fix · Schedule maintenance; watch the trend.

  • low likelihoodThermal/power-induced transient

    Fix · Fix cooling/power; re-observe.

Validation
  • ✓No new uncorrectable ECC / fatal Xids after remediation
Escalation

Uncorrectable ECC or fatal Xid → vendor RMA.

Data to collect first
  • ·ECC counters (single vs. double bit)
  • ·Xid codes from dmesg
  • ·Temperature/power at fault time
Related alerts
ecc-uncorrectablexid-fatalcorrectable-ecc