Operate

Observability

The GPU telemetry that matters — DCGM (NVIDIA) and ROCm-SMI (AMD) — plus a severity-ranked alert catalog and copy-ready Prometheus rules. Each alert links to its troubleshooting workflow.

Live telemetry

GPU telemetry reference

Export these with DCGM-exporter (NVIDIA) and the AMD/ROCm SMI exporter. Bands are general guidance — calibrate to the specific parts and workloads.

Metric	Signal	Healthy	Watch	Critical
NVIDIA GPU utilization DCGM_FI_DEV_GPU_UTIL Percent of time the GPU was busy. Low values on a training node usually mean a data or communication bottleneck, not spare capacity.	Compute	60–95%	<40% sustained	~0% while jobs pending
NVIDIA Framebuffer used DCGM_FI_DEV_FB_USED GPU memory in use. Approaching total capacity precedes out-of-memory kills.	Memory	<85% of total	>90%	>98%
NVIDIA GPU temperature DCGM_FI_DEV_GPU_TEMP Core temperature. Sustained highs trigger thermal throttling and shorten hardware life.	Thermal	<80°C	80–87°C	>90°C (throttle)
NVIDIA Power draw DCGM_FI_DEV_POWER_USAGE Instantaneous board power. Persistent capping can indicate power-limited performance.	Power	<TDP	at TDP sustained	power-capped throttle
NVIDIA ECC double-bit errors DCGM_FI_DEV_ECC_DBE_VOL_TOTAL Uncorrectable memory errors. Any nonzero value means data integrity risk and likely node drain.	Errors	0	—	≥1
NVIDIA Xid errors DCGM_FI_DEV_XID_ERRORS Driver-reported GPU errors. Certain Xids (e.g. 79, 48) indicate fatal hardware faults requiring reset/RMA.	Errors	0	transient/benign Xids	fatal Xid (79, 48, 94)
NVIDIA NVLink bandwidth DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL Aggregate NVLink throughput. Collapses during all-reduce point to fabric or NCCL issues.	Interconnect	near link rate	erratic	near 0 during collectives
NVIDIA SM clock DCGM_FI_DEV_SM_CLOCK Streaming-multiprocessor clock. Drops below base clock under load reveal thermal or power throttling.	Compute	≥ base clock	dipping under load	far below base (throttle)
NVIDIA Memory copy utilization DCGM_FI_DEV_MEM_COPY_UTIL Time spent moving data over the memory bus. High values with low compute suggest an input-bound pipeline.	Memory	balanced w/ compute	high w/ low GPU util	—
AMD GPU utilization amd_gpu_use_percent Percent GPU busy on Instinct/ROCm. Same interpretation as NVIDIA utilization.	Compute	60–95%	<40% sustained	~0% while jobs pending
AMD VRAM used amd_gpu_memory_used HBM in use. MI300X/MI325X have very large capacity but the same near-full failure mode.	Memory	<85% of total	>90%	>98%
AMD GPU temperature amd_gpu_temperature Edge/junction temperature. Sustained highs throttle clocks.	Thermal	<85°C	85–95°C	>100°C junction
AMD Power draw amd_gpu_power Board power. High-TDP MI300-class parts need headroom-checked facility power.	Power	<TDP	at TDP sustained	power-capped throttle
AMD Uncorrectable ECC amd_gpu_ecc_uncorrectable Uncorrectable HBM errors on ROCm. Nonzero means integrity risk and node drain.	Errors	0	—	≥1
AMD Infinity Fabric bandwidth amd_gpu_xgmi_bandwidth XGMI/Infinity Fabric throughput between GPUs. Drops during RCCL collectives indicate fabric problems.	Interconnect	near link rate	erratic	near 0 during collectives

Alert catalog

Severity

Vendor

16 alerts

Copy-ready Prometheus rules

Generated from the alerts above that carry an expression. Pin metric names to your exporters and tune thresholds before applying.

gpu-alerts.rules.yaml

groups:
  - name: gpu-platform.rules
    rules:
      - alert: GPUUnavailable
        expr: kube_node_status_allocatable{resource="nvidia_com_gpu"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "GPU unavailable"
      - alert: ECCUncorrectableError
        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[10m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "ECC uncorrectable error"
      - alert: FatalXidError
        expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fatal Xid error"
      - alert: ThermalThrottling
        expr: DCGM_FI_DEV_GPU_TEMP > 87
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Thermal throttling"
      - alert: LowGPUUtilization
        expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) < 40
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Low GPU utilization"
      - alert: HighGPUMemoryPressure
        expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High GPU memory pressure"