Live Telemetry
Current fleet state from the configured telemetry provider — per-cluster utilization, thermals, power, and ECC, with alerts assessed against the reference bands and deep-linked to their troubleshooting workflows.
Provider sample — a synthetic snapshot derived from the sample fleet + GPU catalog (no cluster required). Set TELEMETRY_PROVIDER=prometheus to scrape a real DCGM/ROCm Prometheus.
1,048
9 clusters
65%
fleet-wide
379 kW
board power
1
1 clusters
87
3 clusters
0 / 3
uncorrectable / capped
GPU over thermal limit
criticalprod-train-us · prod-train-us-node-26 · gpu592°C ≥ 90°C — thermal throttling; check cooling and airflow.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu293% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu490% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu093% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu193% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu090% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu293% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu394% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu792% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-05 · gpu291% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-07 · gpu093% framebuffer used.
Clock throttling (power-limited)
warningprod-train-us · prod-train-us-node-07 · gpu6Board capped at 637W / 700W — performance-limited.
+ 76 more firing…
| Cluster | Vendor | Utilization | Peak temp | Power | Faults | State |
|---|---|---|---|---|---|---|
prod-train-us H100 SXM5 · 256 GPU | NVIDIA | 82% | 92°C | 154.5 kW | throttle 3 | critical |
prod-train-eu H100 SXM5 · 128 GPU | NVIDIA | 78% | 77°C | 74.7 kW | — | watch |
prod-train-us2 Instinct MI300X · 64 GPU | AMD | 70% | 81°C | 37.1 kW | — | watch |
prod-inf-us L40S · 192 GPU | NVIDIA | 64% | 71°C | 49.3 kW | — | watch |
prod-inf-eu L40S · 96 GPU | NVIDIA | 57% | 68°C | 23.1 kW | — | healthy |
rag-us A100 80GB SXM · 64 GPU | NVIDIA | 73% | 74°C | 20.4 kW | — | healthy |
vision-us L4 · 120 GPU | NVIDIA | 55% | 66°C | 5.8 kW | — | healthy |
dev-us A30 · 48 GPU | NVIDIA | 23% | 51°C | 3.5 kW | — | healthy |
legacy-us A100 40GB PCIe · 80 GPU | NVIDIA | 37% | 58°C | 10.8 kW | — | healthy |
Snapshot 2026-07-26 18:59:29 UTC · bands from the observability reference.