Live Telemetry
Current fleet state from the configured telemetry provider — per-cluster utilization, thermals, power, and ECC, with alerts assessed against the reference bands and deep-linked to their troubleshooting workflows.
Provider sample — a synthetic snapshot derived from the sample fleet + GPU catalog (no cluster required). Set TELEMETRY_PROVIDER=prometheus to scrape a real DCGM/ROCm Prometheus.
1,048
9 clusters
65%
fleet-wide
379 kW
board power
1
1 clusters
104
3 clusters
1 / 0
uncorrectable / capped
Uncorrectable ECC error
criticalprod-train-us · prod-train-us-node-16 · gpu01 uncorrectable ECC error(s) — data-integrity risk; drain and diagnose the GPU.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu492% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu591% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu692% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu193% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu390% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu293% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu393% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-05 · gpu392% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-07 · gpu793% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-09 · gpu290% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-09 · gpu391% framebuffer used.
+ 93 more firing…
| Cluster | Vendor | Utilization | Peak temp | Power | Faults | State |
|---|---|---|---|---|---|---|
prod-train-us H100 SXM5 · 256 GPU | NVIDIA | 82% | 79°C | 155.1 kW | ECC 1 | critical |
prod-train-eu H100 SXM5 · 128 GPU | NVIDIA | 77% | 76°C | 74.2 kW | — | watch |
prod-train-us2 Instinct MI300X · 64 GPU | AMD | 68% | 79°C | 36.7 kW | — | watch |
prod-inf-us L40S · 192 GPU | NVIDIA | 65% | 71°C | 49.7 kW | — | healthy |
prod-inf-eu L40S · 96 GPU | NVIDIA | 57% | 67°C | 23 kW | — | healthy |
rag-us A100 80GB SXM · 64 GPU | NVIDIA | 74% | 74°C | 20.7 kW | — | healthy |
vision-us L4 · 120 GPU | NVIDIA | 53% | 66°C | 5.7 kW | — | healthy |
dev-us A30 · 48 GPU | NVIDIA | 22% | 52°C | 3.4 kW | — | healthy |
legacy-us A100 40GB PCIe · 80 GPU | NVIDIA | 38% | 58°C | 10.9 kW | — | watch |
Snapshot 2026-10-19 23:26:12 UTC · bands from the observability reference.