criticalNetworking & collectives

NCCL / RCCL errors or timeouts

Distributed training fails or hangs on collective ops (all-reduce), often with an NCCL/RCCL timeout.

Scope
Multi-GPU/multi-node communication. Isolate whether it's a rank, a link, or the fabric.
Check recent changes
  • ·Fabric/switch changes
  • ·New nodes
  • ·MTU/PFC config changes
  • ·Topology changes
Guided diagnosis

Does it fail immediately at startup, or hang after running a while?

Diagnostics

Enable collective debug logging

bash
export NCCL_DEBUG=INFO   # RCCL_DEBUG=INFO for AMD
# Rerun; the logs name the ring/step where it stalls.

Check the interconnect

bash
nvidia-smi nvlink -s   # or rocm-smi --showtopo for XGMI

Expect: All links active at expected bandwidth.

Likely causes & fixes
  • high likelihoodFabric misconfig (MTU / PFC / ECN) causing loss

    Fix · Align MTU end-to-end; fix lossless-fabric config.

  • high likelihoodA dead/straggler rank or failed link

    Fix · Identify and cordon the node; let the job reschedule.

  • medium likelihoodWrong NCCL interface/topology selection

    Fix · Set NCCL_SOCKET_IFNAME / topology hints correctly.

Validation
  • ✓All-reduce micro-benchmark hits expected bandwidth
  • ✓Training resumes and advances steps
Escalation

Hardware link failure → RMA; systemic fabric issues → network engineering.

Data to collect first
  • ·NCCL/RCCL debug logs
  • ·Per-link bandwidth (NVLink/XGMI)
  • ·Fabric counters (PFC pauses, drops)
  • ·MTU end-to-end
Related alerts
nvlink-failurerdma-failuretraining-stalled