How collective communication (NCCL/RCCL) works
Distributed training synchronizes gradients with collective operations like all-reduce, run by NCCL (NVIDIA) or RCCL (AMD).
In synchronous data-parallel training, every GPU computes gradients on its shard, then all GPUs exchange and average them via an all-reduce. NCCL/RCCL implement these collectives efficiently over NVLink (intra-node) and the RDMA fabric (inter-node).
All-reduce bandwidth is often the bottleneck: if the fabric is oversubscribed or lossy, collectives stall and GPU utilization craters. This is why training clusters use rail-optimized, non-blocking fabrics.
When a run hangs at startup or mid-training, it's usually a collective problem — a wrong interface, MTU mismatch, PFC storm, or a dead rank. See the NCCL/RCCL troubleshooting workflow.