Networking Architecture
Size the fabric that carries collective-communication traffic. Plan per-GPU bandwidth, bisection, and leaf-spine switches, choose between InfiniBand and RoCEv2, and get the lossless-config essentials.
400 Gb
line rate
3200 Gb
8× rails
51.2 Tb
full non-blocking
8
rail-optimized
8
~64-port
4
non-blocking
InfiniBand (rail-optimized, non-blocking)
Oversubscription: 1:1 (non-blocking) — collectives must not queue
- ·Run a subnet manager; verify link speed and topology.
- ·Enable GPUDirect RDMA; consider SHARP in-network reduction.
- ·Attach the RDMA NIC to pods via Multus + SR-IOV CNI.
Simplified 64-port-radix model — validate switch counts against real SKUs and cabling.
| Fabric | RDMA | Latency | Lossless | Best for |
|---|---|---|---|---|
| InfiniBand | Lowest (~1 µs) | Native credit-based flow control; subnet manager | Large distributed training where collective performance dominates Tradeoff: Separate fabric + subnet manager; higher cost and specialized ops. | |
| RoCEv2 (RDMA over Ethernet) | Low (~2–3 µs) | Requires PFC + ECN tuned end-to-end (lossless Ethernet) | Training on an Ethernet fabric, reusing Ethernet tooling and staff Tradeoff: Lossless config is finicky; PFC storms and ECN mistuning cause stalls. | |
| Standard Ethernet (TCP) | Higher (TCP stack) | None required | Inference, storage, and north-south traffic Tradeoff: No RDMA — collective-heavy training bottlenecks here. |
Cilium (eBPF)
Default pod networking, network policy, and observability
Calico
Mature policy enforcement and BGP routing
Multus
Attaching a second (RDMA) NIC to training pods
SR-IOV CNI
Exposing NIC virtual functions for GPUDirect RDMA
OVN-Kubernetes
OpenShift and overlay-heavy environments
Rail-optimized topology
Give each GPU its own NIC and rail so all-reduce traffic spreads across independent paths instead of colliding on one uplink.
GPUDirect RDMA
Let NICs DMA directly to/from GPU memory, bypassing the CPU. Keep GPU↔NIC pairs on the same PCIe switch.
Separate networks
Isolate compute (collectives), storage, management, and out-of-band onto distinct fabrics/VLANs so one can't starve another.
Lossless config (RoCE)
Tune PFC and ECN end-to-end and match MTU (jumbo frames). A single misconfigured hop causes pauses and NCCL timeouts.
NCCL / RCCL tuning
Set the right interface (NCCL_SOCKET_IFNAME), topology hints, and algorithm; validate with all-reduce micro-benchmarks.
Non-blocking bisection
For synchronous training, size the fabric 1:1 (non-blocking) so collectives never queue behind oversubscription.