Distributed foundation-model training
Synchronous data- and model-parallel training across many GPUs, bounded by the collective-communication fabric.
Architecture
Gang-scheduled worker pods run one process per GPU. Data/tensor/pipeline parallelism is coordinated by the framework; all-reduce runs over a rail-optimized RDMA fabric. Checkpoints stream to a parallel file system.
Sizing
GPU count from model memory (weights + optimizer + activations) and target tokens/sec; keep jobs on dense 8-GPU nodes for on-node NVLink.
Networking
Rail-optimized InfiniBand or lossless RoCEv2, non-blocking (1:1); GPUDirect RDMA end-to-end.
Storage
Parallel FS for burst checkpoint writes and dataset reads; object storage cold tier.
Scaling
Gang scheduling (Kueue/Volcano) — all ranks start together or none do. Elastic/resumable training reschedules a failed rank from checkpoint.
Monitoring
Per-rank throughput, NCCL/RCCL timings, fabric PFC/ECN counters, GPU utilization, checkpoint duration vs. interval.
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
name: llm-train
spec:
pytorchReplicaSpecs:
Worker:
replicas: 8
template:
spec:
containers:
- name: pytorch
image: your-registry/train:latest
resources:
limits:
nvidia.com/gpu: 8
rdma/rdma_shared_device_a: 1
tolerations:
- key: nvidia.com/gpu
operator: Exists