advancedTraining ~45 min

Run a distributed training job

Launch a multi-GPU training job with gang scheduling.

Scenario

You need all ranks to start together so the collective doesn't deadlock.

Prerequisites
  • ·Multiple GPU nodes
  • ·A batch scheduler (Kueue/Volcano)
  • ·RDMA fabric for multi-node
Tasks
0 / 3 tasks
  1. bash
    kubectl apply -f pytorchjob.yaml

    Expected · Workers are created together (gang).

  2. bash
    kubectl logs -f pytorchjob-worker-0
  3. bash
    # In logs, NCCL_DEBUG=INFO should show the RDMA interface and expected bandwidth.
Success criteria
  • ✓All ranks started together
  • ✓Steps advance at expected throughput
  • ✓Collectives use RDMA
Hints
  • ·A hang at startup is usually a wrong NCCL interface or MTU mismatch.
  • ·Enable elastic training so a failed rank resumes from checkpoint.