Checkpointing without stalling training

A training checkpoint includes weights, gradients, and optimizer state — often ~6× the model's raw size in mixed precision. Writing it must complete inside the checkpoint interval, or the GPUs sit idle waiting on I/O.

Use async and sharded checkpointing so the write overlaps ongoing compute, and put checkpoints on a parallel file system that sustains the burst throughput (NFS rarely does). The Storage planner sizes both capacity and the required GB/s.

Keep a rolling window plus periodic long-term checkpoints, and replicate final weights to a second site for disaster recovery.