Storage Architecture
Size storage for models, checkpoints, and datasets — and the throughput they demand during bursts. Pick the right tier per workload and follow the principles that keep GPUs fed.
140 GB
FP16 weights
420 GB
per checkpoint
10.1 TB
24 retained
26 TB
+ overhead
36.2 TB
capacity
0.5 GB/s
checkpoint burst
64 GB/s
aggregate data-loading
Parallel FS (checkpoints + datasets) + object cold tier
Burst checkpoint writes and high-throughput dataset reads need a parallel file system; keep cold data in object storage.
- ~64 GB/s aggregate read — enable GPUDirect Storage and cache datasets near the GPUs.
- Size for throughput (GB/s) and IOPS during bursts, not only total capacity.
Planning heuristics — validate against the framework, checkpoint format, and data pipeline.
| Type | Throughput | Scale | Best for |
|---|---|---|---|
| Local NVMe | Highest per-node (GB/s) | Node-local only | Scratch, dataset cache, checkpoint staging Caveat: Not shared — data is lost with the node. |
| Parallel FS (Lustre / GPFS / BeeGFS) | Very high aggregate (TB/s at scale) | Cluster-wide, scales with servers | Training datasets and burst checkpoint writes Caveat: Complex to operate; needs dedicated storage servers + network. |
| Object (S3 / MinIO) | High aggregate, higher latency | Effectively unlimited | Datasets, model weights, artifacts, cold tier Caveat: Not a POSIX filesystem; stream or cache for training. |
| NFS | Moderate | Shared, limited by the server | Home directories, configs, small shared assets Caveat: Rarely sustains large checkpoint bursts. |
| CephFS / Rook | Moderate–high | Software-defined, scales out | General shared storage on commodity hardware Caveat: Tuning matters; not always parallel-FS fast. |
Distributed training
Parallel FS sustains burst checkpoint writes and high-throughput dataset reads; keep cold datasets in object storage.
Fine-tuning
Cache the base model and adapters on local NVMe; pull from object storage.
Real-time inference
Model weights live in object storage / a registry; cache them on local NVMe to cut cold-start.
RAG
Vectors in the vector store, source documents + embeddings in object storage, hot contexts cached.
Batch inference
Stream inputs from and write results to object storage; cache the model locally.
GPUDirect Storage
Let storage DMA directly into GPU memory, bypassing the CPU bounce buffer — cuts data-loading overhead for training.
Tiering
Hot (local NVMe) → warm (parallel FS) → cold (object). Move data down as it ages to control cost.
Checkpoint strategy
Async and sharded checkpointing overlaps I/O with compute so a checkpoint never stalls the GPUs; size storage for the burst.
Data locality & caching
Cache datasets near the GPUs and reuse across epochs; a cold read path silently starves utilization.
Throughput, not just capacity
Size for the required GB/s and IOPS during checkpoints and data loading, not only for total terabytes.