Compute Architecture
Design GPU nodes and choose how to share them. Size CPU, memory, storage, networking, and power per node, pick a partitioning strategy, and match a scheduler to your workloads.
72
~9/GPU
1,664 GB
host memory
12 TB
scratch/cache
3200 Gb
~8× 400 Gb rail NICs (GPUDirect RDMA)
6.4 kW
PSU ~8.5 kW
256
GPU + NIC
- ·Dense 8-GPU node — NVLink/NVSwitch keeps all-reduce on-node.
- ·~6 kW/node at this density — plan liquid cooling and rack power.
- ·Needs ~256 PCIe lanes — verify the platform's CPU/switch lane budget.
- ·Align each GPU's pod to the nearest NUMA node and CPU socket.
1 workload per GPU · NVLink/IF-linked for multi-GPU jobs
Large training or high-throughput inference wants the full GPU and interconnect bandwidth.
Caveat · Underutilized if the workload can't fill the GPU — revisit if utilization stays low.
Sizing figures are planning heuristics — validate against the platform and workload.
| Strategy | Isolation | Granularity | Best for |
|---|---|---|---|
MIG (Multi-Instance GPU) NVIDIA (A100 / H100 / B-class) | strong | Up to 7 hardware instances, each with dedicated memory + compute | Multi-tenant sharing and mixed small/interactive workloads needing isolation Caveat: Fixed profiles; reconfiguration drains the GPU. Only MIG-capable parts. |
Time-slicing NVIDIA / AMD (any GPU) | weak | Oversubscribe one GPU to N pods via context switching | Dev/test and bursty, low-priority inference where isolation isn't required Caveat: No memory isolation or fairness — one pod can starve or OOM others. |
MPS (Multi-Process Service) NVIDIA | moderate | Concurrent spatial sharing of SMs across cooperative processes | Co-locating cooperative processes to raise utilization at low latency Caveat: Weak fault isolation; a crash can affect co-located clients. |
SR-IOV / vGPU AMD SR-IOV · NVIDIA vGPU | strong | Virtual functions / vGPU profiles assigned to VMs | VM-level multi-tenancy and virtualized platforms (vSphere, KubeVirt) Caveat: vGPU needs licensing; VF count and profiles are hardware-bound. |
| Scheduler | Gang | Fair-share | Quotas | Preempt | Best for |
|---|---|---|---|---|---|
| Default kube-scheduler | Simple inference and single-GPU jobs | ||||
| Volcano | Batch and distributed training (all-or-nothing gangs) | ||||
| Kueue | Job queueing + quotas on otherwise-vanilla Kubernetes | ||||
| Apache YuniKorn | Hierarchical multi-tenant queues with fair-share | ||||
| Slurm (HPC) | HPC/MPI batch jobs; bridge to Kubernetes via Slinky / the Slurm operator |
CPU-to-GPU ratio
Budget ~6–8 CPU cores per GPU (more for data-heavy training) so data loading and collectives never starve the GPUs.
NUMA alignment
Pin each GPU's workload to the CPU socket and memory closest to it; misaligned NUMA silently halves effective bandwidth.
PCIe topology
Give each GPU a full x16 link and keep GPU↔NIC pairs on the same PCIe switch for GPUDirect RDMA.
NVLink / NVSwitch
Intra-node NVLink/NVSwitch (or Infinity Fabric) carries collective traffic; dense 8-GPU nodes keep all-reduce on-node.
Huge pages & IOMMU
Enable huge pages for large models and RDMA; configure IOMMU (intel_iommu/amd_iommu=on) for passthrough and SR-IOV.
Power & cooling
700–1000W-class parts need rack-level power planning and often direct-liquid cooling; size PSUs with headroom.