Inference
Batch / offline inference
Throughput-first offline scoring or embedding generation, ideal for spot capacity.
RayArgo WorkflowsvLLM (offline)TensorRT
Architecture
A work queue feeds GPU batch workers that read from and write to object storage. Idempotent, resumable work runs safely on preemptible nodes.
Sizing
Dataset size ÷ target completion time; scale workers horizontally at high batch utilization.
Networking
North-south to object storage; keep workers close to data.
Storage
Object in, object out; cache the model locally; shard work for parallelism.
Scaling
Scale workers to zero when the queue drains; checkpoint/resume for preemption.
Monitoring
Items/sec, cost per item, queue depth, preemption/retry rate.
starting point
apiVersion: batch/v1
kind: Job
metadata:
name: batch-embed
spec:
parallelism: 8
completions: 8
template:
spec:
restartPolicy: OnFailure
containers:
- name: worker
image: your-registry/embed:latest
resources: { limits: { nvidia.com/gpu: 1 } }