Public cloudLLM inferenceVisionMultimodal

Batch Inference Platform

An offline platform optimized for throughput and cost per item, not latency. Work is queued and processed in large batches, ideal for spot/preemptible capacity.

Customize in Designer
Business objective

Score, embed, or transform large datasets at the lowest cost per item, on a schedule or on demand.

Personas
Data engineerML engineerPlatform engineer
Key assumptions
  • ·No latency SLO; completion-time and cost matter.
  • ·Work is idempotent and resumable — safe to run on preemptible capacity.
Logical architecture

Rendering diagram…

Design

Compute

Maximize batch size and utilization; cheaper inference GPUs often win on cost per item. Preemptible/spot friendly with checkpointing.

Network

North-south to object storage; keep workers close to data to avoid egress cost and latency.

Storage

Object storage in and out; shard work to parallelize; cache models locally.

Kubernetes

Batch orchestration (Argo Workflows, Ray, or Kubernetes Jobs) with a work queue; scale-to-zero when idle.

Security

Scoped access to input/output buckets, encrypted data, and least-privilege workers.

Observability

Throughput (items/sec), cost per item, queue depth, and preemption/retry rates.

High availability
Not required in the serving sense — resilience is retry + resume. A preempted worker's shard is re-queued.
Disaster recovery
Inputs are durable in object storage; reruns are deterministic. Track processed offsets to resume, not restart.
Capacity
Sized by dataset size ÷ target completion time. Scale workers horizontally; more workers = shorter wall-clock at similar cost.
Cost profile
Lowest-cost profile: spot/preemptible + high batch utilization + scale-to-zero. Cost per item is the headline metric.
Risks
  • mediumNon-idempotent work corrupts results on retry after preemption.
  • lowSmall batches waste GPU throughput and raise cost per item.
Implementation phases
  1. 1

    Queue + workers

    Work queue, GPU batch workers, sharded input/output.

  2. 2

    Spot-hardening

    Checkpoint/resume and idempotent writes for preemptible capacity.

  3. 3

    Scale-to-zero

    Autoscale workers to zero when the queue drains.

Validation checklist
  • Reprocessing after preemption produces identical results.
  • Cost per item within target at production batch size.
  • Workers scale to zero when idle.
Recommended GPUs for this pattern
NVIDIA

L4

24 GB · 72 W

NVIDIA

L40S

48 GB · 350 W

NVIDIA

A100 40GB PCIe

40 GB · 250 W

AMD

Instinct MI210

64 GB · 300 W