highServing

Inference latency spikes

Serving p99 latency exceeds the SLO.

Scope
Serving path: autoscaling, batching, cold starts, or GPU saturation.
Check recent changes
  • ·Traffic growth
  • ·New model version
  • ·Autoscaler/batching config change
Diagnostics

Latency vs. load

text
# Correlate p99 latency with QPS and replica count.
# Spikes on scale-up ⇒ cold-start weight loads.
Likely causes & fixes
  • high likelihoodAutoscaling too slow for bursts

    Fix · Raise min replicas / predictive scaling; keep headroom.

  • high likelihoodCold-start weight pulls

    Fix · Warm caches; keep weights regional/local.

  • medium likelihoodBatching misconfigured

    Fix · Tune max batch/latency window (vLLM/Triton).

  • medium likelihoodGPU saturated

    Fix · Add capacity or quantize (FP8/INT8).

Validation
  • ✓p99 back within SLO at peak QPS and during scale-up
Escalation

Cluster-wide capacity exhaustion → emergency capacity request.

Data to collect first
  • ·Latency p50/p95/p99
  • ·Replica count vs. queue depth
  • ·GPU utilization on servers
  • ·Cold-start durations
Related alerts
inference-latency-breach