Serving one replica is solved. The hard part is running many under bursty, long-running, failure-prone agent traffic — where the usual Kubernetes reflexes are not just suboptimal, they're actively wrong. Here's what breaks and how production teams fix it.

1 · Why standard autoscaling is blind

Kubernetes' default HPA scales on CPU. For LLM serving that fails: the GPU does the compute (CPU stays near-idle) and vLLM pre-allocates the KV cache at startup (GPU memory looks constant). A pod handling 50 concurrent requests can sit at 5–8% CPU while its GPU is at 95% and latency hits 30s — HPA sees "healthy" and never scales.

2 · Scale on queue depth (KEDA)

The fix: scale on application metrics with KEDA, not CPU with HPA. The best signal is queue depth (vllm:num_requests_waiting). Push traffic up and watch the two approaches diverge — CPU-based stays stuck at one replica while requests pile up:

Recommended queue thresholds scale with model size: 10–20 for <10B, 5–10 for 10–70B, 2–5 for >70B. KEDA also enables scale-to-zero (cost-effective only below ~5–10 sustained RPS).

3 · The cold-start problem

A vLLM pod can take 2–5 minutes to become ready — and the dominant cost is usually pulling the container image (4–6 min uncached), not loading weights. So pre-pull images, put weights on a fast PVC, and keep scale-down cooldowns ≥300s to avoid thrash.

4 · Durable execution — surviving crashes

Agents run minutes to hours and make expensive, side-effecting calls. A plain loop loses everything on a crash. Durable execution (Temporal, Restate) journals every completed step, so after a crash it replays and skips completed steps — exactly-once semantics. Trigger a crash and watch it resume:

5 · Reliability patterns & compounding error

Multi-step reliability multiplies: independent steps at r each give rⁿ overall. That's why you verify per step and add retries (backoff + jitter), fallbacks, and circuit breakers. See how fast it decays: