The self-hosted stack on Kubernetes
Reported The typical pattern: stateless agent pods (or Temporal workers) → externalized state → model serving on a GPU node pool → per-session sandboxes → event-driven autoscaling on queue depth.
pip install ray[serve,llm] — engine-agnostic (wraps vLLM, SGLang), OpenAI-compatible, with built-in autoscaling, load balancing, fault tolerance and observability.
Framework checkpointing (LangGraph/CrewAI/ADK) saves state, but leaves failure detection, automatic recovery and distributed coordination to you — which is why durable-execution engines get layered on top. Long-running workflows force "rainbow deployments" (multiple worker versions live at once).
HPA scales on CPU/GPU; KEDA scales event-driven on queue backlog — reacting faster than CPU metrics — and can scale workers to zero when idle. Reference pattern: Temporal + Composio + KEDA + K8s.
Why sandboxing is non-negotiable
Reported LLM-generated code and tool calls are treated as untrusted / zero-trust, so each session runs hardware-isolated. The isolation ladder — stronger isolation costs more overhead:
gold standard
compute-heavy fit
middle ground
avoid
AI gateway & model-serving runtimes
Reported The AI gateway is the control plane for model traffic; the serving runtime is what turns GPUs into an inference endpoint. Azure APIM specifics are primary-sourced.
| AI gateway | Model | What it gives you |
|---|---|---|
| LiteLLM | OSS · self-host | Most-adopted OSS gateway. OpenAI-compatible across providers, virtual keys, budgets, semantic caching. Docker/Helm/Terraform, no per-call fee. |
| Portkey | Managed | Routing across 200+ providers, fallback chains, edge + semantic caching, guardrails, observability. |
| Kong AI Gateway | Enterprise | Universal LLM API, prompt-injection scanning, vector-embedding semantic cache, per-consumer cost budgets, per-token rate limiting. |
| Cloudflare AI Gateway | Edge | Global edge network for geographic caching + routing + observability. |
| Azure APIM GenAI | Primary docs | llm-token-limit quotas (rejects over-limit prompts before backend), semantic cache via Managed Redis, backend pools with circuit breakers for cross-region failover. |
| Serving runtime | Edge | When to reach for it |
|---|---|---|
| vLLM | PagedAttention | Broadest support; default choice for most agent backends (~70% of field deployments, third-party estimate). |
| SGLang | RadixAttention | Best for agent workloads — many tool calls, multi-turn, structured outputs, prefix-heavy RAG. Up to ~6.4× throughput on structured work. |
| Triton + TensorRT-LLM | Multi-model | Heterogeneous fleets — one API for LLM + embedding + vision on shared GPUs. NIM packages these as prebuilt microservices. |
| TGI (Hugging Face) | Maintenance | Moved to maintenance mode Dec 2025; new deployments steered to vLLM/SGLang. |
Utilization dominates the bill, not the model
Reported The single biggest infra-cost lever is GPU utilization / batching. Same H100 serving Mixtral 8×7B, only the request rate changed:
via batching
- Bursty demand → ~12% average GPU utilization if you provision for peak. Autoscaling recovers a large share (>80% GPU savings vs. static peak) — but can't fully absorb spikes, because demand outruns cold starts.
- Cold starts are a first-order constraint: a 128k-context cold start can push time-to-first-token to ~160 s.
- KV-cache tiering is the emerging memory pattern — keep HBM for active generation, offload inactive KV cache to NVMe/CPU RAM (HBM is too scarce to hold long-lived state).
- Scale-to-zero via KEDA is the main lever to cut idle GPU/worker cost for bursty agent workloads.
Three autoscaling headaches specific to agents
SSE/WebSocket streams make request-count scaling and graceful drain hard.
Large-batch decode is memory-bandwidth-bound (HBM saturation); GPUs are scarce and expensive.
Spikes arrive faster than GPU cold starts — over-provision, or eat the latency.