I
The main event · infrastructure
What it actually takes to run an agent in production
An agent is not a stateless API call. It holds state across a multi-step session, calls external tools, runs sandboxed code, and can execute for minutes to hours. Verified That's why agent infrastructure needs externalized session state, a tool/MCP layer, and long-lived compute — things stateless LLM serving never required. Here's the full topology, tier by tier.
production reference architecture · request flows top → bottom · observability spans all tiers
0
Client / channel ingress
Web/mobile app, embedded chat widget, Slack/Teams bot — or another agent over A2A. Long-lived SSE/WebSocket connections, because responses stream token-by-token.
Web · MobileChatKit widgetSlack / TeamsA2A peer
1
Edge / API gateway auth · rate limit
TLS termination, authN/authZ (OAuth/OIDC, API keys), WAF, coarse rate limiting and routing. A conventional API gateway.
Kong · APISIXCloudflareAWS API GWAzure APIM
2
AI / model gateway the control plane
A specialized proxy between agent and models: unified OpenAI-compatible API, model routing, provider fallback, token-based rate limits & budgets, cost tracking, semantic caching, guardrails. Increasingly governs MCP + A2A traffic too.
LiteLLMPortkeyKong AI GWCloudflare AI GWAzure APIM GenAI
3
Agent runtime / orchestrator the loop
Where reason → call tool → observe → repeat executes. Stateless — holds no durable state itself; attaches/detaches session state each turn. Serverless-managed or self-hosted.
AgentCore RuntimeVertex Agent EngineK8s podsRay ServeTemporal workers
4
Tools / MCP servers capabilities
APIs, Lambda functions, databases and SaaS exposed as MCP tools — often behind an MCP gateway that adds OAuth, tool discovery and per-agent access control.
MCP serversMCP gatewayLambda / APIsOAuth scopes
5
Sandboxed execution zero-trust
Per-session isolated environment for running LLM-generated code and browser automation. Treated as untrusted — hardware-isolated, one sandbox per session.
Firecracker microVMgVisorCode interpreterCloud browser
6
State / memory / vector externalized
Hybrid tier: a Redis-like KV store for live session/thread state (with TTL) + a vector DB for long-term semantic retrieval. Externalized so the runtime stays stateless.
Redis / KVMemory bankPinecone · WeaviateQdrant · pgvector
7
Model providers inference
Hosted APIs (Bedrock, OpenAI, Gemini, Anthropic) and/or self-hosted inference servers on GPUs.
Hosted APIsvLLM · SGLangTriton / NIMGPU node pool
✳
Observability & governance cross-cutting · all tiers
OpenTelemetry tracing of every step, tool call and token; evals; cost/usage metering; agent identity and policy enforcement — wrapping every hop above.
OTel tracingCost meteringAgent identityPolicy (Cedar)
Verified
Reality check: production deployment is still largely DIY.
Managed platforms exist, but as of 2026 most production teams still ship agents on FastAPI + self-managed infrastructure. The managed offerings below are catching up fast — but "buy" hasn't beaten "build" yet.