Inference internals — how models are served
To scale agents you eventually have to understand what happens on the GPU. It's not trivia — it explains why output tokens cost more than input, why batching cuts your bill ~85%, and why throwing a bigger GPU at a latency problem often does nothing. Almost every serving optimization exists to exploit one distinction.
1 · Prefill vs decode — the central split
Inference has two phases with opposite bottlenecks. Prefill processes your whole prompt in one parallel pass — lots of math, GPU cores busy: compute-bound. Decode generates one token at a time, and each step must read the entire model weights + KV cache from memory to produce a single token: memory-bandwidth-bound.
For a 70B model at FP16, decode moves ~140 GB per token. On an H100 (3.35 TB/s) that's a ~42 ms-per-token floor set by memory bandwidth — a second GPU doesn't lower it, because the bottleneck is the memory bus, not compute. It's also why output tokens cost 3–5× input tokens.
2 · Continuous batching — the biggest lever
Because decode leaves compute idle, the fix is to serve many requests at once: instead of fixed batches, the server dynamically adds/removes requests every step to keep the GPU saturated. The effect on cost-per-token is enormous — drag concurrency and watch:
3 · The optimizations that stack
Serving frameworks layer several tricks, and they compound multiplicatively. Toggle them and watch the effective cost reduction multiply:
4 · KV-cache tiering — the memory hierarchy
Weights and the KV cache both live in scarce GPU memory (HBM) and compete for it — an H100 has just 80 GB, and a single 128K-context user can need ~40 GB of KV cache. So inactive KV blocks are offloaded down a hierarchy, trading bandwidth for capacity:
Utilization is everything: the same GPU at 10% load turns $0.013/1k tokens into $0.13. Self-hosting a 7B model only beats API pricing above ~50% GPU utilization. Next chapter: how to scale and keep those GPUs busy without the usual Kubernetes reflexes (which fail here).