Caching at every layer
Caching is the highest-leverage cost-and-latency win in the whole stack — but "caching" means four different things at four different layers, and teams constantly confuse them. This chapter walks each one, deepest-in-the-GPU to highest-in-your-app, and lets you watch the bill drop.
1 · KV cache — inside a single generation
When a model decodes text, attention would normally recompute the Key and Value vectors for every prior token at each step. The KV cache stores them after first computation, so each new step only computes the new token. It's what makes generation fast at all — without it, producing token 1,000 would redo the work for tokens 1–999.
2 · Prefix cache — reuse across requests
Prefix caching extends the KV cache across requests: the KV state of shared leading tokens — a system prompt, a big reference doc, few-shot examples — is stored and reused by the next request that starts the same way. The one hard rule: the cached prefix must be byte-for-byte identical — one changed character invalidates it. (vLLM does this automatically; SGLang's RadixAttention shares overlapping prefixes across many requests via a radix tree.)
3 · Prompt / context caching — the API feature you pay for
This is prefix caching exposed as a billing feature. You mark a stable prefix as cacheable, and repeat calls skip reprocessing it. The economics are dramatic — Anthropic charges cache reads at 0.1× the base input price (~90% off); OpenAI gives ~50% off. The design implication (from Chapter 4): put static content first so the cacheable prefix is as long as possible. Watch the bill for a big fixed knowledge base:
4 · Semantic cache — skip the model entirely
The highest layer, in your app. Embed the incoming query, search a small vector store of past queries, and if a previous one is close enough (cosine threshold), return its cached answer without calling the model at all. It adds an embedding + lookup to every request but can cut spend ~70% on repetitive workloads — the caveat being it returns an approximate match, so it's ideal for FAQ traffic and dangerous where small wording changes the correct answer.
The four layers at a glance
| Layer | Where | What it saves |
|---|---|---|
| KV cache | GPU, within one call | recompute of past tokens during decode |
| Prefix cache | server, across calls | reprocessing a shared prompt prefix |
| Prompt cache | API billing | 50–90% off the cached prefix tokens |
| Semantic cache | your app | the entire model call on near-duplicate queries |
The KV cache we started with lives in scarce GPU memory. Next we open up inference serving — prefill vs decode, batching, and why a faster GPU won't fix your latency.