How a large language model actually works
This is the most important chapter in the book, because everything an AI agent does — writing code, answering a customer, browsing the web — reduces to one operation you're about to fully understand: a model predicting the next token. Get this, and nothing later is magic.
We'll build a language model up from nothing, one idea at a time, and you'll play with three of the core mechanisms rather than just read about them. Let's start with an analogy and then earn the real thing.
A large language model is an extraordinarily sophisticated autocomplete: given some text, it predicts what token most plausibly comes next, adds it, and repeats. That's the entire engine. Its intelligence is an emergent side-effect of doing that one thing extremely well, at enormous scale.
1 · Text becomes tokens
A model never sees letters or words the way you do. The very first thing that happens to your text is tokenization: it's chopped into tokens — chunks that are usually sub-words — using an algorithm called Byte-Pair Encoding (BPE). BPE starts from single characters and repeatedly merges the most frequent adjacent pairs until it has a fixed vocabulary of, say, ~100,000 tokens. Common words ("the", "is") become a single token; rarer words get split into familiar pieces ("tokenization" → "token" + "ization").
A useful rule of thumb: 1 token ≈ 4 characters ≈ ¾ of a word in English. Each token maps to an integer ID, and it's those integers the model actually processes.
Everything downstream is measured and billed in tokens, not words. Context limits ("200K tokens"), latency, and cost are all token-denominated. When you optimize an agent's cost later in the book, you're optimizing tokens.
2 · Tokens become vectors, and meaning becomes geometry
A token ID like 4021 means nothing on its own — it's just a row number. So the model looks it up in an embedding table and turns it into a vector: a long list of numbers, often a few thousand of them (e.g. 4,096). Every token gets its own vector, and those numbers are learned during training, not assigned by hand.
Here's the idea that makes it beautiful. Think of each vector as a coordinate in a high-dimensional space. Training nudges the coordinates so that tokens used in similar contexts end up near each other — because that's what lets the model predict the next token well. Linguists call this the distributional hypothesis: "you shall know a word by the company it keeps." The consequence: meaning becomes geometry. "king" and "queen" land close together; "Paris," "London," and "Tokyo" cluster; "cat," "dog," and "lion" form their own neighborhood.
We can't draw 4,096 dimensions, so the map below is a 2-D projection — but the intuition is exact. Two things to feel here: (1) distance = similarity (nearby vectors mean similar things — this is literally what vector search and RAG will use later), and (2) directions carry meaning — the step from "man" to "king" is the same direction as "woman" to "queen," which is why the famous analogy king − man + woman ≈ queen actually works.
This one idea powers a whole later chapter. Retrieval-Augmented Generation (RAG) works by embedding your documents into this same kind of space and, at query time, fetching the chunks whose vectors sit nearest your question's vector. "Find similar meaning" becomes "find nearby points" — a geometry problem your database can solve in milliseconds.
3 · Attention: how a token looks at the other tokens
Now the engine. A transformer processes all tokens together through a stack of layers, and the heart of each layer is self-attention. Intuitively: to understand a word, you need the words around it. In "the animal didn't cross the street because it was too tired," what does "it" refer to? You resolved that instantly using context. Attention is the mechanism that lets the model do the same.
Mechanically, each token produces three vectors via learned weight matrices:
- a Query — "what am I looking for?"
- a Key — "what do I offer?"
- a Value — "what do I contribute if you attend to me?"
A token's attention to another is the dot product of its Query with that token's Key — a similarity score. Those scores are pushed through softmax (so they're positive and sum to 1) to become weights, and the token's new representation is the weighted blend of all the Value vectors. Run several of these in parallel — the original Transformer used 8 heads, each free to focus on a different kind of relationship — and combine them. Because attention has no built-in notion of order, a positional encoding is added so the model knows which token came first.
Every token attends to every other token, so n tokens create n² relationships. That quadratic blow-up is the root reason context windows are finite and expensive, why long prompts get slower and pricier, and why "context engineering" (a later chapter) is a real discipline. Hold onto this — it explains a lot of the infrastructure later in the book.
4 · Next-token prediction: the whole engine in one loop
After the final layer, the model produces a logit — a raw score — for every token in its vocabulary. Softmax turns those scores into a probability distribution over "the next token." The model picks one, appends it to the input, and runs the whole thing again to get the token after that. This is called being autoregressive, and that loop is text generation.
So how does it "pick"? That choice is yours to tune, and it's where personality comes from:
- Greedy — always take the single most probable token. Safe, but repetitive.
- Temperature — a dial that sharpens or flattens the distribution. Low temperature makes the top token even more dominant (focused, deterministic); high temperature evens out the odds (creative, but risks nonsense).
- Top-p (nucleus) — sample only from the smallest set of tokens whose probabilities add up to p; it adapts to how confident the model is.
5 · Why models hallucinate
Now a consequence that trips everyone up. The model's objective is plausible next tokens — not true ones. It has no built-in database and no step where it checks reality. When it doesn't "know" something, it doesn't stop; it produces the most likely-sounding continuation, confidently. That's a hallucination — and it isn't a bug you can patch away, it's a direct property of the mechanism you just built.
This single fact motivates a huge fraction of everything later in the book. Retrieval (RAG), tool use, and grounding all do the same job: put real facts into the model's context so that "most plausible" and "actually correct" line up.
6 · The context window is the model's entire memory
Here's the idea to carry into every later chapter: the model is a stateless function. Tokens in → next-token distribution out. It remembers nothing between calls. Its only working memory is the context window — the tokens you pass in this call (system prompt + conversation history + tools + retrieved documents). Two consequences shape all of agent engineering:
Recap — what you now know cold
- Text → tokens (BPE, ~4 chars each) → vectors where meaning is geometry.
- Attention lets each token blend in context via Query·Key→softmax→Value; it costs n².
- The model emits a probability distribution over the next token; sampling (temperature/top-p) shapes the output; it generates autoregressively.
- Hallucination is intrinsic — plausibility, not truth — which is why grounding exists.
- The model is stateless; the context window is its only memory, it's finite, and position matters.
An LLM is a stateless, next-token predictor with a finite attention budget. Everything else — agents, tools, memory, RAG, caching — is machinery built around that stateless core. Next, we wrap it in a loop and give it tools, and it becomes an agent.