Language models have three hard limits from Chapter 1: a fixed knowledge cutoff, no access to your private data, and a finite window. Retrieval-Augmented Generation (RAG) fixes all three by fetching relevant text at query time and placing it in the context — so the model's "most plausible" answer is also grounded in your facts. It's the most-deployed pattern in production AI, and most teams get the details wrong. Let's build the whole machine.

1 · The core idea: retrieve, then generate

Instead of hoping the model memorized your docs, you: (1) split your documents into chunks, (2) turn each into an embedding vector (Chapter 1 — meaning as geometry), (3) store them in a vector database. At query time you embed the question, find the nearest chunks (similar meaning = nearby vector), and paste them into the prompt. Watch it happen:

2 · Embeddings & the vector database

An embedding model maps text to a vector so similar meanings land near each other; relevance is measured by cosine similarity (the angle between vectors, 1 = identical, 0 = unrelated). Comparing your query to every stored vector is exact but O(N) — too slow at billions. So vector DBs use Approximate Nearest Neighbor (ANN) indexes that trade a sliver of recall for huge speed:

• HNSW — a layered "small-world" graph; search descends from a sparse top layer to the answer. ~95%+ recall, highest QPS, more RAM. Tuned by M, efConstruction, efSearch.
• IVF — cluster the space with k-means, probe only the nearest clusters. Memory-efficient at scale.
• Quantization (scalar/product) shrinks vectors 4–32× so billions fit in RAM/disk (DiskANN).

Rule of thumb: HNSW below ~85% filter ratio, IVF at 85–95%, brute-force only when you need >99% recall. Every index exposes a knob trading accuracy for latency; tune to 95–99% recall within your budget.

3 · Chunking — the underrated lever

Before embedding, documents are split. This matters as much as the embedding model (per a Vectara study). Options: fixed-size (simple, cuts sentences), recursive character splitting (paragraphs→lines→sentences — the pragmatic default), and semantic (cut where meaning shifts — expensive, often underwhelming).

4 · Advanced retrieval — where the real gains live

• Hybrid search: dense vectors miss exact terms (error codes, SKUs); classic BM25 keyword search catches them. Fuse both with Reciprocal Rank Fusion. One measured jump: MRR 0.41 → 0.67.
• Rerankers (cross-encoders): retrieve broad (N≈50–100), then a cross-encoder that reads query+doc together reorders to a top K≈5. Adds ~12–18 NDCG@5 points.
• Contextual retrieval: prepend a chunk-specific summary before embedding — Anthropic measured retrieval failure dropping 35% → 49% → 67% stacking contextual embeddings + BM25 + reranking.
• GraphRAG (knowledge-graph retrieval) and agentic RAG (the agent decides what/when to retrieve, iteratively — 49.6% vs 8.4% recall@1 on BRIGHT).

Recap

• RAG = embed docs → store vectors → at query time fetch nearest chunks → inject → generate.
• ANN indexes (HNSW/IVF) + quantization make it fast at scale.
• Chunking matters as much as the model; 200–500 tokens, 10–20% overlap.
• Hybrid + reranking + contextual retrieval are where accuracy actually improves.