Getting an agent to work in a demo is easy. Keeping it correct, debuggable, safe, and affordable in production is the actual job — and it's four distinct disciplines. Skip them and you ship something that dazzles in a screenshot and quietly fails in the field (see: Klarna's long-tail walk-back).

1 · Evaluation — you can't ship what you can't measure

The critical insight: output-only evaluation is insufficient — a correct final answer can hide broken reasoning or a hallucinated tool call. So agents are evaluated at three granularities:

Hallucination isn't one metric — it's four, and they can disagree on the same answer:

2 · Observability — tracing the tree

A single agent request is a tree of prompts, model calls, tools, retries, and sub-agents — plain logs can't expose it. The standard is now OpenTelemetry GenAI conventions: a top-level invoke_agent span with child chat and execute_tool spans. Store prompts as span events (not attributes — they're indexed and leak PII), and propagate trace context across every hop.

3 · Security — the lethal trifecta

Prompt injection is OWASP's #1 LLM risk — the model processes instructions and data in the same channel, so anyone who controls data the model reads can inject commands. The most useful mental model (from Ch9): data exfiltration needs three things at once, and breaking any one stops it.

4 · Cost optimization — stack the levers

Unoptimized agent workflows waste 40–60% of tokens. The big levers stack — toggle them on a nominal $1.00 bill: