Full curriculum

86 chapters. 12 parts. Open.

Every chapter, ordered by part — from your first model call to deployed, monitored production systems. Concise, eval-first, and free.

Chapters
86
Words
169,933
Parts
12
86/ 86
Part 0

Start Here

What an AI engineer is, the software baseline, thinking in probabilistic systems, and the escalation ladder the whole book follows: prompt, context, retrieval, tools, agent, fine-tune.

4 chapters1hBeginnerPart overview →
  1. What an AI engineer is (and isn't) AI engineer vs ML engineer vs researcher: the API line, the three roles, and exactly what this handbook teaches and skips. beg 6 min
  2. The software baseline The seven engineering skills this book assumes (Python, HTTP, JSON, async, Git, Docker, Postgres), why each one matters for AI work, and where to refresh it. beg 6 min
  3. Thinking in probabilistic systems Why the same LLM call returns different answers even at temperature 0, the five failure modes, and why graceful degradation is the default posture. beg 7 min
  4. The escalation ladder The priority order the whole book runs on: prompt, context, retrieval, tools, agent, fine-tune. Climb the cheapest rung that fixes the failure. beg 9 min
Part 1

How Models Behave

The model landscape, tokens and context windows, sampling, embeddings, reasoning models, the cost-quality-latency triangle, and just enough ML to make good system decisions.

7 chapters1hBeginnerPart overview →
  1. The model landscape Place any model into a tier, and know what each tier is for: frontier, mid, fast, and local, plus open-weight vs hosted and chat vs reasoning vs multimodal. beg 9 min
  2. Tokens and context windows Tokens are the unit every LLM bill, limit, and prompt is measured in. How to count them in Python before you send, and why non-Latin text costs more. beg 9 min
  3. Sampling and temperature How temperature and top-p reshape a model's probability distribution, how to pick them per task, and why temperature 0 is not a determinism guarantee. beg 10 min
  4. Embeddings What embeddings are, why cosine similarity and dot product agree on normalized vectors, and the four engineering patterns built on them: search, memory, routing… beg 15 min
  5. Reasoning models What test-time compute is, the effort and budget knobs across providers, why you pay for hidden thinking tokens, and when reasoning earns its cost. beg 8 min
  6. The cost-quality-latency triangle The book's recurring mental model: cost, quality, and latency are three axes you can't max at once. Name the one you sacrifice, then learn the levers that cheat… int 15 min
  7. Just enough ML The three ML ideas an AI engineer who never trains a model still needs: the data split, overfitting, and distribution shift, framed as eval decisions. beg 10 min
Part 2

Working with Model APIs

Calling models reliably: streaming, structured outputs, tool calling, errors and fallbacks, model selection, gateways and routing, and prompt caching economics.

8 chapters1hBeginner to IntermediatePart overview →
  1. Calling models What happens when you POST a chat completion: messages, roles, the system prompt, and the three provider shapes you need to know side by side. beg 10 min
  2. Streaming How token-by-token streaming wins on perceived latency, why it breaks output validation, and the four ways to get both. beg 9 min
  3. Structured outputs Why schema-enforced JSON, not free text, is the contract between an LLM and the code that consumes its output, and how to wire it up with Pydantic. beg 10 min
  4. Function and tool calling The model proposes a tool call, you execute it, the model continues: how to define tool schemas, run the loop, parallelize, and return errors the model can act … int 9 min
  5. Errors, retries, fallbacks How to handle the timeouts, rate limits, and overloaded responses every LLM API throws at you, with the right backoff, fallback, and idempotency rules. int 10 min
  6. Choosing models and reading benchmarks Public leaderboards don't predict your product's quality. Shortlist by tier, test on your own eval set, decide. That's the whole workflow. beg 8 min
  7. Gateways and routing One API in front of many providers, cheap-then-expensive cascades, provider failover, and the rule for when a gateway earns its operational cost. int 10 min
  8. Prompt caching The single biggest cost lever in API-native LLM products. A 50k-token agent over 10 turns drops from $1.53 to $0.47 with one config change. beg 10 min
Part 3

Prompt Engineering

Production prompts: anatomy, few-shot, chain-of-thought and the reasoning-model inversion, templates and chaining, and prompt versioning with rollback.

5 chapters1hIntermediatePart overview →
  1. Anatomy of a production prompt The five sections every production system prompt has, why monolithic god-prompts fail, and how to wire defensive instructions against prompt injection. beg 15 min
  2. Zero-shot and few-shot When examples earn their tokens and when they don't. The decision rule, the over-prompting cliff, and how prompt caching rewrites the math. beg 10 min
  3. Chain-of-thought, ReAct, self-consistency The three techniques that survived the 2023 prompting literature, and the reasoning-model inversion that flips the rules: when to use each, and when to delete t… int 10 min
  4. Templates and chaining Treating prompts as typed code, and the call you make every week: one big prompt or several small ones in a chain. int 10 min
  5. Prompt versioning Treat a prompt change like a code change: registry, version pinning, rollback, and review, with a minimal setup you can build in a week. int 10 min
Part 4

Eval Fundamentals

The minimum eval loop, taught before RAG and agents on purpose: look at your data, build your first 50-example eval set, and wire assertions into CI. Nothing after this part ships without evals.

4 chapters1hIntermediatePart overview →
  1. Why you can't ship without evals The vibes-driven failure mode, evals as the AI engineer's CI, and the maturity ladder that gets you off the whack-a-mole loop. beg 10 min
  2. Look at your data The single highest-ROI habit in LLM evaluation: read your traces by hand and build a failure taxonomy before automating anything. beg 15 min
  3. Your first eval set Bootstrap a 50-example eval set in a week: real or realistic inputs, binary pass/fail labels, versioned in git, growing from production failures. beg 15 min
  4. Assertions and unit tests for LLM output The cheap, deterministic checks that should gate every LLM commit before any judge model runs: schema, contains, regex, code-runs, SQL-parses, wired into pytest… beg 10 min
Part 5

Context Engineering

Building the full world the model sees: context assembly and routing, compression and budgets, memory and state, tool context, and testing what was actually assembled.

6 chapters1hIntermediatePart overview →
  1. Context is more than the prompt The six things that go into a context window every turn, why prompt engineering only covers one of them, and what the discipline of context engineering actually… int 10 min
  2. Context assembly and routing Why a 1M-token window doesn't make context free, and the route-then-assemble pipeline that decides what the model actually sees. int 10 min
  3. Compression and context budgets More context isn't better context. How lost-in-the-middle and context rot degrade long prompts, and when to trim, summarize, or re-retrieve. int 10 min
  4. Memory and state What an LLM remembers between turns, between sessions, and across years, and the techniques that make each scope work without crowding the context window. int 15 min
  5. Tool context and metadata Models have no clock, no role awareness, and no sense of locale. The metadata you inject and the way you describe tools prevent whole classes of confidently wro… int 15 min
  6. Testing and observing context Snapshot the assembled context, regression-test the assembly logic, and capture traces that can answer why the model saw what it saw. int 10 min
Part 6

Retrieval (RAG)

Retrieval done right: when you actually need RAG, chunking, vector and hybrid search, reranking, query understanding, citations, retrieval evaluation, and production indexing.

9 chapters2hIntermediatePart overview →
  1. When you actually need RAG RAG vs long context vs fine-tune as a cost decision: long context pays per query, RAG pays once to index. The default rule, the exceptions, and when RAG is the … int 10 min
  2. Document processing and chunking How parsing and chunk boundaries set your retrieval ceiling, why overlap and chunk size are eval questions, and the defaults that survive contact with real corp… int 15 min
  3. Embeddings and vector search How to pick an embedding model on your own data, why pgvector is the default store, what HNSW and IVFFlat actually do, and the honest threshold where a dedicate… int 15 min
  4. Hybrid search and reranking Why BM25 still beats vector search on exact terms, how to fuse them with RRF, and when a reranker earns its latency. int 10 min
  5. Query understanding Rewriting, multi-query, decomposition, and routing: closing the gap between what users type and what your retriever can match. int 15 min
  6. Citations and grounding Make RAG answers verifiable: span-level citations, refusal on empty retrieval, and the faithfulness vs fluency tension that decides both. int 10 min
  7. Advanced retrieval Three patterns that beat vanilla RAG on specific queries: agentic retrieval, GraphRAG, and vision-native PDF search. When each earns its complexity, and when it… adv 10 min
  8. Evaluating retrieval RAG eval is mostly retrieval eval. Recall@k and MRR in plain terms, building an eval set from real queries, and the 2x2 that separates retrieval failure from ge… int 10 min
  9. Production RAG The five operational concerns that turn a RAG prototype into a system you can run: incremental indexing, the reindex tax, multi-tenant isolation, metadata filte… adv 15 min
Part 7

Agent Engineering

Workflows versus agents, tool design, MCP and its security model, memory, human-in-the-loop, long-running agents, multi-agent systems, agent archetypes, voice agents, and choosing a framework.

13 chapters3hIntermediate to AdvancedPart overview →
  1. Workflow vs agent If you can write the control flow in Python, it's a workflow. If the model decides at runtime, it's an agent. Most production wins are workflows. int 10 min
  2. The five workflow patterns Chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer: each with code, a use case, and the failure mode that bites in production. int 15 min
  3. The agent loop Think, act, observe, and the part nobody talks about: how the loop ends. Stop conditions, reflection that actually works, and why production agents look nothing… int 15 min
  4. Tool design The agent-engineering skill: schemas that make invalid calls impossible, descriptions the model can't misread, errors it can recover from, and the right number … int 10 min
  5. MCP: Model Context Protocol How to expose tools, data, and prompts to any AI host through one open protocol; build a server and client end-to-end. int 10 min
  6. MCP security The four attack classes MCP creates: tool poisoning, cross-server shadowing, rug pulls, and injection via tool results, and the architectural defenses that actu… int 10 min
  7. Agent memory and task state The four places an agent can put information, why most teams use the wrong one, and how to design state that survives a pod restart mid-task. int 10 min
  8. Human-in-the-loop Approval gates as tool calls, confidence-based escalation, and handoff design: the three independent things you have to engineer for selective human oversight t… int 10 min
  9. Long-running agents Budgets, kill switches, checkpoints, replay, and the sync vs async choice: the harness work that keeps a multi-minute agent from burning a month's API spend. adv 15 min
  10. Multi-agent systems When multi-agent topology earns its 15x token cost, when it doesn't, and the single-writer pattern that handles most production work. adv 15 min
  11. Agent archetypes Research, Coding, Browser, and Computer Use agents differ in one thing that cascades into everything else: what the model sees at each step. int 10 min
  12. Voice agents The 800ms budget that defines voice agents, the cascaded vs native trade-off, and why barge-in is the hardest correctness problem in the stack. int 15 min
  13. Choosing a framework Frameworks buy state, retries, and tracing. Most teams don't need them. The decision rule for plain SDK, light structure, and full graph runtime. int 10 min
Part 8

Evaluation and Observability

The deep spine: LLM-as-judge and its failure modes, component evals for RAG and agents, regression gates in CI, online evaluation, tracing, monitoring, and the data flywheel.

9 chapters2hIntermediate to AdvancedPart overview →
  1. The eval taxonomy Four axes that decide which kind of evaluation answers your question: offline vs online, component vs end-to-end, reference-based vs reference-free, automatic v… int 15 min
  2. LLM-as-judge Build a second-LLM judge you can trust: binary pass/fail over Likert, prompt patterns, and the human-agreement loop you must run before deploying it. int 15 min
  3. Judge failure modes Position bias, length bias, self-preference, and style bias in LLM-as-judge pipelines, with the numbers and the mitigations that actually work in production. int 15 min
  4. Evaluating RAG and agents Component evals, tool-choice accuracy, trajectory evaluation, and the weakest-link pattern that tells you which piece of a compound system to fix. int 10 min
  5. Regression suites and CI gates Wiring evals into CI as a real merge gate: two-tier golden sets, differential thresholds over absolute floors, and how to keep flaky probabilistic tests trustwo… int 10 min
  6. Online evaluation and A/B testing Why your offline eval lied about a 3-point lift, the implicit signals that don't, and the sample sizes LLM A/B tests actually need. int 15 min
  7. Tracing Spans for LLM calls, retrieval, and agent loops; the gen_ai.* OpenTelemetry conventions; why vendor-proprietary tracing locks you in. int 9 min
  8. Monitoring and dashboards Why LLM systems need cost, latency, quality, and usage on one screen; how to alert on quality regressions and cost spikes; and which observability platform to p… int 10 min
  9. Feedback & the data flywheel How production traffic compounds into better evals and a better product, instead of becoming a treadmill of sprint-by-sprint bug fixes. int 10 min
Part 9

Security and Governance

Prompt injection, the lethal trifecta and agent security, guardrails, abuse prevention, PII and privacy, compliance, and red-teaming your own system.

7 chapters1hAdvancedPart overview →
  1. Prompt injection Why prompt injection has no general fix, the difference between direct and indirect attacks, and the defense-in-depth posture you ship with. int 10 min
  2. Agent security and the lethal trifecta The three-leg threat model for agentic exfiltration, why all three together is game-over, and how to cut at least one leg before you ship. int 10 min
  3. Guardrails Input filters, output validators, content moderation models, and the streaming-versus-validation problem nobody warns you about. int 10 min
  4. Abuse prevention Rate limits, spend caps, per-tenant budgets, and the behavioral signals that catch automated abuse before it shows up on the bill. int 10 min
  5. PII and privacy Detect and redact before the model call, navigate the controller-processor-subprocessor chain, and pick the right retention and residency knobs. adv 10 min
  6. Compliance and audit What engineers actually build for SOC 2 and the EU AI Act: structured audit trails, tamper-evident retention, and the model documentation regulators look for. adv 15 min
  7. Red-teaming Automated probe suites, manual protocols, and continuous red-teaming in CI: how to attack your own AI system before someone else does. adv 10 min
Part 10

Production AI

Operating AI in production: hosted versus self-hosted, the customization menu, cost and latency engineering, eval-gated rollouts, SLOs, incident response, and AI infrastructure patterns.

8 chapters1hAdvancedPart overview →
  1. Hosted vs self-hosted The four-line TCO model that decides when self-hosting beats per-token billing, why GPU utilization is the silent multiplier, and the 'start hosted, migrate the… int 9 min
  2. Self-hosting essentials What you actually need to know to run an open-weight model competently: vLLM as the default server, KV cache as the resource constraint, continuous batching, an… int 9 min
  3. The customization menu Fine-tuning, LoRA, and distillation as decisions, not techniques: when each is the right answer, what it costs, and the contamination trap that makes broken sys… int 10 min
  4. Cost engineering Five orthogonal levers (caching, batching, routing, quantization, distillation) that compound multiplicatively to cut a five-figure monthly bill by ~80% without… int 10 min
  5. Latency engineering Why LLM latency isn't a single number: the waterfall, TTFT vs total time, parallel tool calls, prefix caching, and the perceived-latency tricks that close the g… int 10 min
  6. Shipping changes Prompt and model changes are deployments. Eval-gated promotion, shadow traffic, canary rollout, and instant rollback as the discipline that turns 'just edit the… int 10 min
  7. SLOs and incident response AI failures return HTTP 200 with garbage. The four AI incident classes (quality regression, injection exploit, cost runaway, hallucination at scale), the semant… int 10 min
  8. AI infrastructure patterns The four AI-specific infrastructure components: vector stores in production, semantic caching, durable agent job queues, and embedding pipelines. Everything gen… int 15 min
Part 11

AI Product Engineering

AI UX, failure recovery, product metrics beyond accuracy, experimentation, cost modeling and pricing, and feedback loops that compound.

6 chapters1hIntermediatePart overview →
  1. AI UX Streaming as a trust signal, not a transport detail. The three failure modes (blank screen, mid-stream stall, unverifiable claim) and what to build for each. int 10 min
  2. Failure recovery UX AI failures aren't edge cases. Designing the fallback cascade, retry affordances, and human handoffs so the failure you can't prevent doesn't destroy trust. int 10 min
  3. AI product metrics Beyond accuracy. Task success, deflection vs. containment vs. resolution, escalation as a quality signal, time-to-completion, and AI-feature retention. int 10 min
  4. Experimentation A/B testing AI features when variance is 2-5x higher than deterministic features, the treatment can drift mid-experiment, and 'no significant effect' usually me… adv 10 min
  5. Cost modeling and pricing Why marginal cost matters again. Cost-per-task (not per-token), the four pricing architectures, and the math that says when a feature is too expensive to ship. int 10 min
  6. Feedback and growth loops Most teams have a log file, not a flywheel. The three preconditions for a working data loop, and why the loop itself (not the data) is the durable advantage. adv 15 min