Part 10: Production AI — The AI Engineering Handbook

10.0intermediate
Hosted vs self-hosted
The four-line TCO model that decides when self-hosting beats per-token billing, why GPU utilization is the silent multiplier, and the 'start hosted, migrate the hot path' default for 2026.
9 min
10.1intermediate
Self-hosting essentials
What you actually need to know to run an open-weight model competently: vLLM as the default server, KV cache as the resource constraint, continuous batching, and how to size GPUs before you spend.
9 min
10.2intermediate
The customization menu
Fine-tuning, LoRA, and distillation as decisions, not techniques: when each is the right answer, what it costs, and the contamination trap that makes broken systems look healthy.
10 min
10.3intermediate
Cost engineering
Five orthogonal levers (caching, batching, routing, quantization, distillation) that compound multiplicatively to cut a five-figure monthly bill by ~80% without changing model quality.
10 min
10.4intermediate
Latency engineering
Why LLM latency isn't a single number: the waterfall, TTFT vs total time, parallel tool calls, prefix caching, and the perceived-latency tricks that close the gap UX-side.
10 min
10.5intermediate
Shipping changes
Prompt and model changes are deployments. Eval-gated promotion, shadow traffic, canary rollout, and instant rollback as the discipline that turns 'just edit the prompt' into safe production change.
10 min
10.6intermediate
SLOs and incident response
AI failures return HTTP 200 with garbage. The four AI incident classes (quality regression, injection exploit, cost runaway, hallucination at scale), the semantic SLIs that detect them, and the runbooks that contain them.
10 min
10.7intermediate
AI infrastructure patterns
The four AI-specific infrastructure components: vector stores in production, semantic caching, durable agent job queues, and embedding pipelines. Everything generic cross-links to HLD.
15 min