Evaluation and Observability
The deep spine: LLM-as-judge and its failure modes, component evals for RAG and agents, regression gates in CI, online evaluation, tracing, monitoring, and the data flywheel.
- Chapters
- 9
- Hours
- 2
- Difficulty
- Intermediate to Advanced
- 8.0intermediate
The eval taxonomy
Four axes that decide which kind of evaluation answers your question: offline vs online, component vs end-to-end, reference-based vs reference-free, automatic vs judge vs human.
- 8.1intermediate
LLM-as-judge
Build a second-LLM judge you can trust: binary pass/fail over Likert, prompt patterns, and the human-agreement loop you must run before deploying it.
- 8.2intermediate
Judge failure modes
Position bias, length bias, self-preference, and style bias in LLM-as-judge pipelines, with the numbers and the mitigations that actually work in production.
- 8.3intermediate
Evaluating RAG and agents
Component evals, tool-choice accuracy, trajectory evaluation, and the weakest-link pattern that tells you which piece of a compound system to fix.
- 8.4intermediate
Regression suites and CI gates
Wiring evals into CI as a real merge gate: two-tier golden sets, differential thresholds over absolute floors, and how to keep flaky probabilistic tests trustworthy.
- 8.5intermediate
Online evaluation and A/B testing
Why your offline eval lied about a 3-point lift, the implicit signals that don't, and the sample sizes LLM A/B tests actually need.
- 8.6intermediate
Tracing
Spans for LLM calls, retrieval, and agent loops; the gen_ai.* OpenTelemetry conventions; why vendor-proprietary tracing locks you in.
- 8.7intermediate
Monitoring and dashboards
Why LLM systems need cost, latency, quality, and usage on one screen; how to alert on quality regressions and cost spikes; and which observability platform to pick in 2026.
- 8.8intermediate
Feedback & the data flywheel
How production traffic compounds into better evals and a better product, instead of becoming a treadmill of sprint-by-sprint bug fixes.