What evaluation is for, and the plan
Why evaluation is the foundation of every LLM change, the loop it supports, the route from golden datasets to regression gates and online monitoring, times and checkpoints, and what you will have built by the end.
#Why this course exists
Teams building with LLMs often decide by feel: "it seems better". Then a prompt change fixes one case and breaks ten others nobody tested, and nobody notices for a week. Evaluation replaces feel with evidence. It lets you change prompts, models and retrieval with confidence.
#The loop
- 1Define successwhat does a good answer look like?
- 2Build a datasetreal cases with expected behaviour
- 3Measuremetrics, judges, humans, with honest statistics
- 4Analyse errorswhere and why does it fail?
- 5Change one thing
- 6Measure againdid it help, beyond noise?
#The route
| Part | Question | Lessons | Time |
|---|---|---|---|
| Start here | What do I need to know first? | 3 | about 1.5 hours |
| Fundamentals | How do I build datasets, metrics, judges and read results? | 5 | about 5 hours |
| In practice | How do I test RAG and agents, run regressions in CI, evaluate online, and operate human review? | 7 | about 8 hours |
| Capstone | Can I evaluate a real feature end to end? | 1 | about 5 hours |
Checkpoints: after Start here you can run a small harness and read its spread; after Fundamentals you have a golden dataset and a judge you have checked against humans; after In practice you have a regression gate that blocks a bad change.
#What you will be able to do
- Build a golden dataset that reflects real usage.
- Choose metrics and report them with uncertainty.
- Use an LLM as a judge, and check that the judge agrees with humans.
- Evaluate retrieval and agents, not just single answers.
- Put regression tests in CI and monitor quality online.
- Run human evaluation without fooling yourself.
#Prerequisites
- Basic Python.
- Having built one LLM feature helps, so you know what you are measuring (applications track).
- Statistics: a primer comes next, so you do not need prior knowledge.
#Exercise
You change a prompt and your three favourite test questions look better. Why is that not evidence, and what would be?
Answer: three hand-picked cases cannot show that the change helps in general; they may even be the cases you tuned for. Evidence is a larger, representative dataset scored the same way before and after, with a comparison that accounts for run-to-run noise.
Next: the statistics you need.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. What is evaluation for?
2. Why is a single score rarely enough?
3. Where does evaluation run in a mature team?
Found this useful?
Send it to someone who is learning the same thing.