Evaluation Fundamentals

Golden datasets, metrics, statistical honesty and LLM-as-judge done properly.

1. 1▶ Up next13 min+50 XPWhy vibes fail: the case for evaluationWhy trying a few prompts by hand misleads you, with a worked example of fixing one case and silently breaking others…
2. 215 min+50 XPBuilding a golden datasetWhere evaluation examples come from, how many you need and why, how to label and measure agreement (with a worked…
3. 316 min+50 XPMetrics and statistical honestyChoose metrics that match the task, put error bars on every score (Wilson intervals, paired comparison, McNemar and…
4. 4Pro15 min+50 XPLLM-as-judge done properlyUse a model to grade open-ended outputs while controlling for position, verbosity and self-preference bias: judging…
5. 5Pro15 min+50 XPError analysis: turning failures into a roadmapLook at your failures systematically: sample, label with a taxonomy, count, and prioritise. Covers building a failure…