Skip to content
⌕Go Pro
☰

Learn

CoursesCareer pathsAll lessonsGlossary

Tools

CalculatorsCan I run it?PlaygroundBenchmarksFixes
Pricing

Know whether your AI actually works

LLM Evaluation and Testing

Replace vibes with evidence: golden datasets, metrics, statistics, LLM-as-judge, RAG and agent testing, regression suites in CI and production monitoring.

2 tracks · 12 lessons · about 4 hours · 4 free lessons · 4 animated diagrams

Start the first lesson freeSee Pro

What you will be able to do

  • ✓Build a golden dataset that reflects real usage
  • ✓Choose metrics and report results with honest error bars
  • ✓Use LLM judges without falling for their biases
  • ✓Gate releases with automated regression evals

Who it is for

Anyone who changes prompts, models or retrieval and needs to know if it got better.

Prerequisites

Basic Python. Having built one LLM feature helps.

Curriculum

Evaluation Fundamentals

Intermediate

Golden datasets, metrics, statistical honesty and LLM-as-judge done properly.

  1. 1Why vibes fail: the case for evaluation1 diagramFree13 min
  2. 2Building a golden datasetFree15 min
  3. 3Metrics and statistical honesty1 diagramFree16 min
  4. 4LLM-as-judge done properly1 diagramPro15 min
  5. 5Error analysis: turning failures into a roadmapPro15 min

Evaluation in Practice

Advanced

Benchmarks and contamination, testing RAG and agents, regression tests in CI, online evaluation and a harness lab.

  1. 1Public benchmarks and contaminationFree15 min
  2. 2Testing RAG pipelines and agentsPro15 min
  3. 3Regression testing in CI1 diagramPro14 min
  4. 4Online evaluation, A/B tests and monitoringPro15 min
  5. 5Lab: build an eval harnessLabPro60 min
  6. 6Human evaluation operations: guidelines, sampling, agreement and costPro15 min
  7. 7Evaluating code, multimodal and safety behaviourPro16 min