Evaluation in Practice
Benchmarks and contamination, testing RAG and agents, regression tests in CI, online evaluation and a harness lab.
0 of 7 complete
- ▶ Up next15 min+50 XPPublic benchmarks and contaminationWhat popular benchmarks measure and miss, how contamination and saturation inflate scores, how to detect…
- Pro15 min+50 XPTesting RAG pipelines and agentsSeparate retrieval quality from generation quality, compute Recall@k, MRR and nDCG by hand, measure faithfulness and…
- Pro14 min+50 XPRegression testing in CIRun evaluations automatically on every prompt, model or retrieval change: tiers, baselines, noise-calibrated gates…
- Pro15 min+50 XPOnline evaluation, A/B tests and monitoringMeasure quality on real traffic: feedback and behavioural signals, stratified sampling and judging, sample-size…
- LabPro60 min+100 XPLab: build an eval harnessWrite a small reusable harness: a JSONL golden set, concurrent runs against a local model with output caching…
- Pro15 min+50 XPHuman evaluation operations: guidelines, sampling, agreement and costRun human evaluation as an operation: write guidelines and rubrics, choose pairwise versus absolute scoring, sample…
- Pro16 min+50 XPEvaluating code, multimodal and safety behaviourDomain-specific evaluation: executable tests and pass@k for code, document and image tasks with field-level scoring…