Measuring quality: perplexity, KL divergence and task evals
How to tell whether a quant, a model swap or a prompt change made things better or worse: perplexity and KL divergence for quantization, task benchmarks, building a golden set, and the statistics that stop you fooling yourself.
14 min·Verified 2026-11Pro+50 XP
Speed is easy to measure: run a benchmark, read a number. Quality is where people fool themselves. "It seems fine" after five prompts is not evidence, and a lower perplexity does not mean a better assistant. This lesson gives you a layered toolkit, from cheap proxies to your own task evaluation, and the discipline to read the results honestly.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.