Online evaluation, A/B tests and monitoring
Measure quality on real traffic: feedback and behavioural signals, stratified sampling and judging, sample-size maths, drift detection with PSI, A/B test design (peeking, novelty, guardrails), privacy-safe logging and the loop that turns failures into tests.
Offline evaluations tell you a change is probably safe on the cases you thought of. Online evaluation tells you whether real users, with real and ever-changing inputs, are actually better off, and notices when quality moves without any deploy. This lesson covers the signals available in production, how to sample and judge them statistically, how to run experiments responsibly, and how to feed what you learn back into your tests.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.