Regression testing in CI
Run evaluations automatically on every prompt, model or retrieval change: tiers, baselines, noise-calibrated gates, paired comparisons, output caching, a GitHub Actions workflow, readable failure reports and how to update baselines deliberately.
Unit tests protect code. Evals protect behaviour. In an LLM application, the behaviour that matters changes when you edit a prompt, swap a model, requantize weights, change a retrieval parameter or upgrade a runtime, none of which a normal test suite notices. Regression testing in CI means every such change is automatically scored against your golden set, and a regression blocks the merge unless someone decides otherwise on purpose.
1. A pull request changes the prompt
A teammate shortens the system prompt to save tokens.
Numbers are illustrative. Run a small smoke set on every PR and the full set nightly or before release.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.