Lab: evaluate a fine-tune properly: intervals, paired tests, error analysis and a regression suite
Take the tuned model from the LoRA lab and decide whether to ship it. Run both models over held-out tickets, compute metrics with confidence intervals and a paired test, read the errors (priority was pulled toward 'medium'), and discover what the fine-tune broke: on a 12-prompt regression suite it fell from 10 to 7 correct and began wrapping answers in quotes.
You trained a model and its held-out score went from 0% to 72%. Should you ship it? That depends on questions the headline number cannot answer: How sure are you of the 72%? Is the improvement real or noise? Where does it still fail? And what did the fine-tune break? This lab answers each with code, using the real model from the LoRA lab. It builds on Merging, evaluating and serving and Metrics and statistical honesty.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.