Preference tuning and distillation: DPO, RLHF and small models
Learn from comparisons rather than single answers: the preference data format, RLHF versus DPO with the DPO loss worked numerically (runnable), reward models and judges, over-optimisation risks, and distilling a large model's behaviour into a small, cheaper one.
17 min·Verified 2026-11Pro+50 XP
Supervised fine-tuning shows the model one good answer. Preference tuning shows it which of two answers is better, which is often easier to judge than to write and can shape qualities hard to specify: helpfulness, tone, concision, safety.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.