LLM-as-judge done properly
Use a model to grade open-ended outputs while controlling for position, verbosity and self-preference bias: judging modes, rubric design, pairwise code with order swapping, calibration against humans with Cohen's kappa, correcting for judge error and the limits of the method.
Open-ended outputs (summaries, explanations, support replies, creative text) have no single correct string, so exact-match scoring fails. Human grading is accurate but slow and expensive. A strong model can grade against a rubric at a tiny fraction of the cost, if you handle its weaknesses and validate it. Done carelessly, an LLM judge produces confident, repeatable, wrong numbers, which is worse than no number.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.