Lab: build an eval harness
Write a small reusable harness: a JSONL golden set, concurrent runs against a local model with output caching, deterministic and judge scorers, Wilson intervals, a paired regression gate and a CI hook. About 200 lines of Python you can reuse on every project.
Build a harness you will reuse on every LLM project. By the end you will have a golden set, a runner, scorers (rule-based and optionally a judge), a regression gate that compares against a baseline using paired statistics, and a CI hook. The whole thing is plain Python with one dependency.
Time: about an hour. You need: Python 3.10+, a local model behind an OpenAI-compatible endpoint (Ollama, llama.cpp server or vLLM) and a task of your own to evaluate. Background: golden datasets, metrics and statistics, regression testing in CI.
Pro content
Keep reading with Inference Academy Pro
Unlock this full lab, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.