Your first evaluation harness: cases, a scorer and a report

A harness is a loop: for each case, run the system, score the output, and summarise. Build the smallest one first; every later lesson in the course extends it.

#The harness

Save as harness.py:

python

import random, math
random.seed(3)

cases = [("2+2", "4"), ("capital of France", "Paris"), ("3*3", "9"), ("opposite of hot", "cold"),
         ("5-1", "4"), ("capital of Italy", "Rome"), ("10/2", "5"), ("opposite of up", "down")]

def fake_model(q):                       # a stand-in that is right about 75% of the time
    answers = dict(cases)
    return answers[q] if random.random() < 0.75 else "I don't know"

def run(n_repeats=25):
    scores = []
    for _ in range(n_repeats):
        scores.append(sum(fake_model(q).strip().lower() == a.lower() for q, a in cases) / len(cases))
    return scores

scores = run()
mean = sum(scores) / len(scores)
sd = math.sqrt(sum((s - mean) ** 2 for s in scores) / (len(scores) - 1))
print(f"mean pass rate over {len(scores)} runs: {mean:.3f}   spread (sd): {sd:.3f}")
print("worst run:", min(scores), "best run:", max(scores))

Output (with this seed):

text

mean pass rate over 25 runs: 0.745   spread (sd): 0.155
worst run: 0.375 best run: 1.0

Look at what the output teaches. The average is near 0.75, as built. But single runs range from 0.375 to 1.0 on just 8 cases. A single run of this evaluation would have told you almost nothing. That is the whole case for repeats, more cases and intervals.

#The three parts

Part	In the code	What to replace it with
Cases	the `cases` list	A real dataset: questions, expected answers or criteria (golden datasets)
Runner	`fake_model`	A call to your model or pipeline
Scorer	exact match after lower-casing	Rules, rubric, or an LLM judge (judge)

#Swapping in a real model

Replace fake_model with a function that calls your server, for example an OpenAI-compatible endpoint from a local runtime:

python

import json, urllib.request

def real_model(q):
    body = json.dumps({"model": "MODEL", "temperature": 0,
                       "messages": [{"role": "user", "content": q}]}).encode()
    req = urllib.request.Request("http://localhost:8080/v1/chat/completions", body,
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["choices"][0]["message"]["content"]

Use temperature 0 for comparisons (and say so in your report), or keep a higher temperature and repeat runs. Adjust the URL and model name to your server.

#Make the scorer fit the task

Answer type	Scorer
A fixed short answer	Exact match after normalising case and whitespace
A number	Parse and compare with a tolerance
Structured output	Validate against a schema, then compare fields
Free text	A rubric with a judge model, checked against human labels
Code	Run the tests

#What to add next

More cases, drawn from real usage, with slices (categories, difficulty).
Save every output to a file so you can read failures.
Per-case results, not only the average, so you can compare versions on the same cases.
A report with the mean, an interval, and the worst failures (the statistics).
A command (python harness.py) that runs in CI (regression testing in CI).

The labs grow this file into a full harness (the lab).

#Exercise

Change the stand-in model's success rate from 0.75 to 0.9 and run it. Predict the mean and whether the spread rises or falls.
Double the number of cases by duplicating the list. What happens to the spread, and why?

Answers: 1. The mean moves near 0.9 and the spread falls, since a higher rate has less variance (p(1-p) is smaller). 2. The spread shrinks, roughly by the square root of two, because each run averages over more cases (the duplicates are not new information in the strict sense, so real new cases help more).

Next: why vibes fail: the case for evaluation.