The statistics you need: averages, spread and confidence intervals

Evaluation is statistics in a thin disguise. You do not need much, but you need it to be right.

#Averages and spread

A pass rate is the share of test cases that pass. It is an estimate of how the system would behave on all similar cases. A different sample of cases would give a different number, and the question is how different.

import random
random.seed(1)
results = [1]*37 + [0]*13           # 37 of 50 test cases passed
acc = sum(results) / len(results)
print("accuracy", acc)              # 0.74

#Bootstrap: let the data tell you the spread

Resample the results with replacement many times, compute the pass rate each time, and look at the middle 95 percent:

boots = []
for _ in range(10000):
    s = [random.choice(results) for _ in results]
    boots.append(sum(s) / len(s))
boots.sort()
print("95% interval:", boots[250], "to", boots[9749])

Output: 95% interval: 0.62 to 0.86. The honest report is 74%, with a plausible range of about 62% to 86%.

#The quick formula

For a pass rate p on n cases, the standard error is sqrt(p(1-p)/n), and a rough 95% interval is p ± 1.96 × se:

import math
se = math.sqrt(acc * (1 - acc) / len(results))
print("approx 95%:", round(acc - 1.96 * se, 3), "to", round(acc + 1.96 * se, 3))   # 0.618 to 0.862

It agrees with the bootstrap here.

#How big a test set do you need?

The interval width shrinks with the square root of n:

To detect a 3 point improvement reliably you need on the order of a thousand cases or a paired comparison.

#Comparing two systems

Model A passes 37/50 (74%), model B passes 41/50 (82%). The gap is 8 points, and each interval is roughly ±12 points, so the intervals overlap heavily: this is not evidence that B is better.

Two ways to do better:

1. More cases.
2. A paired comparison. Run both on the same cases and look only at where they disagree. If B wins 7 of the 8 disagreements, that is much stronger than the raw pass rates suggest, because case difficulty cancels out. The harness lesson uses this idea in regression tests (regression testing).

#Other sources of noise

• Sampling randomness. At temperature above zero the same prompt gives different answers. Run each case several times or use greedy decoding for comparisons, and say which.
• Judge noise. An LLM judge is itself uncertain (LLM as judge).
• Dataset drift. Real traffic changes; a stale test set measures the past.
• Many comparisons. If you test twenty variants, one will look good by chance. Hold out a final test set.

#Reporting checklist

1. State the number of cases.
2. Give an interval, not only a point.
3. Say how outputs were sampled (temperature, number of runs).
4. Show per-slice results (categories, difficulty) as well as the average.
5. Include examples of failures.

#Exercise

1. A system passes 90 of 100 cases. Give the approximate 95% interval.
2. A prompt change moves the pass rate from 90 of 100 to 93 of 100. Is that convincing? What would you do?

Answers: 1. p = 0.9, se = sqrt(0.9 × 0.1 / 100) = 0.03, so about 0.84 to 0.96. 2. No: a 3 point change is inside the noise for 100 cases. Add more cases, and run both versions on the same cases looking at the disagreements.

Next: your first evaluation harness.