Metrics and statistical honesty

Two questions decide whether an evaluation is trustworthy: are we measuring the right thing? and could this result be luck? The first is about metrics, the second about statistics. Most evaluation mistakes in LLM projects are one of these two.

#Choose metrics by task

#Precision, recall and F1

For a classifier with a positive class (for example "this ticket is urgent"):

precision = true positives / (true positives + false positives)   # of flagged, how many were right
recall    = true positives / (true positives + false negatives)   # of real ones, how many were found
F1        = 2 × precision × recall / (precision + recall)

Worked example: 1,000 tickets, 20 truly urgent. The system flags 25, of which 16 are urgent. Precision 16/25 = 64%, recall 16/20 = 80%, F1 0.71. Accuracy would be (16 + 971) / 1000 = 98.7%, which looks wonderful and hides the fact that 4 urgent tickets were missed and 9 false alarms raised. Always use the metric that reflects the cost: for urgent tickets recall matters most; for spam filters precision may.

#pass@k

If a model solves a problem in c of n samples, the chance at least one of k samples passes is:

pass@k = 1 − C(n − c, k) / C(n, k)

For n = 10, c = 3: pass@1 = 0.30; pass@5 = 1 − C(7,5)/C(10,5) = 1 − 21/252 = 0.92. It rewards diversity and is suitable where trying again is cheap (code with tests). For reliability, where every attempt must work, look at pass^k instead: the chance that all k attempts succeed, p^k (0.8³ = 0.51). Report which you use and the sampling settings.

#Beware proxy metrics

Overlap metrics (BLEU, ROUGE) reward matching words, not being right. Perplexity measures how surprised a model is by text, not whether it is useful for your task (measuring quality). Prefer task-specific, checkable metrics where you can, and verify any proxy against human judgement on a sample.

A good habit: when you adopt a metric, read twenty outputs it scored high and twenty it scored low. If you disagree with it often, the metric is wrong for your task.

#Every score needs an error bar

If you flip a biased coin 100 times you will not get exactly its true rate. Likewise, your pass rate on n examples is an estimate. For a pass rate p:

standard error = sqrt(p × (1 − p) / n)
approx 95% interval = p ± 1.96 × standard error

To halve the margin you need four times the data. To detect a 2-point improvement you need thousands of examples, unless you compare paired results (below).

#A better interval for small samples

The simple formula misbehaves for small n or extreme rates (it can give intervals above 100 percent or collapse to zero width at 10 of 10). Use the Wilson interval:

import math

def wilson(k, n, z=1.96):
    if n == 0: return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, centre - half), min(1.0, centre + half)

print(wilson(9, 10))    # (0.60, 0.98): 9 of 10 is compatible with anything from 60% to 98%
print(wilson(10, 10))   # (0.72, 1.00): 10 of 10 does not prove perfection

Read it as: nine of ten correct is compatible with a true rate anywhere from about 60 to 98 percent. Ten of ten still allows a true rate down to 72 percent. This is why "it passed all our tests" is weaker than it sounds when the tests are few.

#Compare versions properly

Run both versions on the same examples and look at pairs:

Only the off-diagonal cells carry information. If B wins 9 cases that A got right and A wins 6 that B got right, the net difference is 3 cases out of 15 disagreements. A simple sign-test z-score is (9 − 6) / sqrt(15) ≈ 0.77, nowhere near the 1.96 needed for significance. McNemar's test (with continuity correction) gives χ² = (|9 − 6| − 1)² / 15 = 0.27, a p-value around 0.6.

import math

def paired(a, b):
    """a, b: dicts id -> 0/1 on the same cases. Returns (only_a, only_b, z)."""
    ids = a.keys() & b.keys()
    only_a = sum(1 for i in ids if a[i] and not b[i])
    only_b = sum(1 for i in ids if b[i] and not a[i])
    n = only_a + only_b
    z = (only_b - only_a) / math.sqrt(n) if n else 0.0     # positive means B is better
    return only_a, only_b, z

When there are few disagreements, the data simply cannot tell the versions apart, however many agreements there are. The cure is more cases, or a bigger real difference.

#Bootstrap for any metric

A bootstrap resamples your cases to see how much the score would wobble. It works for averages, F1, judge scores and anything else:

import random

def bootstrap_diff(a, b, iters=5000):
    """a, b: lists of per-case scores on the same cases. 95% interval for mean(b) - mean(a)."""
    n = len(a); diffs = []
    for _ in range(iters):
        idx = [random.randrange(n) for _ in range(n)]
        diffs.append(sum(b[i] - a[i] for i in idx) / n)
    diffs.sort()
    return diffs[int(0.025 * iters)], diffs[int(0.975 * iters)]

If the interval includes zero, you have not shown a difference. Resample cases, not individual outputs, and keep a case's results from both versions together (that is what makes it paired).

#How big a test set do you need?

Decide the smallest difference worth detecting, and size accordingly. For unpaired comparison of two rates near p with 95 percent confidence and 80 percent power:

n per version ≈ 16 × p × (1 − p) / d²

For p = 0.8 and d = 0.05 (5 points): 16 × 0.16 / 0.0025 = 1,024 per version. For d = 0.10: 256. Paired comparison needs far fewer when the two versions mostly agree, because only the disagreements matter, but you do not know that in advance. If you cannot reach the size you need, say so, and treat small differences as hypotheses, not findings.

#Non-determinism

If you sample with temperature above zero, repeat each example several times and average, or fix the seed and temperature to 0 for comparisons. Always record settings with results. Repeats reduce the noise from sampling; they do not reduce the noise from having few examples. Both matter.

Measure your noise directly: run the unchanged system three to five times and look at the spread of the pass rate. That spread is the smallest change you can honestly claim to see (CI regression testing).

#Multiple comparisons and the winner's curse

Test twenty prompt variants on the same 100 cases and one will look about 6 points better than baseline purely by chance. Picking the best of many noisy results systematically overstates it. Defences:

• Confirm winners on a held-out set that you did not use to choose.
• Decide your comparisons before running them; do not go fishing.
• Be sceptical of small wins from many variants.
• Use a stricter threshold when you try many things (Bonferroni-style: divide the significance level by the number of comparisons).

#Slices and averages

An overall pass rate can rise while a segment collapses (one language, one category, one customer). Report per-tag results, with intervals, and set separate gates for critical slices. Be careful: each slice has fewer cases and a wider interval, so a one-point drop in a 15-case slice means nothing; a 40-point drop means a lot.

#Report honestly

• Show the score with its interval and n.
• Break down by tag, not just the mean.
• Say which version of the dataset was used (golden datasets).
• Look at the actual failures: numbers tell you how much, reading outputs tells you why.
• Be wary of multiple comparisons.
• Report what you did not test.

A results line that is honest looks like: "Pass rate 84% (95% CI 76 to 90, n = 100) on dataset v3; vs baseline 80%: paired difference not significant (7 gained, 3 lost). Refund slice 70% (n = 20, CI 48 to 86)."

#Common mistakes

1. Reporting a single number with no interval.
2. Using accuracy for rare events.
3. Comparing two versions on different samples (unpaired and noisy).
4. Believing small differences on small sets.
5. Selecting the best of many variants and reporting its score as if untouched.
6. Using proxy metrics you never validated.
7. Ignoring sampling noise when temperature is above zero.

#Exercise

Version A scores 41 of 50 (82%) and version B scores 44 of 50 (88%) on the same cases. B passes 5 cases A failed; A passes 2 cases B failed.

1. What are the Wilson intervals roughly for each, and do they overlap?
2. What does the paired analysis say?
3. What would you do?

Answers:

1. Roughly A: 69 to 91 percent; B: 76 to 94 percent. They overlap heavily.
2. Seven disagreements, 5 for B and 2 for A: z = (5 − 2) / sqrt(7) ≈ 1.13, not significant (p around 0.3).
3. Do not claim B is better. Collect more cases (a few hundred), look at the seven disagreement cases for a pattern (perhaps B fixes a specific category), and re-run. If B is cheaper or faster at equal quality, you may still prefer it on other grounds, saying so explicitly.