Public benchmarks and contamination

Public benchmarks are useful for shortlisting, and poor at deciding. They answer "is this model in the right neighbourhood?", not "will this work for us?". Knowing how they break is what lets you use them without being fooled.

#What common benchmarks measure

Names and versions change often, so check what the current leaderboards actually run and when they were last updated.

#Problems to know about

1. Contamination. Benchmarks are public, so their questions (and answers) may appear in training data. Scores then measure memory as well as skill.
2. Saturation. When top models score 90 percent or more, differences are small and noisy, and the benchmark no longer discriminates.
3. Format sensitivity. Small changes to the prompt template, the number of examples or the answer-extraction method can move scores by several points.
4. Overfitting to the benchmark. Models and fine-tunes can be tuned for specific leaderboards, intentionally or not.
5. Mismatch with your task. A model strong at multiple-choice trivia may be weak at your extraction or support task.
6. Unreproducible numbers. Vendor-reported scores often use different prompts, shots or settings than third-party runs.
7. Quantization effects. A leaderboard score is for the original weights. Your Q4 build may differ slightly (measuring quality).
8. Noise. Differences of one or two points are usually within the benchmark's own error bar; see metrics and statistics.
9. Narrow coverage. None of them test your data, your prompts, your retrieval, your formats or your safety requirements.

#How contamination happens

• Benchmark files are copied into GitHub repositories, blog posts and papers that end up in web-scale training crawls.
• Models are fine-tuned on synthetic data generated by other models that had seen the benchmark.
• Instruction-tuning datasets sometimes include benchmark training splits, or paraphrases of test items.
• Deliberate benchmark optimisation exists, although many cases are accidental.

Contamination does not make a model bad. It makes its benchmark score uninformative about capability on new problems.

#Detecting contamination

• Check dates. Was the benchmark released before the model's training cutoff? If so, assume exposure is possible.
• Perturbation test. Compare performance on the original benchmark and on freshly written or perturbed versions: reordered answer options, rephrased questions, changed numbers in maths problems. A big gap suggests memorisation.
• Canary and overlap checks. Look for n-gram overlap between test items and known corpora where you can; some benchmarks embed canary strings that should not appear in model outputs.
• Completion test. Give the model the first half of a benchmark question and see whether it completes the rest verbatim.
• Prefer benchmarks that refresh (new problems added over time, "live" benchmarks) or hold back private test sets.

A worked perturbation check: you take 100 benchmark maths questions and write variants with different numbers. The model scores 90 percent on the originals and 68 percent on the variants. Allowing for noise (about ±9 points at n = 100), a 22-point gap is clear evidence that part of the original score was recall.

#Chat arenas and Elo ratings

Arena-style leaderboards collect human votes between two anonymous model answers and fit an Elo-like rating. The expected win probability of A over B is:

P(A beats B) = 1 / (1 + 10^((R_B − R_A) / 400))

Useful facts: a gap of 30 or 40 points is only a 54 to 56 percent win rate, which is hard to feel in practice and is usually within the statistical uncertainty (look for confidence intervals on the board). Arenas reward style (longer, better-formatted answers) as well as correctness, and their voters are not your users. Treat them as a measure of general chat preference, not task accuracy.

#Use benchmarks well

1. Shortlist 3 to 5 candidate models from reputable leaderboards for your size and licence needs (choosing a model).
2. Run them on your golden set with your prompts and your quantization (golden datasets).
3. Compare cost and speed, not only quality (measuring speed).
4. Spot-check by reading outputs: twenty failures teach more than a point of MMLU.
5. Re-evaluate when new models arrive, using the same harness so comparisons are cheap.
6. Prefer third-party reproduced numbers to vendor claims, and check the prompt format they used.

#Reading a model card critically

Be wary of cards that show only benchmarks they win.

#Run benchmarks yourself

Tools such as lm-evaluation-harness run many standard benchmarks against local OpenAI-compatible endpoints. Use them to reproduce numbers on your own hardware and quantization:

pip install lm-eval
lm_eval --model local-completions \
  --model_args model=my-model,base_url=http://localhost:8000/v1/completions,num_concurrent=4 \
  --tasks hellaswag --limit 200

Flags vary by version; check the tool's documentation. --limit is for quick tests only, because small samples give wide error bars. When comparing models, keep everything identical: tasks, number of shots, chat template or not, answer extraction, and generation settings. When comparing quantizations of one model, run the same tasks on each and look at paired differences, not only the totals.

Reproduction is also a sanity check on the vendor: if your run is many points below the published score at the same settings, find out why before trusting either.

#Build a small private benchmark

The most reliable defence against contamination and mismatch is a benchmark nobody else has: your own golden set, kept out of public repositories and training data. Even 100 well-chosen cases will tell you more about your decision than a leaderboard. Treat it as sensitive: if you publish it, you will eventually contaminate it (golden datasets).

#A decision procedure

1. Write requirements (task, languages, context, latency, memory, licence).
2. Filter by hard constraints: memory, licence, context length.
3. Shortlist from public benchmarks relevant to your task type.
4. Evaluate finalists on your private set, at the quant you will deploy.
5. Break ties with cost, speed and failure-mode review.
6. Record the decision with evidence and a re-evaluation date.

#A healthy attitude

A benchmark score says "this model is probably in the right neighbourhood". Your own evaluation says "this model works for us". Trust the second.

#Common mistakes

1. Choosing the top of a leaderboard without testing.
2. Comparing numbers from different prompts and settings.
3. Ignoring error bars on small differences.
4. Treating arena rank as accuracy.
5. Publishing your own test set and then relying on it.
6. Assuming vendor numbers are reproducible.

#Exercise

Model X scores 85 percent on a popular benchmark, model Y 82 percent. X was released before the benchmark's last update; Y is newer and trained with a later cutoff. Your task is extracting fields from invoices.

1. What is wrong with picking X because of the benchmark?
2. What would you do instead?

Answers:

1. The 3-point gap is within typical noise, X's score may include contamination (or Y's may), and a general knowledge benchmark says little about invoice extraction.
2. Run both (and a third candidate) on 100 of your own invoices at the quantization you plan to deploy, scoring field accuracy and format validity, compare paired per-case results and speed, and choose on that evidence (eval harness lab).