Building a golden dataset

Your evaluation is only as good as its dataset. A golden dataset is a curated collection of inputs paired with a way to judge outputs. It is the most valuable evaluation asset you will own: models, prompts and tools will change many times, but a good dataset keeps measuring the same thing.

#Where examples come from

1. Real traffic (best). Sample production requests, remove personal data, keep the messy ones. Real users surprise you with typos, mixed languages, vague requests and questions you never imagined.
2. Subject-matter experts. Ask the people who know the domain for tough, realistic questions and their ideal answers.
3. Failures. Every bug report and bad answer becomes a permanent test case. This is how the dataset improves with the product.
4. Synthetic generation. Use a strong model to draft questions from your documents, then review them by hand. Fast, but synthetic data tends to be easier and more uniform than real traffic: the model copies the vocabulary of the source passage, which real users do not.
5. Adversarial cases. Edge cases, ambiguous questions, prompt-injection attempts, off-topic requests, other languages, extreme lengths.
6. Counter-examples. Inputs where the right behaviour is to refuse, to say "I do not know", or to leave things unchanged.

If you have no traffic yet, start with 20 questions from domain experts plus 20 adversarial cases, and replace them with real ones as soon as you can.

#What each example contains

{"id": "refund-007",
 "input": "I bought this 45 days ago, can I still return it?",
 "expected": "No. Returns are accepted within 30 days. Offer store credit review.",
 "must_include": ["30 days"],
 "must_not_include": ["guarantee"],
 "tags": ["refunds", "policy", "english"],
 "difficulty": "medium",
 "source": "ticket-18422",
 "added": "2026-10-04",
 "notes": "Policy v3, effective 2026-09-01"}

Different tasks use different grading information: an exact answer, required and forbidden strings, a reference answer, a rubric, a schema, or retrieved documents that should be cited. Store the reason an answer is expected (the policy or source), because that is what you need when someone disputes the label.

#How many examples?

More is better, but quality beats quantity. As a guide:

• 30 to 50: a smoke test that catches big regressions
• 100 to 300: useful for comparing versions and reporting to stakeholders
• 500+: needed to detect small differences or to slice results by category

Why those numbers? For a pass rate near 80 percent, the 95 percent margin of error is about ±11 points at 50 examples, ±8 at 100, ±4 at 400 (metrics and statistics). If you want to see a 3-point improvement, you need either several hundred cases or paired comparison on the same cases. Decide how small a difference matters to you, then size the set to detect it.

Slices need their own samples. If you want per-language results with ±8 points, that is about 100 cases per language.

#Cover the space

Tag every example by category, difficulty, language and format. Then check coverage: do you have enough examples of each task type and of the rare-but-costly cases? Report scores per tag as well as overall, since an average can hide a failing segment (the earlier story: refunds improved while shipping and accounts got worse).

A simple coverage table:

Notice the deliberate over-sampling of costly or risky categories: a golden set is not a mirror of traffic, it is a tool for catching what matters.

#Labelling well

• Write clear grading guidelines so two people would label the same way, with examples of borderline cases.
• Have two reviewers on a subset and measure agreement; disagreement means the guideline or the task is ambiguous.
• Record why an answer is expected, not only what it is.
• Prefer checkable criteria (contains X, valid JSON, cites a source) to subjective ones where possible.
• Label in batches with a review step, and fix systematic confusions in the guideline, not case by case.

#Measuring agreement: a worked kappa

Raw agreement overstates quality when one label dominates. Two reviewers label 100 items pass or fail:

Observed agreement po = (80 + 10) / 100 = 0.90. Each reviewer passes 85 percent, so by chance they would agree pe = 0.85 × 0.85 + 0.15 × 0.15 = 0.745. Cohen's kappa is (0.90 − 0.745) / (1 − 0.745) ≈ 0.61: substantial but not excellent agreement. If kappa is below about 0.6, improve the guideline before labelling more. The same measure tells you whether an LLM judge is trustworthy (LLM-as-judge).

#Splits

• Dev set (about 70 to 80 percent): you look at failures and tune prompts against it.
• Held-out test set (about 20 to 30 percent): used rarely, to confirm real improvement. If you iterate against it, it stops being a fair test.

Draw the split randomly within each tag, so both sets cover the same categories, and keep near-duplicates on the same side (otherwise the held-out set leaks). When the held-out score drops well below the dev score, you have overfitted your prompt to the dev set.

#Avoid duplicates and leakage

• Deduplicate near-identical inputs, or one scenario will dominate the score.
• Check that examples do not appear in your prompt (few-shot examples must not be in the test set), your fine-tuning data or your retrieval index. If they do, scores are inflated (benchmarks and contamination).
• Do not let the model that generates synthetic questions also be the model you evaluate without care: it may find its own questions easy.

#Synthetic data, done carefully

A workable recipe for RAG questions:

1. For each chunk, ask a strong model to write 2 to 3 questions answerable from that chunk, in the style of a real user, without reusing the passage's exact phrases.
2. Have a human review at least a sample, discard bad ones and fix labels.
3. Rewrite a share by hand using synonyms and colloquial phrasing.
4. Mix with real questions, and report scores separately for real and synthetic subsets; if synthetic is far easier, trust the real subset.

#Keep it alive

1. Add new failures weekly. Every confirmed production failure becomes a case.
2. Remove or fix examples whose correct answer has changed (policies, prices, facts).
3. Version the dataset and record which version produced each result, so comparisons are fair.
4. Watch for leakage: if examples (or near-copies) appear in your prompts, fine-tuning data or retrieval index, scores will be inflated.
5. Prune obsolete or flaky cases, and re-validate labels periodically.
6. Review the tail: read the cases your system fails every time; are they wrong labels or real gaps?

#Storage and versioning

A simple layout that scales:

evals/
  data/
    golden-v3.jsonl          # dataset, versioned in git
    DATASET.md               # sources, labelling guideline, split method, change log
  runs/
    2026-11-03-baseline.json # per-case results with config and dataset version

Keep the dataset in the repository next to the code, review changes in pull requests like code, and record the dataset version in every results file (regression testing).

#Privacy

Production-derived data is personal data. Anonymise it, restrict access and apply retention limits (PII, compliance). Prefer scrubbed or synthetic equivalents for datasets shared across teams or checked into repositories, and do not send real customer data to external judge or annotation services without a basis.

#Common mistakes

1. Only easy cases, because they were the ones you thought of.
2. A dataset that mirrors traffic exactly, with too few rare-but-costly cases.
3. Unclear labels, so noise hides real differences.
4. Tuning against the held-out set.
5. Never updating, so the set measures last year's product.
6. Synthetic-only data that makes everything look great.
7. No tags, so you cannot see which segment broke.

#Exercise

You are building a golden set for an invoice-extraction assistant. You have 500 real invoices processed last month.

1. How would you sample, and how many cases would you start with?
2. What tags would you use?
3. How would you check label quality?

Answers:

1. Stratify by vendor, layout type, language and quality (clean scan, photo, handwritten notes), sample about 100 invoices proportionally, then add 20 to 30 deliberately hard cases (missing fields, multiple currencies, credit notes, rotated scans). Start with 100 to 150, hold out 20 to 30 percent.
2. Vendor type, language, currency, layout, scan quality, field type being tested, difficulty.
3. Have two people label 30 of them independently, compute agreement and kappa per field, and rewrite the guideline wherever they disagree (dates, decimal separators, tax versus total).