Why vibes fail: the case for evaluation

You changed the prompt, switched to a smaller quantization or upgraded the model. Is the system better or worse? If your answer is "seems fine", you are running on vibes. Vibes are how most LLM projects start, and why so many get worse in ways nobody notices until users complain.

#A story in numbers

A team has a support assistant. A customer reports a wrong refund answer, so they tweak the prompt, retry that one question, and it is now correct. They ship it.

What they did not see:

The change fixed the case they looked at and three more, and quietly broke five others. The overall system is slightly worse, and they will learn it from support tickets in two weeks. A 60-case test set that takes a minute to run would have shown the drop immediately. This kind of trade-off is the rule, not the exception: prompts, models and quantizations all shift behaviour across the board.

#Why vibes fail

• Tiny samples. Five examples cannot distinguish a real change from luck. One flipped result moves the score by 20 points.
• Cherry-picking. You test the cases you remember, usually the easy or the recently failed ones.
• Recency bias. The last bad answer dominates your impression, and the last good one makes you feel finished.
• Non-determinism. Sampling makes the same prompt succeed and fail on different runs, so a single try proves little.
• Hidden regressions. Fixing one case quietly breaks others you did not re-check.
• Confirmation bias. You want the change to work, so you read ambiguous outputs charitably.
• Nobody else can reproduce it. Opinions do not transfer between teammates, and they do not survive staff turnover.
• Drift. Hosted models, data and user behaviour change underneath you with no deploy at all.

None of this means you are careless. It means that outputs from a statistical system need statistics.

#Five prompts is not evidence

Suppose version A truly succeeds on 70 percent of your real traffic and version B on 60 percent, so B is really worse. You try each on five random prompts. The chance that B scores at least as well as A is about 50 percent: a coin flip. And if A truly were better by 10 points, five prompts would usually fail to show it. To distinguish a 10-point difference you need on the order of a hundred or more examples, or paired comparisons (metrics and statistics).

#The pipeline that replaces vibes

1. Dataset: representative inputs, with expected answers or grading criteria (golden datasets).
2. Run: your whole system (prompt + model + retrieval + tools + settings) produces outputs.
3. Score: metrics, rules or a judge turn outputs into numbers (metrics, LLM-as-judge).
4. Report: aggregate scores with uncertainty, broken down by tag, and a list of failures to inspect.
5. Gate: compare to a baseline and block changes that get worse (regression testing in CI).

A skeleton makes it concrete:

import json

def evaluate(system, path="golden.jsonl"):
    cases = [json.loads(l) for l in open(path)]
    results = []
    for c in cases:
        output = system(c["input"])                      # your prompt + model + retrieval
        results.append({"id": c["id"], "tags": c["tags"], "pass": score(c, output), "output": output})
    rate = sum(r["pass"] for r in results) / len(results)
    print(f"pass rate {rate:.1%} on {len(results)} cases")
    for r in results:
        if not r["pass"]:
            print("FAIL", r["id"], r["output"][:100])
    return results

Everything in this course elaborates one of those lines.

#Offline and online evaluation

You need both. Offline catches regressions early; online finds what you did not think of and feeds new cases back into the offline set (online evaluation).

#Three kinds of question evals answer

1. Is this version better than that one? (model, prompt, quantization comparisons, choosing a model)
2. Is it good enough to ship? (a threshold such as "at least 90 percent of the golden set passes, and zero critical failures")
3. Did anything get worse? (regression detection)

Notice that none of these needs a perfect metric. They need a consistent one.

#Eval-driven development

The loop that works:

write failing cases  →  change prompt/model/retrieval  →  run evals  →  read failures  →  repeat
        ▲                                                                       │
        └──────────── add every new real-world failure as a case ◀──────────────┘

This is test-driven development adapted to statistical systems. You write down what "good" means before you tinker, you change one thing at a time, and you keep the cases that taught you something. Teams that work this way iterate faster, not slower, because they stop arguing about opinions and stop shipping regressions.

#Start small

You do not need a platform. A first useful evaluation is:

• 50 to 100 real or realistic examples in a JSONL file,
• a Python script that runs them and scores each pass or fail,
• a printed pass rate and a list of failures.

You will build exactly this in the lab. Everything else in this course makes that script more trustworthy and more useful.

#Objections you will hear

#What to evaluate

Not just "accuracy". Typical dimensions: correctness, faithfulness to sources, format validity, safety, refusal behaviour, tone, latency and cost. Choose a few that matter for your product and measure each separately. A change that raises accuracy two points but doubles latency may be a loss.

#Your first week

By Friday you will have a number you can defend and a process you can repeat.

#Common mistakes

1. Treating a handful of examples as evidence.
2. Never writing the examples down.
3. Changing several things at once, so you cannot tell what mattered.
4. Measuring only the average, hiding a collapsed segment.
5. Building an elaborate platform before a simple script.
6. Evaluating once, then never again.

#Exercise

You compare two prompts on 10 hand-picked questions: prompt A gets 8 right, prompt B gets 9. Your colleague says "B is better, ship it".

1. Why is that weak evidence?
2. What would you do?

Answers:

1. Ten hand-picked questions give very wide uncertainty: 8 of 10 and 9 of 10 are easily the same underlying rate, the difference is one question, and nothing shows whether B broke other cases.
2. Build a golden set of at least 50 to 100 real questions, run both prompts on the whole set (temperature 0), compare pass rates with intervals and the paired list of cases that changed, and decide on that, also checking per-tag results.