Measuring speed correctly

"It runs at 50 tokens/second" is meaningless without what model, which quant, how much context, which phase and which hardware. Speed claims in forum posts and vendor charts are usually missing at least three of those. Measuring correctly takes ten minutes and saves you from picking hardware, models and settings on bad numbers.

#Metrics

TTFT is roughly queue wait + prompt_tokens / pp. Response time is roughly TTFT + output_tokens / tg (how generation works).

#A repeatable method

llama-bench -m model.gguf -ngl 99 -p 512,4096 -n 128 -r 5

1. Same model file, quant and flags across comparisons; change one thing at a time.
2. Warm up (discard the first run), then take the median of 5 or more runs and note the spread.
3. Report context depth. Decode slows as the cache grows; benchmark at the context you actually use (-p for prompt size, and a depth option in newer builds).
4. Fix the environment: AC power, performance mode, no background GPU load, thermals settled (let the machine cool between long runs).
5. Record versions: runtime commit or build, driver, OS and model file hash. Speed changes between releases.
6. Save raw results (CSV or JSON) alongside the command line, so you can reproduce and compare later.

Example output (illustrative):

| model          | size    | backend | ngl |  test |      t/s |
| llama 8B Q4_K_M | 4.58 GiB | CUDA   | 99 | pp512 | 3120 ± 40 |
| llama 8B Q4_K_M | 4.58 GiB | CUDA   | 99 | pp4096| 2650 ± 35 |
| llama 8B Q4_K_M | 4.58 GiB | CUDA   | 99 | tg128 |  94.1 ± 0.6 |

Read it carefully: prompt processing is about 30 times faster than generation, which is normal; it also drops somewhat with a longer prompt.

#Measure what users see: a client-side script

llama-bench measures the engine. To measure a running server end to end, time a streamed request:

import time, json, httpx

def measure(url, prompt, max_tokens=256):
    t0 = time.perf_counter(); first = None; chunks = 0
    with httpx.stream("POST", url, timeout=300, json={
        "model": "local", "stream": True, "max_tokens": max_tokens, "temperature": 0,
        "messages": [{"role": "user", "content": prompt}]}) as r:
        for line in r.iter_lines():
            if line.startswith("data: ") and line != "data: [DONE]":
                if first is None: first = time.perf_counter() - t0
                chunks += 1
    total = time.perf_counter() - t0
    return {"ttft_s": first, "tokens": chunks,
            "tg_tok_s": chunks / (total - first) if total > first else None}

runs = [measure("http://localhost:8080/v1/chat/completions", "Explain RAID levels.") for _ in range(6)]
runs = runs[1:]                                   # discard the warm-up run
for k in ("ttft_s", "tg_tok_s"):
    vals = sorted(r[k] for r in runs)
    print(k, "median", vals[len(vals)//2], "min", vals[0], "max", vals[-1])

Counting streamed chunks as tokens is an approximation (most servers send one token per chunk). For exact counts use the usage field or tokenize the output. What matters is applying the same method every time. For multiple simultaneous users, use the load-testing approach instead.

#Traps

• Comparing different quantizations as if they were the same model.
• Reporting prefill and decode as one number.
• Benchmarking an empty context and shipping a 30K-token workload.
• Measuring with streaming disabled and a client that adds latency.
• Cherry-picking the best of 10 runs.
• Cold runs: the first run includes loading and kernel compilation.
• Thermal throttling: a laptop that is fast for 20 seconds and slow after two minutes. Measure sustained speed.
• Mixed workloads: averaging a short and a long prompt hides both.
• Prompt cache hits: repeating the identical prompt makes the second run unrealistically fast (prompt caching).
• Different sampling: speculative decoding, grammar constraints or high temperature change speed.
• Comparing across machines with different power limits, drivers or memory configurations.
• Averaging tokens/s across runs (a mean of ratios): compute total tokens divided by total time instead, or use the median.

#Reporting results

A result is only useful if someone else can interpret it:

Hardware:   RTX-class GPU, 24 GB, 1,008 GB/s; driver X; Linux
Software:   llama.cpp build bNNNN, CUDA backend, flash attention on
Model:      Llama 3.1 8B Instruct, Q4_K_M (4.58 GiB), sha256 ...
Settings:   -ngl 99 -c 8192 -b 2048 -ub 512, KV f16
Workload:   pp512 / pp4096 / tg128, 5 runs, median ± sd
Results:    pp512 3120 ± 40 t/s; tg128 94.1 ± 0.6 t/s; peak VRAM 6.3 GiB
Check:      ceiling 1008 / 4.9 = 205 t/s; efficiency 46%  -> investigate

#Validate against theory

Estimate the ceiling with the bandwidth formula: tg ≤ bandwidth / model size. If you are far below it (efficiency under 60 percent), fix the setup before tuning flags: check that all layers are on the GPU, the right backend is in use, the machine is not throttling and nothing else is using the device. The benchmark lab walks through this diagnosis, with sweeps over flash attention, offload and context depth.

#Choosing what to measure for your use case

#Common mistakes

1. Quoting one number without context.
2. Trusting a single run.
3. Forgetting the KV cache in long-context speed.
4. Benchmarking on battery or in a power-saving mode.
5. Not writing down versions.
6. Comparing your measurement with a vendor's best case.

#Exercise

You measure a 14B Q4_K_M model (8.9 GB) on a 273 GB/s device: pp512 = 410 t/s, tg128 = 21 t/s, with a short context.

1. What is the efficiency?
2. What would you expect tg to be at 16K context, and why?
3. What is the time for a 3,000-token prompt and a 200-token answer?

Answers:

1. Ceiling 273 / 8.9 ≈ 30.7; 21 / 30.7 ≈ 68%: healthy.
2. Lower: the KV cache must be read at every step. With about 192 KiB per token, 16K tokens is 3 GiB (about 3.2 GB), so per-token reads rise from 8.9 GB to about 12.1 GB, a drop to roughly 15 to 16 t/s, before any attention overhead.
3. Prefill 3000 / 410 ≈ 7.3 s, decode 200 / 21 ≈ 9.5 s: about 17 seconds in total, with the first token after about 7 seconds. Prompt caching would remove most of the first number on repeat calls.