Prefill, decode and the roofline: predicting inference speed

If you can only learn one idea in inference engineering, learn this one: generating a token is limited by how fast you can move bytes, not by how fast you can multiply. With it you can predict tokens per second on any hardware, understand why batching, quantisation and speculative decoding work, and know which optimisations will not help.

#Two phases, two different bottlenecks

An LLM request has two phases:

• Prefill: process the whole prompt (say 2,000 tokens) in one parallel pass, producing the first output token and the KV cache.
• Decode: generate one token at a time. Each step takes the previous token, reads the KV cache and the weights, and produces the next token.

They stress hardware differently. Prefill is a large matrix-matrix multiplication; decode is a matrix-vector multiplication (at batch 1). That single difference explains most of the field.

#FLOPs and bytes of one decode step

For a dense model with P parameters:

• FLOPs per token ≈ 2P (one multiply and one add per weight).
• Bytes read per step ≈ P × bytes_per_param (every weight once) plus KV cache reads.

For an 8B model in BF16 (2 bytes/param): 16 GFLOPs of work and 16 GB of weights to read.

Arithmetic intensity is FLOPs per byte moved:

intensity = 2P / (P × bytes_per_param) = 2 / bytes_per_param   (at batch size 1)

For BF16 that is 1 FLOP/byte; for INT4 (0.5 bytes) it is 4 FLOP/byte.

#The roofline model

A GPU has a peak compute rate F (FLOP/s) and a peak memory bandwidth B (bytes/s). The best achievable performance for a kernel with intensity I is:

performance ≤ min(F, I × B)

The ridge point F / B is the intensity where the two limits meet. Left of it you are memory-bound; right of it you are compute-bound.

(Check current datasheets; these are rounded and vary by SKU. Note that marketing figures often quote "with sparsity", which doubles tensor-core TFLOPs; use the dense number.)

Decode at batch 1 has intensity ≈ 1, hundreds of times below the ridge, so it is deeply memory-bound: the compute units are idle almost all the time.

#Worked example: how fast can an 8B model decode?

Step time is bounded below by the larger of the memory time and the compute time:

t_mem  = weight_bytes / B  = 16e9 / 3.35e12 = 4.78 ms     (H100)
t_comp = 2P × batch / F    = 16e9 × 1 / 989e12 = 0.016 ms
step   = max(t_mem, t_comp) = 4.78 ms  →  ≤ 209 tokens/s for one stream

Same model, other GPUs (batch 1): A100 8.0 ms → 125 tok/s; RTX 4090 15.9 ms → 63 tok/s. These are ceilings. Real engines achieve perhaps 50 to 85% of the ceiling depending on kernels, KV reads, sampling and scheduling overheads. If you measure 20% of the ceiling, something is wrong (offloading, CPU bottleneck, bad kernels).

This is why the first-order rule for local inference is: tokens/s ≈ memory bandwidth ÷ model size in bytes. It is also why quantisation speeds decoding almost proportionally to size reduction (quantisation in depth).

#What batching does

Serve b sequences at once. Weights are read once per step regardless of b, but FLOPs scale with b:

step = max( weight_bytes / B , 2P·b / F )

Computed for the 8B BF16 model:

Throughput scaled ~256× from batch 1 to 256 at constant step time. That is the economic engine of LLM serving: batching converts idle compute into tokens. The crossover (batch ≈ ridge point ≈ 295 for BF16 on H100, because intensity ≈ batch) is where adding sequences stops being free. On an A100 it arrives near batch 156.

Two caveats the simple model hides:

1. KV cache traffic. Each sequence reads its own KV cache every step. For Llama-3.1-8B (32 layers, 8 KV heads, head dim 128, BF16) that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token; an 8k context is ~1.07 GB per sequence. At batch 64 with 8k contexts the KV reads (≈69 GB per step) dwarf the weights (16 GB). Long contexts make decode KV-bandwidth-bound, which is why GQA, KV quantisation and paged attention matter (attention optimisations).
2. Memory capacity. Batch 256 × 8k context needs ~275 GB of KV cache, more than one GPU holds. Capacity, not bandwidth, caps the batch.

#Prefill: the compute-bound phase

Prefilling N tokens costs about 2PN FLOPs while reading the weights once, so intensity ≈ N. With N = 2,000 you are far right of the ridge: compute-bound.

t_prefill ≈ 2 × 8e9 × 2000 / 989e12 ≈ 32 ms   (H100 ideal; real is 1.5–3× slower)

So a 2,000-token prompt costs roughly the compute of ~6 decode steps' worth of time but 2,000 tokens' worth of work. Consequences:

• TTFT (time to first token) is dominated by prefill plus queueing.
• Long prompts hurt other users when prefill and decode share a GPU: a big prefill stalls everyone's decode. This motivates chunked prefill and disaggregated serving (disaggregation).
• Prefix caching skips prefill for repeated prefixes (caching).

#Run the numbers yourself

GPUS = {"A100-80GB": (2.0e12, 312e12), "H100-SXM": (3.35e12, 989e12)}

def step_time(params, bytes_per_param, batch, bw, flops, kv_bytes_per_seq=0):
    mem = (params * bytes_per_param + batch * kv_bytes_per_seq) / bw
    comp = 2 * params * batch / flops
    return max(mem, comp), mem, comp

for name, (bw, fl) in GPUS.items():
    for b in (1, 64, 512):
        t, m, c = step_time(8e9, 2, b, bw, fl)
        print(name, b, f"{t*1e3:.2f} ms", f"{b/t:,.0f} tok/s", "memory" if m >= c else "compute")

Try changing bytes_per_param to 1 (FP8/INT8) and 0.5 (INT4) and watch the ceiling rise.

#What this tells you to do

#Where the simple model breaks

• MoE models: only a fraction of experts is active per token, so FLOPs per token are lower than 2 × total_params, but with batching different tokens hit different experts so more weights are read; the memory story is subtler (parallelism).
• Kernel efficiency: real kernels reach 60 to 90% of peak bandwidth; small matrices and launch overheads cost more.
• Multi-GPU: communication adds time (all-reduce per layer).
• Sampling and CPU work: tokenisation, scheduling and detokenisation can bottleneck tiny models.
• Quantisation dequantisation overhead: low-bit weights save bytes but add compute to unpack.

#Common mistakes

1. Quoting TFLOPs to predict decode speed.
2. Using the "with sparsity" FLOPs figure.
3. Ignoring KV cache bytes at long context.
4. Expecting linear speedups from batching forever; the ridge and memory capacity end it.
5. Benchmarking at batch 1 and extrapolating to a loaded server.

#Exercise

A 70B model in INT4 (0.5 bytes/param, 35 GB of weights) runs on a GPU with 3.35 TB/s bandwidth and enough memory.

1. Estimate the single-stream decode ceiling.
2. The same model in BF16 needs 140 GB. On two such GPUs with perfect tensor parallelism (aggregate bandwidth 6.7 TB/s), what is the ceiling?
3. At what batch size does the INT4 case's compute time equal its memory time (use compute time = 2P·b/F)?

Answers:

1. 35e9 / 3.35e12 = 10.4 ms → ≈ 96 tokens/s.
2. 140e9 / 6.7e12 = 20.9 ms → ≈ 48 tokens/s (before communication overhead), so INT4 on one GPU beats BF16 on two for single-stream speed.
3. Memory time 10.4 ms equals compute time when 2 × 70e9 × b / 989e12 = 10.4e-3 → b ≈ 73. Beyond about 73 sequences the INT4 step becomes compute-bound (ignoring dequantisation cost).

#Going deeper

• Compute the ceiling for your own GPU and model, then measure with llama-bench or your engine and compare (benchmarking).
• Read the next lesson on attention optimisations.