How generation works: prefill, decode and the KV cache

LLMs are autoregressive: they produce one token at a time, each conditioned on everything before it. That simple fact splits inference into two very different phases, and understanding them tells you why some hardware choices matter and others do not.

#A quick tour of the model

A transformer language model is a stack of identical layers. Each layer has two main parts:

• Attention, which lets each token look at earlier tokens and gather the information it needs.
• A feed-forward network (MLP), which transforms each token's representation.

Tokens go in as numbers (embeddings), pass through dozens of layers, and come out as a logit (score) for every token in the vocabulary. Sampling turns those scores into one chosen token (sampling). The model's "parameters" (the weights) are the matrices inside these layers: an 8B model has about eight billion of them.

#Phase 1: prefill (reading your prompt)

All prompt tokens are processed in parallel in big matrix multiplications. The GPU has lots of independent work, so its arithmetic units stay busy: prefill is compute-bound. It determines time to first token (TTFT).

A useful estimate: each token costs about 2 × parameters floating-point operations.

8B model:       2 × 8e9           = 16 GFLOP per token
4,000-token prompt: 4,000 × 16 GFLOP = 64 TFLOP
GPU sustaining 40 TFLOPS:           64 / 40 ≈ 1.6 seconds

So a 4,000-token prompt takes about 1.6 seconds on such a GPU, and perhaps 20 seconds or more on a laptop CPU that sustains a fraction of a TFLOPS. Prefill speed (prompt processing, "pp" tokens per second) scales with compute, and it scales with prompt length: double the prompt, double the wait. This is why long system prompts and big retrieved documents hurt responsiveness, and why prompt caching is valuable.

#Phase 2: decode (writing the answer)

Now the model emits one token, appends it, and repeats. Each step must read all active weights from memory to produce a single token, and there is only one token's worth of arithmetic to do with them. The arithmetic units mostly wait for data: decode is memory-bandwidth-bound.

for each new token:
    logits = model(last_token, kv_cache)   # reads all weights + the KV cache
    token  = sample(logits)                # temperature, top-p, ...
    kv_cache.append(new keys/values)

That gives a simple, powerful rule:

decode speed (tokens/s)  ≈  memory bandwidth / bytes read per token
                         ≈  bandwidth / model size (plus a little for the KV cache)

Worked examples (the model is 8B at 4-bit, about 4.9 GB):

Real speeds reach about 60 to 85 percent of the ceiling. The key point: a faster chip with the same memory bandwidth decodes at the same speed, and a model twice as large decodes at half the speed. See memory bandwidth for how to look up your own numbers.

#The KV cache

Attention needs the keys and values of every previous token. Recomputing them at every step would repeat the same work over and over. Instead, the runtime stores them: the KV cache. It makes decoding fast, but it:

• grows linearly with context length (a few hundred KiB per token for large models),
• multiplies with the number of concurrent sequences, and
• must be read at every decode step, so long contexts slow decoding.

Without it, generating a 500-token answer after a 2,000-token prompt would redo the attention work for thousands of earlier tokens on every step. With it, each step only computes the new token's keys and values and reads the stored ones. The memory arithmetic is in tokens and context and VRAM math.

#What the metrics mean

A user-perceived response time is roughly TTFT + output_tokens / tg. For a 4,000-token prompt and a 300-token answer on a GPU doing 40 TFLOPS and 90 tokens/s: 1.6 s + 300/90 ≈ 1.6 + 3.3 ≈ 4.9 seconds.

Try the inference visualiser to watch both phases and the KV cache grow.

#Why one user and many users behave differently

At one user, decode wastes most of the GPU's arithmetic capacity: it reads the whole model to produce one token. If several sequences share each read (batching), the cost of reading the weights is split, and total throughput rises almost linearly, until compute or KV memory runs out. That is why servers like vLLM can serve dozens of users from one GPU, while a single-user chat feels bound by bandwidth. The Pro lessons on GPU offload and batching and load testing go deeper.

#Putting it to use

• Choosing a model size: divide your memory bandwidth by the model's file size to predict speed before downloading anything.
• Slow first token, fast after? Your prompt is long: shorten it, cache the prefix, or add compute.
• Fast start, slow stream? You are bandwidth-bound: use a smaller or more heavily quantised model.
• Speed drops as the chat gets longer? The KV cache and attention cost are growing: trim history or quantise the cache.
• Offloading layers to CPU RAM tanks speed? Those layers now run at system-memory bandwidth (GPU offload).

Key insight: bigger weights per token means slower decode. That is why quantization (fewer bytes per weight) makes models both smaller and faster (quantization basics).

#Exercise

Your machine has 273 GB/s of memory bandwidth and sustains about 20 TFLOPS. You run a 14B model at 4-bit (about 8.5 GB) and send a 3,000-token prompt expecting a 200-token reply.

1. Estimate the decode ceiling and a realistic speed.
2. Estimate prefill time.
3. Estimate total response time.

Answers:

1. Ceiling 273 / 8.5 ≈ 32 tokens/s; realistic about 20 to 27 tokens/s.
2. Per-token cost 2 × 14e9 = 28 GFLOP; for 3,000 tokens 84 TFLOP; at 20 TFLOPS that is about 4.2 seconds (real hardware is often less efficient, so expect 4 to 7 seconds).
3. Prefill about 4.5 s plus decode 200 / 24 ≈ 8.3 s, so roughly 13 seconds, with the first token after about 4.5 seconds. Caching the prompt prefix would remove most of the first number on repeat requests.