Memory bandwidth: the speed limit of decoding

During decode, producing one token requires streaming essentially all active weights through the processor, while doing only a sliver of arithmetic with them. The chip spends most of its time waiting for memory. So:

tokens/sec  ≈  efficiency × memory_bandwidth / bytes_read_per_token

with an efficiency of roughly 0.5 to 0.8 in good runtimes. This one formula predicts local LLM speed better than any benchmark chart, and it tells you what to buy and how to tune.

#Where does the formula come from?

Imagine reading a 5 GB book aloud to a friend, one word at a time, but you must re-read the whole book before each word. Your reading speed (bandwidth), not your cleverness (compute), limits how many words per second you produce. A language model does the same: to emit the next token, every weight that participates must travel from memory to the arithmetic units. Fast arithmetic does not help if data cannot arrive fast enough.

#Worked examples (8B model at Q4_K_M, about 4.9 GB read per token)

These are estimates; real numbers depend on runtime, context length and quantization kernels. Measure yours in the benchmark lab.

#Finding your bandwidth

Two traps: single-channel RAM is surprisingly common in budget laptops and halves your CPU speed, and spec-sheet peak is not sustained, which is why the efficiency factor exists.

#Consequences

• Smaller quant, faster decode. Q4 reads about half the bytes of Q8, so it decodes roughly twice as fast. Quantization is a speed feature, not just a memory saver (quantization basics).
• Bandwidth beats raw compute for chat-style generation. A GPU with fast memory can outrun a "faster" one with slow memory. Compare bandwidth first, compute second.
• Bigger model, slower in proportion. A 70B at Q4 (about 43 GB) is nine times the bytes of an 8B (4.9 GB), so about one ninth the speed on the same hardware.
• Same bandwidth, same speed. A GPU twice as fast at arithmetic but with the same bandwidth decodes at the same rate. It will, however, process prompts faster.
• Long context adds KV-cache reads at every step, slowing decode as the conversation grows (tokens and context).
• Batching amortises weight reads across many requests, which is how servers get high aggregate throughput from the same bandwidth (batching).
• Offloading layers to CPU RAM forces those layers to stream at system-memory bandwidth, which can be 5 to 10 times lower than GPU memory.

#Mixture-of-experts: big in memory, light per token

MoE models (Mixtral, Qwen MoE variants and many recent models) contain many "expert" sub-networks, and a router picks a few for each token. All experts must be loaded, but only the chosen ones are read per token.

Example: Mixtral 8x7B has about 46.7B parameters in total but about 12.9B active per token. At 4-bit:

So Mixtral decodes at about the speed of a dense 13B while needing memory like a 47B model. That makes MoE a great fit when you have lots of memory (a high-memory Mac, or a GPU plus ample system RAM) and want speed. It is a poor fit when memory is tight.

#Beyond weights: KV cache reads

Per-token bytes are the weights plus the KV cache read at that step:

bytes per token ≈ model bytes + kv_bytes_per_token × current_context

On Llama 3.1 8B (128 KiB per token) at Q4 (4.9 GB), with 32K tokens of context the cache is 4 GiB, so each step reads about 9 GB instead of 4.9. Speed falls by nearly half, which matches what you observe in real long-context chats. Quantizing the KV cache or using attention optimisations helps (flash attention and context).

#Check your efficiency

Run llama-bench (benchmark lab) and compute:

efficiency = measured_tg / (bandwidth_GB/s / file_size_GB)

#Compute still matters for prompts

Bandwidth explains decode. Prefill (reading your prompt) is limited by compute, so GPUs with lots of arithmetic throughput process prompts much faster (how generation works). A hardware choice that is great for decode (high bandwidth, modest compute) may feel slow on long prompts. Know which one your use case stresses: chat with short prompts is bandwidth-bound; RAG with long prompts also needs compute.

#What to do with this

1. Predict before you download. Divide bandwidth by file size to know if a model will feel usable.
2. Pick the quant for speed as well as fit. If Q5 fits but Q4 gives you 25 percent more speed with equal task quality, take Q4.
3. Buy bandwidth. Compare GB/s per dollar when choosing a GPU or Mac (buying guide).
4. Fill memory channels on CPU machines.
5. Prefer MoE when you have memory and want speed.
6. Watch context length for speed-sensitive uses.

Compare devices side by side in the benchmarks matrix.

#Common mistakes

1. Comparing GPUs by TFLOPS for chat workloads.
2. Assuming two machines with the same chip behave the same, when memory configuration differs (Pro versus Max, single versus dual channel).
3. Expecting speed to scale with parameter count linearly in both directions: quantization changes bytes, not parameters.
4. Forgetting the KV cache when explaining a speed drop.
5. Quoting peak bandwidth as if it were sustained.

#Exercise

You have two options for the same price: a GPU with 24 GB at 600 GB/s, or one with 16 GB at 900 GB/s. You want to run a 32B model at Q4 (about 19.9 GB) and, separately, a 14B model at Q4 (about 8.9 GB).

1. Which options can run each model?
2. What are the ceilings?

Answers:

1. The 24 GB card can run both (the 32B fits with little spare for context). The 16 GB card can only run the 14B; the 32B does not fit.
2. 14B: 24 GB card 600 / 8.9 ≈ 67 tok/s; 16 GB card 900 / 8.9 ≈ 101 tok/s. 32B on the 24 GB card: 600 / 19.9 ≈ 30 tok/s. The 16 GB card is faster for what fits, the 24 GB card runs a class of models the other cannot. The right choice depends on which models you need: capacity first, then bandwidth.