Why memory bandwidth, not compute, decides how fast an LLM runs

If you compare GPUs for local LLMs by their TFLOPS, you will pick wrong. For one person chatting with a model, the limit is not arithmetic. It is how quickly the hardware can read the model's weights from memory.

#Why

To produce one token, the model multiplies a vector by every weight matrix, which means reading every active weight once. A 4.5 GiB model therefore moves about 4.5 GiB through memory for each token. The multiplication itself is a tiny amount of work compared with the reading. The chip spends most of its time waiting for data.

That gives a simple ceiling:

Real runs reach about 65 percent of the theoretical figure, so that is the efficiency we use for estimates. The KV cache adds to the bytes read, which is why speed drops slowly as a conversation gets longer.

#Worked example

Llama 3.1 8B at Q4_K_M reads about 4.9 GB of weights per token. On a card with 1,008 GB/s of bandwidth (an RTX 4090), the ceiling is about 205 tokens per second, and at 65 percent efficiency with an 8K cache the estimate is about 110.

The same model across hardware, estimated this way:

Notice that the RTX 4060 Ti 16GB is slower than the older RTX 3060 12GB for this job: it has more memory but a narrower bus. Notice too that an Apple M4 Pro and an RTX 4060 Ti land in the same place, because their bandwidths are alike. These are estimates, so measure your own machine (the benchmarking lab).

#What follows from this

• Smaller weights are faster. Halving the bits per weight roughly doubles speed, which is a second reason to quantise (quantisation guide).
• Mixture-of-experts models are fast for their size. Mixtral 8x7B has 46.7 billion parameters but reads only about 12.9 billion per token, so it decodes at roughly the speed of a 13B model while needing memory for all 46.7 billion.
• More GPU cores will not help a memory-bound run. A faster GPU on the same memory speed barely changes single-user chat.
• Batching changes the picture. When a server handles many requests at once, one read of the weights serves all of them, so the limit moves toward compute. That is why serving engines care about throughput and chat apps care about bandwidth.
• Prefill is different. Reading a long prompt is compute-bound, so time to first token depends on compute, not bandwidth (prefill and time to first token).

#How to use this

When buying hardware for local inference, compare memory capacity first (does the model fit?) and memory bandwidth second (how fast will it run?). TFLOPS come a distant third. The lesson Memory bandwidth goes deeper, and the hardware checker estimates speed for your own machine.