Which quantisation should you pick? Q4_K_M, Q5, Q8 and the rest

Quantisation stores a model's weights with fewer bits. That shrinks the file, lowers the memory needed, and, because generation is limited by how fast weights can be read, usually makes it faster. The price is some loss of quality. The question is how much, and where to stop.

#The levels

These are the common llama.cpp GGUF types, with the average bits per weight including block scales:

llama.cpp's own quantisation table reports how much each type raises perplexity (lower is better, 0 is the original) on a Llama 2 7B model: about 0.0004 for Q8_0, 0.004 for Q6_K, 0.014 for Q5_K_M, 0.054 for Q4_K_M, 0.24 for Q3_K_M and 0.87 for Q2_K. The step from 8 bits down to 4 costs very little. Below 4 bits the loss grows fast.

#What it does to memory

For a 70B model (Llama 3.1 70B) at an 8,192-token context, the total memory is:

Quantisation is the difference between needing a server and fitting on a high-end Mac or two consumer GPUs (can you run a 70B model at home?).

#A simple decision rule

1. Pick the model size you want first, then the quantisation that fits.
2. Start with Q4_K_M. It is the default for good reason.
3. If you have spare memory, move up to Q5_K_M or Q6_K before moving to a bigger model, unless the bigger model also fits at Q4.
4. If you do not have enough, drop the model size before dropping below 4 bits. An 8B model at Q4 generally beats a 70B model forced into Q2, and runs much faster.
5. For code, maths and structured output, stay at 5 bits or above if you can. These tasks show quantisation damage sooner than casual chat.

#Quantisation is not the only kind

• The KV cache can be quantised separately to 8 bits to roughly halve cache memory, which matters for long contexts (KV-cache quantisation).
• AWQ, GPTQ and EXL2 are other 4-bit formats mostly used on GPUs with servers such as vLLM, as opposed to GGUF for llama.cpp and Ollama (comparison).

#Check, do not assume

Published numbers describe other models and other test sets. If it matters, run your own prompts through two quantisations and compare. The lesson GGUF and K-quants explains how block scales work, and the quantisation explorer lets you see the size and quality trade-off interactively.