GGUF and the K-quant family

GGUF is llama.cpp's model format: one file containing the tensors, hyperparameters, the tokenizer and the chat template. It is memory-mappable, so loading is fast and the OS can share pages between processes. Ollama, LM Studio and llama.cpp all use it, and it is by far the most common way people run models locally.

#What is inside a GGUF

• Header and key-value metadata: architecture (llama, qwen2, ...), layer count, head counts, context length, rope settings.
• Tokenizer: the vocabulary and merge rules, so no separate tokenizer files are needed.
• Chat template: the Jinja template that formats messages for this model.
• Tensors: each stored at its own quantization type. A "Q4_K_M" file is actually a mix of types, with the most sensitive tensors at higher precision.

You can inspect a file with the gguf Python package (pip install gguf, then gguf-dump model.gguf) or from the model page on Hugging Face, which shows the metadata and tensor list.

#Decoding the names

Q4_K_M reads as: Quantized, 4-bit base, K-quant scheme (super-blocks of 256 weights with quantised scales), Medium (a balanced mix of precisions across tensors; Small uses less, Large more).

Older legacy formats (Q4_0, Q4_1, Q5_0, Q8_0) use simple blocks of 32 with one scale; Q8_0 is still used as a near-lossless option. K-quants are generally better than the legacy 4-bit and 5-bit types at the same size.

*Increase over F16 on Llama-2 7B as published in llama.cpp's quantization docs; the ordering is representative, the exact numbers vary per model and are smaller for larger models.

Reading the table: going from Q8_0 to Q5_K_M costs almost nothing in perplexity but saves about a third of the memory; Q4_K_M saves nearly half for a delta that is small compared with the gap to Q3. The curve bends sharply below 4 bits.

#Size of an 8B model by quant

#How K-quants work

K-quants quantise in two levels. Weights are grouped into blocks of 32, and 8 blocks form a super-block of 256. Each block has its own small scale (and for some types a minimum), and those block-level scales are themselves stored in few bits, with one higher-precision scale for the whole super-block. This gets accuracy close to using fine-grained scales at a lower overhead. The S/M/L variants additionally keep the most sensitive tensors (for example parts of attention and the feed-forward down-projection) at higher precision (quantization basics explains why scales matter).

#I-quants and imatrix

IQ4_XS, IQ3_M, IQ2_XS and friends use codebooks (vectors of weights mapped to learned patterns) and an importance matrix (imatrix): a calibration pass that records which weights matter most on representative text. At 3 bits and below, I-quants beat plain K-quants at the same size, but can be slower on CPU, and quality depends on the calibration data.

You can use an imatrix with any quant type, including K-quants, and it improves low-bit results:

llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.dat -ngl 99
llama-quantize --imatrix imatrix.dat model-f16.gguf model-IQ3_M.gguf IQ3_M

Pre-made imatrix files are often published alongside popular quants. If your use case is non-English or code-heavy, a calibration text in that domain matters.

#Choosing a file

1. Compute your memory budget: weights + KV cache + overhead (VRAM calculator).
2. Pick the largest quant that fits with 10 to 15 percent headroom.
3. Prefer Q4_K_M or Q5_K_M unless you have a reason not to; go higher for small models, tool use and code.
4. For 3 bits or fewer, choose an IQ quant with an imatrix.
5. Check whether you want a bigger model at a lower quant rather than a smaller model at a higher quant: usually yes, down to about 4 bits.
6. Download from the model's official or a well-known quantizer account, and read the model card (it lists the exact base model, chat template and any known issues).
7. Verify the file hash if provided (supply chain).

Worked fit: a 12 GB GPU, a 14B model (14.7B parameters), 8K context. Q4_K_M is about 8.3 GiB, KV about 1.5 GiB (at 192 KiB per token) and overhead about 1 GiB: total about 10.8 GiB, which fits. Q5_K_M (about 9.8 GiB) would bring the total to 12.3 GiB, which does not.

# llama.cpp can pull a quant straight from Hugging Face
llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -c 8192 -ngl 99

Large repositories split files into shards (model-00001-of-00003.gguf). Keep all shards in one folder and point -m at the first one.

#Converting and quantizing yourself

# 1) Convert Hugging Face weights to an F16 GGUF
python convert_hf_to_gguf.py ./hf-model --outfile model-f16.gguf

# 2) Quantize
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# 3) Sanity-check size, then quality
ls -lh model-Q4_K_M.gguf
llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw -ngl 99

Make sure you start from the right base weights and the right chat template (carried in the tokenizer config). A conversion from the wrong template gives a model that runs but answers badly. For quality comparisons use KL divergence and a task test rather than perplexity alone.

#Troubleshooting

#Common mistakes

1. Choosing Q8_0 "to be safe" on a machine where Q5 or Q6 would let you use a larger model.
2. Using Q2_K for anything important.
3. Mixing up the base model and the instruct model (a base model will not follow chat instructions).
4. Ignoring the chat template when converting.
5. Comparing quants from different quantizers as if they were identical; calibration and tensor choices differ.

#Exercise

You have 24 GiB of VRAM (about 23 usable), want a 32B model (32.8B parameters), 16K context, one user, KV 256 KiB per token. Which of Q4_K_M, Q5_K_M and Q6_K fit?

Answers: Q4_K_M: 32.8e9 × 4.85 / 8 = 19.9 GB ≈ 18.5 GiB, KV 4 GiB, overhead about 1.2: 23.7 GiB, does not fit as is; with an 8-bit KV cache (2 GiB) it totals 21.7 GiB and fits. Q5_K_M (about 21.7 GiB of weights) and Q6_K (about 25 GiB) do not fit at all. So Q4_K_M with a quantised cache, or a shorter context, is the answer.