Quantization explorer
Watch quantization error appear as you lower the bit width, then compare real GGUF quants by size and quality.
Block-wise quantization demo
A slice of synthetic weights (blue) and their reconstruction (orange) after rounding to a few levels per block. Outliers stretch a block's scale and hurt everything else in it.
Levels
16
Signal-to-noise
16.2 dB
Effective bits/weight
4.50
Effective bits include one 16-bit scale per block. Real GGUF K-quants use more elaborate super-blocks, so numbers differ.
GGUF quant comparison
File size and relative quality for a chosen model.
| Quant | Size | Relative size | Perplexity delta* | Note |
|---|---|---|---|---|
| F16 | 15.0 GiB | +0 | Unquantized reference | |
| Q8_0 | 7.9 GiB | +0.0004 | Near-lossless | |
| Q6_K | 6.1 GiB | +0.0044 | Very close to F16 | |
| Q5_K_M | 5.3 GiB | +0.0142 | Great quality/size | |
| Q4_K_M | 4.5 GiB | +0.0535 | Best default for most | |
| IQ4_XS | 4.0 GiB | n/a | Smaller 4-bit (imatrix) | |
| Q3_K_M | 3.7 GiB | +0.2437 | Noticeable loss | |
| Q2_K | 3.1 GiB | +0.8698 | Last resort; big loss |
*Perplexity increase over F16 on Llama-2 7B from llama.cpp's published quantization table; it is illustrative, and the effect differs by model and task.
Learn more