Quantization basics

Trained weights are usually stored as 16-bit floating-point numbers. Quantization stores them with fewer bits (8, 6, 5, 4, 3 or even 2) plus a small amount of metadata to reconstruct approximate values at run time. The result is a model that is two to four times smaller, decodes faster, and is a little less accurate. Understanding how that works tells you which quant to pick and why.

#The core idea

q  = round(w / scale)         # store a small integer
w' = q × scale                # reconstruct at inference time
error = w − w'

With 4 bits you have only 16 levels, so the choice of scale matters.

#A worked example

Take eight weights in one block: [0.12, -0.50, 0.31, 0.02, -0.27, 0.45, -0.08, 0.19]. Use signed 4-bit integers from −7 to 7 and set scale = max|w| / 7 = 0.50 / 7 = 0.0714.

Every error is at most half a step (0.036). The weights are approximately right, and each took 4 bits instead of 16.

#Now add one outlier

Replace 0.45 with an outlier of 3.0. Now scale = 3.0 / 7 = 0.429, and the small weights suffer:

• 0.12 → 0.28 → stored 0 → reconstructed 0.000 (error 0.12, 100 percent)
• 0.31 → 0.72 → stored 1 → 0.429 (error 0.12)
• 0.19 → 0.44 → stored 0 → 0.000

Most ordinary weights collapse to zero or one coarse level, because the step size grew sixfold to accommodate one large value. This is the central difficulty of quantization, and why large outliers in LLM weights and activations drive so much of the engineering. You can see it live in the quantization explorer.

#Block-wise scales

Instead of one scale for a whole tensor, formats split weights into blocks (for example 32) and store a scale per block, so an outlier only damages its own block.

The scales cost memory. For the simplest 4-bit format (Q4_0): 32 weights × 4 bits + one 16-bit scale = 144 bits per 32 weights = 4.5 bits per weight. For Q8_0: 32 × 8 + 16 = 272 bits per 32 = 8.5 bits per weight. That is why "4-bit" formats really cost 4.25 to 4.85 bits per weight, and why you should always use the effective number when sizing memory (VRAM math).

Smaller blocks track the local range better (higher quality) but spend more bits on scales. Newer schemes use two levels: groups of 32 inside super-blocks of 256, with the scales themselves quantised. That is the idea behind the K-quants (GGUF and K-quants).

#Why quality survives

Neural networks are redundant and noise-tolerant: many weights matter little, and small random perturbations rarely change the output much. Empirically:

• Down to about 5 bits, the change in perplexity is tiny.
• At 4 bits, it is small but measurable (often a percent or two of perplexity).
• At 3 bits and below, damage grows quickly, unless clever methods (importance matrices, codebooks, per-group tuning) are used.
• Larger models tolerate quantization better than small ones: a 70B at 4-bit is typically better than an 8B at 8-bit, while a 1B model at 4-bit can noticeably suffer.

#Where quality loss shows up first

Averages hide it. Aggressive quantization tends to hurt:

• Maths and multi-step reasoning, where small errors compound across steps.
• Code, which needs exact tokens and syntax.
• Long-context recall, finding a fact buried in a long prompt.
• Non-English languages and rare tokens, which have less "slack" in the weights.
• Tool calls and strict formats, where one wrong token breaks the output.
• Subtle instruction following.

Casual chat is the last to show damage, which is why a heavily quantised model can feel fine in a demo and fail in production. Always test on your tasks (measuring quality).

#Two benefits, one cost

• Smaller files mean bigger models fit in memory.
• Faster decode: fewer bytes read per token, and decode is bandwidth-bound (memory bandwidth). Q4 reads about half the bytes of Q8.
• Cost: some accuracy, especially on the tasks above. Dequantization also costs a little compute, which kernels make cheap on modern hardware.

Prefill (prompt processing) is compute-bound, so quantization helps it much less, and some formats can be slower there.

#Kinds of quantization you will meet

The GGUF family (llama.cpp, Ollama, LM Studio) is the easiest to start with. GPU-serving formats such as AWQ, GPTQ and FP8 matter when you run vLLM (AWQ, GPTQ and EXL2).

#Try it

Open the quantization explorer to drag the bit width and watch the error grow on real-looking weight distributions, with and without outliers.

#Quick guidance

#How to decide

1. Fit first. Find the largest quant that fits with headroom for your context (VRAM math).
2. Test on your own prompts at the quant you plan to use, comparing with a higher quant as the reference.
3. If the smaller quant ties on your tests, take it and spend the savings on context or a bigger model.
4. The choose-a-quant lab walks through measuring size, speed, KL divergence and task quality.

#Common mistakes

1. Using the headline bit width to size memory; add the scale overhead.
2. Assuming a lower quant of a small model is fine because it was fine for a large one.
3. Judging quality by chat feel rather than a task test.
4. Comparing quants of different models.
5. Downloading random re-quantizations without checking the source (supply chain).

#Exercise

A 14B model has 14.7 billion parameters. Compute its size at Q8_0 (8.5 bits per weight) and at Q4_K_M (4.85 bits per weight), and the decode ceiling of each on a 400 GB/s device.

Answers: Q8_0 is 14.7e9 × 8.5 / 8 = 15.6 GB, ceiling 400 / 15.6 ≈ 26 tok/s. Q4_K_M is 14.7e9 × 4.85 / 8 = 8.9 GB, ceiling 400 / 8.9 ≈ 45 tok/s. The 4-bit version is 1.75 times smaller and about 1.75 times faster, for a small quality cost.