KV-cache quantization
Shrink the cache that grows with every token of context: exact sizing for real models, q8_0 and q4_0 trade-offs, the decode-speed effect, per-runtime flags, and a procedure to test quality on your own prompts.
20 min·Verified 2026-111 animated diagramPro+50 XP
During generation the model keeps keys and values for every earlier token. At short context this is small. At long context, or with many users, the KV cache can be larger than the weights. KV-cache quantization stores those keys and values in 8 or 4 bits instead of 16, often at almost no cost in quality.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.