Quantised matrix multiplication: INT8 and 4-bit kernels
Implement per-row INT8 and grouped 4-bit weight quantisation, measure the error against float, understand dequantise-on-the-fly dot products and block formats, and see in a C benchmark why 4x fewer bytes does not automatically mean 4x faster.
22 min·Verified 2026-11Pro+50 XP
Decode reads every weight for every token, so storing weights in fewer bits means moving fewer bytes (roofline). This lesson shows how a quantised matrix-vector product works and what it costs in accuracy and engineering.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.