Lab: choose a quant for your hardware
Measure size, memory, speed, KL divergence and task quality for three quants of the same model, then make a defensible, data-driven pick with a decision rule and a written result.
45 min·Verified 2026-11Hands-on labPro+100 XP
Goal: pick between Q4_K_M, Q5_K_M and Q6_K for an 8B model on your machine and your tasks, using numbers instead of folklore, and write down the result so the decision is repeatable when a new model arrives.
Time: about 45 minutes, most of it waiting for downloads and perplexity runs. Prerequisites: llama.cpp built or installed (guide), about 30 GB of free disk, and the background from GGUF and K-quants and measuring quality.
Pro content
Keep reading with Inference Academy Pro
Unlock this full lab, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.