Speculative decoding
Use a small draft model to propose tokens and a big model to verify them in one pass: why it is lossless, the math behind the speedup, how to pick k and a draft model, and how to measure the real gain.
22 min·Verified 2026-111 animated diagramPro+50 XP
Decode is memory-bound: to produce one token the GPU streams every weight from memory, and its arithmetic units sit mostly idle. Speculative decoding spends that idle compute to produce several tokens per weight read. Done correctly it gives 1.5 to 3x faster generation with exactly the same output distribution, which is rare: most speed tricks cost quality.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.