Generation: sampling strategies and the KV cache
Turn probabilities into text: greedy, temperature, top-k and top-p with tested code; why generation is slow without a KV cache and how the cache changes cost from quadratic to linear per token; repetition, stop conditions and what engines add.
16 min·Verified 2026-112 animated diagramsPro+50 XP
The model outputs a probability for each possible next token. Decoding decides which token to take, appends it and repeats. Those choices change the personality of the output, and the mechanics of the repetition decide how fast it runs.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 2 animated diagrams, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.