Prefill and time to first token: tuning the part before the first word
Why long prompts feel slow, what sets time to first token, and the techniques that shrink it: batch sizes, flash attention, prefix reuse, shorter prompts, chunking, and where to spend compute versus memory.
13 min·Verified 2026-11Pro+50 XP
Share
When you press enter, nothing visible happens until the whole prompt has been processed. For a short question that is a blink. For a 20,000-token document it can be many seconds. This delay is prefill, and it is tuned with different tools from the ones that speed up generation.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.