Kernel fusion, launch overhead and kernel selection: where small-batch decode time really goes
Why fusing operations cuts memory traffic, why hundreds of tiny kernel launches can dominate a fast decode step, when CUDA graphs help and when they do nothing, and how tile and wave quantisation make the same matrix multiply fast or slow depending on its shape. Numbers come from small models you can run.
You can have the right model, the right quantisation and a fast GPU and still be slower than you should be, because the time is lost between operations rather than inside them. This lesson covers three things the software stack overview only mentioned: fusion, launch overhead and kernel selection.
All numbers below come from small calculations you can run. They are models of the mechanism, not measurements of any particular GPU, so measure your own system before you act (benchmarking).
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 2 animated diagrams, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.