Bottlenecks and Hardware

Prefill versus decode, the roofline model, attention optimisations and GPU architecture for inference.

1. 1▶ Up next18 min+50 XPPrefill, decode and the roofline: predicting inference speedWhy decode is memory-bound and prefill is compute-bound, how to compute the arithmetic intensity of a transformer…
2. 2Pro16 min+50 XPAttention optimisations: FlashAttention, GQA, paged KV and sliding windowsWhy attention is memory-bound, how FlashAttention tiles to avoid materialising the score matrix, how MQA/GQA and MLA…
3. 3Pro15 min+50 XPGPU architecture for inference: memory hierarchy, tensor cores and generationsWhat inside a GPU matters for LLM serving: SMs, tensor cores and precisions, the memory hierarchy from registers to…