Performance Tuning
GPU offload, batching, flash attention, speculative decoding and prompt caching.
0 of 5 complete
- ▶ Up next12 min+50 XPGPU offload, threads and batch sizesSplit a model between GPU and CPU with -ngl, see with arithmetic why partial offload is so costly, tune threads…
- Pro20 min+50 XPFlash attention and long-context performanceWhy attention costs grow with context, how flash attention avoids the memory blow-up, how GQA, sliding windows and…
- Pro22 min+50 XPSpeculative decodingUse a small draft model to propose tokens and a big model to verify them in one pass: why it is lossless, the math…
- Pro20 min+50 XPPrompt and prefix cachingReuse the KV cache across requests so long system prompts, tool definitions and documents are processed once: how…
- LabPro40 min+100 XPLab: benchmark your rig and compare with theoryRun llama-bench properly, compare your measurements with the bandwidth model, sweep offload, flash attention, context…