Making It Fast
Quantised matrix multiplication, CPU speed techniques, continuous batching and a benchmark against llama.cpp.
0 of 4 complete
- ▶ Up nextPro22 min+50 XPQuantised matrix multiplication: INT8 and 4-bit kernelsImplement per-row INT8 and grouped 4-bit weight quantisation, measure the error against float, understand…
- Pro17 min+50 XPSpeed on the CPU: SIMD, threads, cache layout and profilingA mental model for making inference kernels fast: measure bandwidth first, vectorise with SIMD, split rows across…
- Pro20 min+50 XPContinuous batching and an OpenAI-style streaming serverWhy serving many requests together multiplies throughput, the difference between static and continuous batching (with…
- LabPro180 min+100 XPLab: run a real model with your engine and benchmark it against llama.cppCapstone for the engine course: make your engine load real weights, validate its logits against a reference…