Skip to content
⌕Go Pro
☰

Learn

CoursesCareer pathsAll lessonsGlossary

Tools

CalculatorsCan I run it?PlaygroundBenchmarksFixes
Pricing

Why inference is fast or slow, and how to make it faster

Inference Engineering

Bottleneck maths, attention and GPU architecture, the software stack, engines compared from the inside, and the techniques that matter: quantisation, speculative decoding, parallelism, caching and disaggregation. Ends with a benchmarking capstone.

4 tracks · 12 lessons · about 5 hours · 1 free lessons · 0 animated diagrams

Start the first lesson freeSee Pro

What you will be able to do

  • ✓Predict throughput and latency from memory bandwidth and FLOPs
  • ✓Explain what each engine optimisation changes and when it helps
  • ✓Choose parallelism, quantisation and caching strategies with trade-offs
  • ✓Benchmark honestly and report results others can reproduce

Who it is for

Engineers who want to reason about inference performance from first principles and tune it with evidence.

Prerequisites

Local LLM Inference basics (tokens, VRAM, prefill vs decode). Comfort reading Python.

Curriculum

Bottlenecks and Hardware

Advanced

Prefill versus decode, the roofline model, attention optimisations and GPU architecture for inference.

  1. 1Prefill, decode and the roofline: predicting inference speedFree18 min
  2. 2Attention optimisations: FlashAttention, GQA, paged KV and sliding windowsPro16 min
  3. 3GPU architecture for inference: memory hierarchy, tensor cores and generationsPro15 min

The Software Stack

Advanced

From CUDA kernels and compilers to the engines: llama.cpp, vLLM, SGLang, TensorRT-LLM and friends compared from the inside.

  1. 1The software stack: from CUDA kernels to inference enginesPro14 min
  2. 2Engines compared from the inside: llama.cpp, vLLM, SGLang, TGI, TensorRT-LLM, DynamoPro15 min

Performance Techniques

Advanced

Quantisation, speculative decoding, parallelism, disaggregated serving and caching, with the maths for each.

  1. 1Quantisation in depth: formats, groups, outliers, activations and KVPro17 min
  2. 2Speculative decoding in depth: acceptance maths, draft models, EAGLE and n-gramsPro15 min
  3. 3Model parallelism: tensor, pipeline, expert and data parallel inferencePro15 min
  4. 4Disaggregated prefill and decode: why and when to splitPro13 min
  5. 5Caching: prefix caching, semantic caching and KV offloadPro14 min

Benchmarking and Capstone

Advanced

Benchmark honestly, avoid common traps, and run the baseline-to-3x capstone.

  1. 1Benchmarking done properly: open loop, percentiles, goodput and honest reportingPro16 min
  2. 2Capstone lab: from baseline to 3x goodputLabPro120 min