What you will learn, and the plan
The goal of Inference Engineering (predict, explain and improve inference performance with evidence), the route through bottlenecks, the stack, techniques, benchmarking and the capstone, and what each part lets you do.
#The goal
Inference engineering is the craft of making a model respond faster, cheaper and more reliably, and knowing why. By the end you can look at a workload and a machine, predict its speed, pick the techniques most likely to help, apply them, and prove the improvement with an honest benchmark.
#The route
- 1Start herethe plan, vocabulary, units
- 2Bottleneckswhat limits speed: bandwidth, compute, memory, hardware
- 3The stackCUDA, kernels, engines, fusion and launch overhead
- 4Techniquesquantisation, speculation, parallelism, MoE, caching, scheduling, long context, profiling
- 5Benchmarkingmeasure honestly, then a capstone
| Part | Lessons | Time | Checkpoint |
|---|---|---|---|
| Start here | 3 | about 1.5 hours | You can compute a decode ceiling and a machine balance point by hand |
| Bottlenecks | 5 | about 5 hours | You can predict tokens per second for a model on given hardware |
| The stack | 3 | about 3 hours | You can say what each layer does and where launch overhead appears |
| Techniques | 12 | about 15 hours | You can choose techniques for a workload and say what each costs |
| Benchmarking and capstone | 3 | about 6 hours | A reproducible report taking a baseline to a measured gain |
#What you will be able to do
- Predict decode speed from memory bandwidth and weight size, and prefill speed from compute.
- Explain the roofline model and where a workload sits on it.
- Compare engines (vLLM, SGLang, TensorRT-LLM, llama.cpp) by what they optimise.
- Choose quantisation, caching, speculation, parallelism and scheduling settings with trade-offs in mind.
- Profile a run and recognise launch-bound, memory-bound and compute-bound behaviour.
- Benchmark with open-loop load, percentiles and goodput, and report honestly.
#Prerequisites
- Local LLM Inference basics: tokens, VRAM, prefill versus decode (the course).
- Python reading ability.
- Comfort with units: bytes, GB/s, FLOPs. The next lesson refreshes exactly this.
#Exercise
A model has 8 billion parameters stored at 16 bits. The GPU reads memory at 3,350 GB/s. In one sentence each: what limits decode speed, and what is the rough ceiling?
Answer: decode is limited by how fast the weights can be read from memory, because each token reads all of them. The weights are 16 GB, so the ceiling is about 3,350 / 16 ≈ 209 tokens per second for one stream, before overheads.
Next: units and vocabulary.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. What is the central skill this course builds?
2. In what order does the course go?
3. What does the capstone ask you to do?
Found this useful?
Send it to someone who is learning the same thing.