What we are building: an inference engine you can read
Start here. What an inference engine is, the pieces we build (forward pass, weight loading, KV cache, quantised matmul, batching, a server), what the code is written in, and how our small engine relates to production engines like llama.cpp and vLLM.
An inference engine takes a trained model (its weights) and turns token ids into more token ids. llama.cpp, vLLM, MLX and TensorRT-LLM are all inference engines. This course builds a small one so that you see every step that those systems optimise.
#What an engine does
- 1Load weightsread a model file into memory
- 2Embed the tokenlook up its vector
- 3Run the layersattention with a KV cache, then the feed-forward block
- 4Output scoresone logit per vocabulary token
- 5Samplepick the next token
- 6Stream it outand repeat
Embed
token id → vector
RMSNorm
normalise
Attention + KV cache
append k,v; attend to all cached keys
SwiGLU MLP
feed-forward, plus residual
Logits
one score per vocabulary token
Sample
pick the next token
KV cache (one entry per token so far)
A token enters the engine
The latest token id is looked up in the embedding table, giving a vector of numbers.
Our small engine follows exactly this path (Llama-style: RMSNorm, RoPE, grouped-query attention, SwiGLU). Production engines run the same steps with heavy optimisation.
The model is trained elsewhere. The engine only needs the forward pass. Everything that makes an engine good is about doing that pass cheaply: reading weights fewer times, storing them in fewer bits, reusing work (the KV cache), serving many users in one pass (batching) and exposing it through an API.
#What we build, piece by piece
| Piece | Lesson part | The code |
|---|---|---|
| Forward pass of a Llama-style decoder (RMSNorm, RoPE, grouped-query attention, SwiGLU) | Engine Core | engine.py |
| Loading weights from a GGUF-style file with memory-mapping | Engine Core | gguf_mini.py |
| KV cache and sampling | Engine Core | inside engine.py |
| INT8 and 4-bit quantised matrix multiplication | Engine Fast | quant.py |
| Threads, SIMD and cache layout (with a C benchmark) | Engine Fast | matvec_bench.c |
| Static versus continuous batching | Engine Fast | batching.py |
| An HTTP server | Engine Fast | server.py |
| Benchmarking against llama.cpp | Engine Fast lab | measured on your machine |
#What language, and why
Almost everything is Python with no dependencies, in short files you can read in one sitting. One benchmark is in C, because the effects of memory layout and SIMD are only visible when you control memory yourself, and we keep it small enough to read without being a C expert (the setup lesson has a short primer).
So the prerequisite is Python, not C or C++. If you can read a Python function, you can follow the engine.
#The honest scale
| Our engine | llama.cpp or vLLM | |
|---|---|---|
| Model | A tiny random-weight Llama-style model (2 layers, width 16) | Billions of trained parameters |
| Language | Python (plus one C file) | C++ and CUDA, Metal or other kernels |
| Speed | Slow on purpose | Highly optimised |
| Correctness | The same architecture and the same algorithms | The same, plus hundreds of optimisations |
With random weights the model's output is meaningless. That is fine: what we test is that the mechanism is right, for example that cached decoding gives the same logits as reprocessing the whole sequence.
#What you will be able to do at the end
- Write the forward pass of a modern decoder and explain each operation.
- Load weights from a GGUF-style file without copying them, and say why memory-mapping matters.
- Implement a KV cache and prove it matches the uncached result.
- Quantise weights to INT8 and 4-bit and measure the error and the speed effect.
- Explain why decode is memory-bound and what threads, SIMD and layout change.
- Simulate static and continuous batching, and serve the engine behind an HTTP API.
- Benchmark honestly against a production engine.
#Exercise
Name the engine pieces, in order, that run when you press enter on the prompt Hello and the engine writes its first token.
Answer: load the weights (once, at start-up); embed the token ids of Hello; run the layers (attention with the KV cache filling as it goes, then the feed-forward block, in every layer); compute logits; sample a token; stream it out. The next steps repeat from embedding the new token, now reading from the cache.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. What is an inference engine?
2. In what language is the engine in this course written?
3. Why does the course build a slow, tiny engine?
4. Which piece makes generating token n+1 cheaper than reprocessing the whole text?
Found this useful?
Send it to someone who is learning the same thing.