A tiny llama.cpp, written by you

Build an Inference Engine

Name: Build an Inference Engine
Price: 12 USD

Implement a complete forward pass, KV cache, sampling, quantised matrix multiplication, batching and an OpenAI-compatible server, first in readable Python and then with the C++ ideas that make it fast.

2 tracks · 7 lessons · about 5 hours · 1 free lessons · 0 animated diagrams

Start the first lesson free See Pro

What you will be able to do

✓Write a correct transformer forward pass with a KV cache
✓Implement 8-bit quantised matrix multiplication and measure the error
✓Add continuous batching and a streaming server
✓Profile and reason about memory-bound performance

Who it is for

Systems-minded engineers who want to know how engines like llama.cpp work.

Prerequisites

Build an LLM from Scratch (or equivalent) and basic C/C++ reading ability.

Curriculum

Engine Core

Advanced

The forward pass end to end, loading weights from GGUF with mmap, and KV cache and sampling inside an engine.

Making It Fast

Advanced

Quantised matrix multiplication, CPU speed techniques, continuous batching and a benchmark against llama.cpp.