Lab: train a tiny GPT in pure Python
Run a complete 4,128-parameter GPT, with tokenizer, attention, MLP, autograd, Adam and sampling, in one dependency-free Python file. Watch the loss fall from ln 27 to about 2.2, then experiment with width, heads, learning rate and data to see what each does.
Goal: run, read and modify a complete GPT. Everything is here: a character tokenizer, an autograd engine, attention with a KV cache, an MLP, RMSNorm, residual connections, Adam with a decaying learning rate and temperature sampling. It uses only the Python standard library.
You need: Python 3.9+ and a few minutes. The run below took about 105 seconds on the author's laptop with a single core (pure Python is slow on purpose: nothing is hidden).
Honest limits. The model is tiny (4,128 parameters) and trained for 300 steps on 130 first names, so it learns letter patterns, not language. The code is checked by running it; the exact loss numbers depend on the random seed and the Python version.
Pro content
Keep reading with Inference Academy Pro
Unlock this full lab, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.