Tokenizer to trained transformer, with code you run
Build an LLM from Scratch
Write a BPE tokenizer, attention, a GPT block, a training loop and a sampler in plain Python first, then see how the same ideas map to PyTorch and to modern upgrades like RoPE, RMSNorm, SwiGLU and grouped-query attention.
4 tracks · 10 lessons · about 4 hours · 1 free lessons · 0 animated diagrams
What you will be able to do
- ✓Implement tokenisation, attention and a transformer block yourself
- ✓Train a tiny language model and read its loss curve
- ✓Generate text with temperature, top-k and top-p, and a KV cache
- ✓Map your code to production architectures and fine-tuning methods
Who it is for
Developers who want to understand exactly what happens inside a language model.
Prerequisites
Python. High-school calculus helps; the course teaches the rest.
Curriculum
Text to Tensors
IntermediateWrite a byte-pair-encoding tokenizer, then turn token ids into batches and embeddings.
Attention and the GPT
IntermediateSelf-attention from first principles, the causal mask, multi-head attention and the full GPT block with exact parameter counts.
Training and Generation
IntermediateAutograd, Adam, a runnable tiny GPT, sampling strategies and the KV cache.
Modern Upgrades and Scaling
AdvancedRoPE, RMSNorm, SwiGLU, GQA, LoRA fine-tuning, scaling laws and the PyTorch mapping.