The GPT block and model: residuals, normalisation, MLP and parameter counts
Assemble a decoder-only transformer: embeddings, repeated blocks of attention and MLP with residual connections and normalisation, the output head, and exact parameter-count formulas verified on a toy model and on GPT-2 small.
17 min·Verified 2026-112 animated diagramsPro+50 XP
A decoder-only language model is a stack of identical blocks between an embedding layer and an output layer. You already have the main ingredient, attention. This lesson puts the pieces together and gives you the arithmetic to size any such model.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 2 animated diagrams, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.