Modern upgrades: RoPE, RMSNorm, SwiGLU and grouped-query attention
What changed between GPT-2 and today's open models: rotary position embeddings (with a tested relative-position property), RMSNorm, SwiGLU feed-forward layers, grouped-query attention, and how each change affects quality, parameters and inference cost.
18 min·Verified 2026-111 animated diagramPro+50 XP
If you read a modern model's config (Llama, Mistral, Qwen, Gemma), four things differ from the GPT-2 design you built. Each is a small, well-motivated change.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.