Modern Upgrades and Scaling
RoPE, RMSNorm, SwiGLU, GQA, LoRA fine-tuning, scaling laws and the PyTorch mapping.
0 of 3 complete
- ▶ Up nextPro18 min+50 XPModern upgrades: RoPE, RMSNorm, SwiGLU and grouped-query attentionWhat changed between GPT-2 and today's open models: rotary position embeddings (with a tested relative-position…
- Pro17 min+50 XPFine-tuning, LoRA and instruction tuningFrom a base model to an assistant: supervised instruction tuning, loss masking, why full fine-tuning is expensive…
- Pro18 min+50 XPScaling up and the PyTorch mappingWhat changes between a 4k-parameter toy and a real LLM: scaling laws and compute budgets (6ND, tokens per parameter)…