Attention and the GPT
Self-attention from first principles, the causal mask, multi-head attention and the full GPT block with exact parameter counts.
0 of 2 complete
- ▶ Up nextPro20 min+50 XPAttention from first principles: queries, keys, values and the causal maskDerive self-attention from the idea of soft lookup, implement causal scaled dot-product attention in plain Python…
- Pro17 min+50 XPThe GPT block and model: residuals, normalisation, MLP and parameter countsAssemble a decoder-only transformer: embeddings, repeated blocks of attention and MLP with residual connections and…