Autograd and the training loop: loss, backpropagation and Adam
Build a scalar automatic-differentiation engine, verify its gradients numerically, then use it to understand cross-entropy loss, gradient descent, the Adam optimiser, learning-rate schedules and how to read a loss curve.
22 min·Verified 2026-111 animated diagramPro+50 XP
Training means adjusting millions of numbers so the model assigns high probability to the true next token. The adjustment direction comes from gradients, and automatic differentiation computes them for you. Here you will build the engine from scratch in about 60 lines and see that there is no magic in it.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.