Datasets, batches and embeddings: from token ids to vectors
How language-model training data is shaped: next-token targets as a one-position shift, context windows and batches, why the model sees every prefix at once, and embedding tables plus positional information. Runnable examples and the size arithmetic that decides memory.
16 min·Verified 2026-111 animated diagramPro+50 XP
The model's job is simple to state: given the tokens so far, predict the next one. Almost everything about data preparation follows from that.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.