What we are building: a language model, end to end

You use language models every day. This course takes the cover off. By the end you will have written the main parts of one yourself, run it, and understood every line. This first lesson gives you the picture of the whole thing so that each later lesson has somewhere to fit.

#The one idea

A language model does exactly one job:

Given some text, predict the next token.

A token is a small piece of text, often a word or part of a word (the first part of the course builds a tokenizer from scratch). Give the model The capital of France is and it produces a probability for every token it knows: high for Paris, low for banana. Pick a token, append it, ask again, and you are generating text.

That is all. Chat, code, summaries and translation are the same trick: if the training text contains patterns, a model that predicts text well has to learn those patterns.

#The whole pipeline on one page

1
Text"The cat sat"
2
Tokenizersplits text into pieces and maps each to an integer id
3
Embeddingseach id becomes a vector of numbers
4
Transformer blocksattention mixes information between positions, an MLP processes each one
5
Output layerone score per token in the vocabulary
6
Softmaxscores become probabilities
7
Samplerpicks the next token
8
Append and repeat

Every box in that picture is a lesson in this course:

Box	What it does	Where you build it
Tokenizer	Text to integer ids and back	Text to Tensors
Embeddings	A learned table: id to vector	Text to Tensors
Attention	Lets each position look at earlier positions	Attention and the GPT
MLP and normalisation	Per-position processing, stable training	Attention and the GPT
Output layer and softmax	Vectors to a probability for every token	Attention and the GPT
Training loop	Adjusts the numbers so predictions improve	Training and Generation
Sampler	Chooses a token (temperature, top-k, top-p)	Training and Generation
KV cache	Makes generation fast	Training and Generation

#Training in one paragraph

The model's behaviour lives in its parameters: millions or billions of numbers. At the start they are random, so predictions are random. Training repeats a simple loop:

Take a piece of text and hide the last token.
Ask the model to predict it.
Measure how wrong it was (the loss, which is low when the model gave the right token a high probability).
Compute how each parameter should change to reduce the loss (backpropagation).
Nudge every parameter a tiny amount (gradient descent, here the Adam optimiser).

Do that millions of times and the numbers organise themselves into something that predicts text. No one writes rules for grammar or facts; they are squeezed out of the data.

#What "from scratch" means here

Real models are trained with libraries such as PyTorch on GPUs. Those libraries are excellent, and they also hide the mechanism. So we do it in two passes:

Plain Python, no dependencies. A tokenizer, attention, a full GPT block, an automatic-differentiation engine and a training loop, all readable in a few hundred lines. Slow on purpose, and nothing is hidden.
Map to production. The same ideas as PyTorch code, then the upgrades modern models use (RoPE, RMSNorm, SwiGLU, grouped-query attention), fine-tuning with LoRA, and what changes at scale.

#How small is "tiny"?

The model you train in the lab has 4,128 parameters and learns to invent plausible first names. That is about a million times smaller than a small production model, so be clear about what to expect:

	The tiny model in this course	A production model
Parameters	about 4 thousand	billions
Training data	130 names	trillions of tokens
Learns	which letters follow which	grammar, facts, reasoning, code
Hardware	your laptop, one CPU core, a couple of minutes	thousands of GPUs, weeks
The code	a few hundred lines	the same ideas, optimised and parallelised

The tiny model is not impressive. It is correct, and correct is what you need to understand the big ones: the architecture, the loss, the gradients and the sampling loop are the same.

#What you will be able to do at the end

Write a byte-pair-encoding tokenizer and explain why token counts behave the way they do.
Implement self-attention with a causal mask and a full GPT block, and count its parameters exactly.
Train a tiny GPT and read its loss curve: what a good one looks like and what a broken one looks like.
Generate text with temperature, top-k and top-p, and speed it up with a KV cache.
Read a modern architecture (Llama-style) and say what each part is for and why it replaced the older choice.
Explain LoRA and instruction tuning well enough to reason about fine-tuning costs.

#A note on honesty

This course will tell you what the tiny models cannot do. It will not claim that a 4,000-parameter model understands language, and it will point out where production systems differ from the teaching code. Every code listing was run before it was published; numbers that depend on random seeds are marked.

#Exercise

Without looking back, write the steps a model performs to extend the text I like by one token, using the words tokenizer, embeddings, transformer blocks, softmax and sampler in the right order.

Answer: the tokenizer turns I like into ids; embeddings turn the ids into vectors; the transformer blocks process the vectors (attention, then MLP, repeated); the output layer produces a score per token and softmax turns them into probabilities; the sampler picks one token, which is appended to the text. Run the loop again for the next token.

Next: the plan shows how the course is laid out and how to work through it.