The maths and Python you need, in one sitting

You do not need much maths for this course, but you need it to be comfortable. Everything below is short, and you can run all of it. If you already know a section, skim it and move on.

Put the code in a file maths.py (or run the pieces one at a time) and check your output against what is shown.

#1. Python you will use constantly

nums = [3, 1, 4, 1, 5]
squares = [n * n for n in nums]        # a list comprehension
print(squares)                         # [9, 1, 16, 1, 25]

total = sum(nums)                      # 14
pairs = list(zip(nums, squares))       # [(3, 9), (1, 1), ...]
biggest = max(nums)

def scale(v, k):                       # a function that returns a new list
    return [k * x for x in v]

print(scale([1.0, 2.0], 0.5))          # [0.5, 1.0]

That is most of the Python in the course: lists, loops, comprehensions, zip, functions, dictionaries and the occasional class. Slicing (a[1:3]) and negative indexes (a[-1] is the last item) also appear.

#2. Vectors and the dot product

A vector is a list of numbers. A neural network represents everything (a token, a position, a hidden state) as a vector.

The dot product multiplies matching entries and adds them up:

q = [1.0, 2.0, 0.5]
k = [0.5, 1.0, 2.0]
dot = sum(a * b for a, b in zip(q, k))
print("dot", dot)                      # dot 3.5

1×0.5 + 2×1 + 0.5×2 = 3.5. If the vectors point in a similar direction the dot product is large; if they are unrelated it is near zero; if they point opposite ways it is negative. Attention uses exactly this to decide which earlier tokens matter (attention from first principles).

#3. Matrices and shapes

A matrix is a grid of numbers, written as shape (rows, columns). Multiplying a matrix by a vector takes a dot product of each row with the vector:

W = [[1, 0, 2],
     [0, 1, -1]]            # shape (2, 3)
x = [3, 4, 5]               # length 3
y = [sum(w * v for w, v in zip(row, x)) for row in W]
print("W @ x =", y)          # W @ x = [13, -1]

The shape rule: (2, 3) times a length-3 vector gives a length-2 vector. The inner sizes must match and then disappear. Almost every shape error you will ever see is this rule being broken, so say it to yourself whenever a message mentions shapes.

Every layer in a transformer is a matrix multiplication plus a few small operations. The parameters are the numbers inside the matrices.

#4. Softmax: scores to probabilities

A model produces a score (a logit) for every token. Softmax turns scores into probabilities that are positive and add up to 1:

import math

def softmax(scores):
    m = max(scores)                     # subtracting the max avoids overflow
    e = [math.exp(s - m) for s in scores]
    t = sum(e)
    return [v / t for v in e]

p = softmax([2.0, 1.0, 0.1])
print("softmax", [round(v, 3) for v in p], "sum", round(sum(p), 6))
# softmax [0.659, 0.242, 0.099] sum 1.0

Higher scores get more probability, and the ordering is kept. Softmax is how the output layer produces a distribution, and how attention turns match scores into weights.

#5. Loss: how wrong was the prediction?

Training needs one number that says how badly the model did. For next-token prediction it is cross-entropy: minus the natural log of the probability the model gave to the correct token.

print("loss if the right answer is index 0:", round(-math.log(p[0]), 3))   # 0.417
print("loss if the right answer is index 2:", round(-math.log(p[2]), 3))   # 2.317
print("loss for a uniform guess over 27 tokens:", round(math.log(27), 3))  # 3.296

Two things to remember. Confident mistakes are punished hard, because the log of a small number is a large negative number. And a model that knows nothing about a vocabulary of V tokens has a loss of about ln V. For the 27 characters in the lab that is 3.3, so a loss of 3.3 at step zero is correct, and the goal is to push it well below.

#6. Derivatives and gradient descent

A derivative tells you how much a function changes when you nudge its input. For training, the function is the loss and the inputs are the parameters. The vector of all these derivatives is the gradient. It points towards higher loss, so we step in the opposite direction.

def f(w):                               # a pretend loss: smallest when w = 2
    return (3 * w - 6) ** 2

w, h = 1.0, 1e-6
slope = (f(w + h) - f(w - h)) / (2 * h)          # estimate the derivative by nudging
print("numeric slope at w=1:", round(slope, 3), " exact:", 2 * (3 * w - 6) * 3)
# numeric slope at w=1: -18.0  exact: -18.0

The slope is negative, so increasing w lowers the loss. Now walk downhill:

w = 0.0
for step in range(20):
    grad = 2 * (3 * w - 6) * 3          # derivative of (3w - 6)^2
    w -= 0.02 * grad                    # step opposite to the gradient
print("after 20 steps w =", round(w, 4), "(the minimum is at 2)")
# after 20 steps w = 1.9997 (the minimum is at 2)

The 0.02 is the learning rate. Too small and you crawl; too large and you overshoot and the loss explodes. A real model does exactly this with millions of parameters at once, and backpropagation is the bookkeeping that computes all the derivatives efficiently with the chain rule. You write that bookkeeping yourself in Autograd and the training loop.

#What each idea is used for

If you can read the six code blocks above and predict what they print, you are ready.

#Common mistakes

• Forgetting softmax needs the max subtracted. Without it math.exp can overflow on large scores.
• Using the wrong log. The course uses the natural log (math.log).
• Moving along the gradient instead of against it. The sign matters.
• Mixing up the two sizes in a shape. Write the shapes down before multiplying.
• Expecting a loss of 0. Real models settle at a level set by how unpredictable the text is.

#Exercise

Work these out on paper first, then check them in Python.

1. The dot product of [2, -1, 0.5] and [1, 3, 4].
2. The shape of (4, 5) @ (5,), and of (4, 5) @ (5, 3).
3. The softmax of [0, 0, 0, 0].
4. The loss when the model gives the right token probability 0.25.
5. If the gradient at w is -18 and the learning rate is 0.02, what does w become when w = 1?

Answers:

1. 2×1 + (-1)×3 + 0.5×4 = 2 - 3 + 2 = 1.
2. (4,) and (4, 3).
3. [0.25, 0.25, 0.25, 0.25]: equal scores give equal probabilities.
4. -ln(0.25) = ln 4 ≈ 1.386.
5. w - 0.02 × (-18) = 1 + 0.36 = 1.36.

Next: you are ready for Text to Tensors, where you write your first piece of a language model.