Activation functions and the MLP block: why nonlinearity, ReLU to SwiGLU, and what matmul costs
Why stacking linear layers without an activation collapses to one layer, the common activation functions (ReLU, GELU, SiLU, gated SwiGLU) with numbers, how an MLP block is built and counted, and how the arithmetic intensity of a matrix multiply explains why small batches are memory-bound. Everything is runnable in pure Python.
Almost every layer in a neural network does two things: a matrix multiply (the linear layer) and a nonlinear function (the activation). You may know it as "multiplying a matrix by a vector". This lesson covers the second, shows how they combine into the MLP block that holds most of a transformer's parameters, and connects the matrix multiply back to why inference speed behaves the way it does.
Everything below runs in plain Python with no libraries.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 2 animated diagrams, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.