Mixture of experts: routing, memory and why MoE decodes fast but needs big GPUs
How an MoE layer routes each token to a few experts, why active parameters set compute but total parameters set memory, how many experts a batch really touches, why large batches erase the sparse advantage, and how load imbalance slows expert parallelism. All numbers come from code you can run.
Mixture of experts (MoE) is how many of the largest open models get high quality without paying the full compute price on every token. It also confuses people, because an MoE is cheap in one way and expensive in another. This lesson separates the two, with numbers you can reproduce.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.