Placing a model across GPUs, CPU and RAM: MoE offload and multi-GPU splits
Decide which tensors live where: keep attention on the GPU and experts in RAM for mixture-of-experts models, split layers across unequal GPUs, choose between layer and row splits, and know what PCIe costs you.
14 min·Verified 2026-11Pro+50 XP
Share
A model has a few kinds of tensors, and they do not all deserve the same memory. Attention weights and the KV cache are read on every token. Expert weights in a mixture-of-experts model are huge but only a few of them are read per token. Knowing that lets you spend your fast memory where it pays. This lesson is about placement: which tensors go on which device.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.