Choosing GPUs and deployment targets
Memory capacity, bandwidth and interconnect: how to choose between consumer, workstation and data-center GPUs, how to split or replicate a model, and how to decide between rented, reserved, owned and serverless capacity using cost-per-token maths.
Picking hardware for production is a trade between three resources: memory capacity (can the model and its caches fit), memory bandwidth (how fast tokens decode) and price per useful token (what you pay for the work you actually deliver). This lesson gives you a method that works for any hardware generation, because the specific cards change every year but the reasoning does not.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.