Capstone lab: from baseline to 3x goodput
Take one model on one GPU from a naive baseline to at least 3x goodput under fixed SLOs, changing one lever at a time (batching limits, prefix caching, chunked prefill, FP8/INT4, KV precision, speculative decoding), measured with an open-loop load generator you can run, and finish with a reproducible report.
Goal: apply the whole course. Starting from an untuned deployment, raise goodput (requests per second that meet your latency SLOs) by at least 3x on the same GPU, with every step measured and explained.
You need: a machine with one NVIDIA GPU (24 GB or more recommended), a serving engine such as vLLM or SGLang, an open model of 7 to 8B parameters, Python 3.10+, and about two hours. If you have no GPU, run the lab against a rented single-GPU instance, or practise the method and the load generator on a hosted OpenAI-compatible endpoint (expect limited control of the levers).
Honest limits. The load generator below was tested against a local fake streaming server, not against your engine; engine flags change between versions, so use the flag names from your engine's current documentation. The levers' effects depend on your hardware and workload; "3x" is a target the lab's method aims at, not a guarantee. If you cannot reach it, the report of what limited you is a valid result.
Pro content
Keep reading with Inference Academy Pro
Unlock this full lab, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.