Lab: an autoscaling simulator
Run a discrete-event simulator of an LLM fleet with realistic request lengths, cold starts, a daily ramp and a sudden spike. Compare static, reactive and scheduled policies, sweep cold start time, remove the spike, change the variance, and write a policy recommendation backed by numbers.
Goal: build intuition for autoscaling by experimenting on a simulator instead of production. You will compare policies on the same traffic, discover which parameters matter (cold start, headroom, variability, spikes), and end with a recommendation you can defend with numbers.
You need: Python 3.10+ and nothing else (standard library only). Time: about 60 minutes. Background: queueing theory, autoscaling policies, cold starts.
Pro content
Keep reading with Inference Academy Pro
Unlock this full lab, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.