Scaling
Queueing maths, autoscaling policies, cold starts, batch workloads and multi-region capacity, with a simulator lab.
0 of 7 complete
- ▶ Up next17 min+50 XPQueueing theory for LLM capacityWhy latency explodes near full utilisation, how pooling many slots changes everything, how to use Little's law and…
- Pro18 min+50 XPAutoscaling policies in depthDesign autoscaling as a feedback control problem: target tracking maths, why delay causes oscillation, headroom from…
- Pro17 min+50 XPCold starts and fast model loadingBreak a GPU replica's start-up into its stages (node, image, weights, runtime, warm-up), compute load times from…
- Pro16 min+50 XPBatch and offline inference at scaleRun millions of generations cheaply: when batch beats online, throughput-oriented engine settings, job queues with…
- Pro17 min+50 XPMulti-region serving and failoverDecide whether you need more than one region, choose between active-passive and active-active, size capacity so…
- Pro16 min+50 XPGPU capacity management across cloudsBuy GPUs like a portfolio: on-demand, committed, spot and owned capacity; the effective-price maths of commitments…
- LabPro60 min+100 XPLab: an autoscaling simulatorRun a discrete-event simulator of an LLM fleet with realistic request lengths, cold starts, a daily ramp and a sudden…