Batch and offline inference at scale
Run millions of generations cheaply: when batch beats online, throughput-oriented engine settings, job queues with idempotency and checkpointing, spot and preemptible capacity, sharing GPUs with online traffic, sizing a run with real arithmetic, and the failure modes that turn a cheap job into an expensive one.
Not every token needs to be delivered in under a second. Embedding a document archive, summarising last month's tickets, generating synthetic training data, running evaluation suites, re-scoring a dataset with a new model: these are batch workloads. They have deadlines measured in hours, not milliseconds, and treating them like online traffic is the most common way to overspend on GPUs.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.