Gateways, routing and failover
Put a gateway in front of your model servers for auth, limits, routing, retries and failover. Covers routing strategies, health checks, timeouts, retry budgets, circuit breakers, fallback models, streaming and example configurations.
14 min·Verified 2026-111 animated diagramPro+50 XP
Clients should never talk to a model server directly. A gateway (a reverse proxy or an API service) gives you one stable endpoint and one place to enforce policy. It is also where most LLM-specific reliability work lives, because the failure modes of GPU inference (slow, long-lived, expensive requests) are unlike ordinary web traffic.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.