Networking, ingress and streaming for LLM traffic
Why LLM traffic breaks default load balancing and timeouts: long-lived SSE streams, connection-based balancing, ingress and Gateway API settings, buffering, load-aware routing and prefix-cache affinity, mTLS and network policy, and client disconnect handling.
15 min·Verified 2026-11Pro+50 XP
Kubernetes networking was designed around short request/response traffic with similar cost per request. LLM traffic is long-lived, streamed and wildly uneven. Each of those properties breaks a default somewhere between the browser and the GPU.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.