Anatomy of a production LLM service: the map for the whole course

Here is the whole system on one page. Every lesson in the course zooms into one box.

#A request, end to end

1
Clientapp or user
2
Gatewayauth, rate limits, routing, failover
3
Routerpicks a replica, prefix-aware where it helps
4
Model serverbatching, KV cache, generation on the GPU
5
Stream backtokens flow to the client
6
Logs, metrics and tracesrecorded on the way

#The systems around it

1
Cost and tenancyquotas, billing, FinOps
2
DeliveryCI for models, canaries, rollbacks
3
Observabilitylogs, metrics, traces, SLOs, alerts
4
Scalingautoscaling, cold starts, batch pools, regions
5
PlatformKubernetes, GPU scheduling, images, weights
6
Servingengine, batching, cache, API

Read it bottom up: you serve a model, run it on a platform, scale it, watch it, change it safely and pay for it.

#Where each area fits

Box	The question	Course area
Model server	How fast and how many users per GPU?	Inference Engineering and Architecture
Platform	How do I run and schedule GPUs?	Kubernetes
Scaling	How do I add and remove capacity?	Scaling
Observability	Is it healthy, and for whom?	Observability
Delivery	How do I change it without breaking it?	Operations and Delivery
Cost	What does a token cost?	Architecture and Delivery

#What makes LLM serving different

Property	Consequence
Requests last seconds and stream	Timeouts, proxies and load balancing need care
GPUs are scarce and slow to start	Autoscaling must plan for cold starts (loading tens of gigabytes)
Work per request varies enormously	Queueing and capacity planning use token counts, not request counts
State lives in the KV cache	Routing can raise hit rates; memory is a hard limit
Quality can regress silently	Evaluation belongs in CI and in production monitoring
Cost is per token and per GPU-hour	Utilisation is the main cost lever

#A production-readiness checklist

Before launch, can you say yes to each of these?

Workload known: peak requests per second, prompt and output lengths, latency targets.
Capacity planned: GPUs sized with headroom; cost per million tokens known.
Health checks and graceful shutdown work; replicas are removed from rotation before they stop.
Autoscaling is tested, including cold starts.
Observability: structured logs, GPU and engine metrics, traces for slow requests.
SLOs and alerts exist, and they page on user pain, not on noise.
Rollouts use canaries; rollback is one command and has been tried.
Limits: per-tenant quotas, request size caps, timeouts, max tokens.
Security basics: authentication, input limits, secrets handling (security course).
Runbooks: what to do when the GPU fails, the queue backs up or latency spikes.
Cost tracking by tenant or feature.
A capstone-style drill: you have run a load test and a failure test, and know what happened.

#How to use this map

Whenever a lesson feels abstract, find its box and ask what breaks without it. The course ends with a capstone that walks this checklist for a service of your own design.

#Exercise

A GPU node dies while 40 requests are streaming. Name three parts of the map that determine how bad this is for users, and one question to ask about each.

Answer: the gateway or router (does it retry or fail over, and are requests safe to retry?), scaling (how long until replacement capacity is ready, given the cold start?), and observability (how quickly does an alert fire, and do traces show which requests were hit?). The client code matters too: its timeouts and retry behaviour decide what users see.

Next: From laptop to production.