What production means for LLMs, and the plan
The difference between a model that runs and a service that is reliable, affordable and safe to change; the six areas of the course in a sensible reading order; times, checkpoints and who should skip what.
#From "it runs" to "it keeps running"
Running a model on one machine is a solved problem. Production adds questions a demo never faces:
- How many users can it serve, and what happens at 10 times that?
- How do I know it is healthy, and how fast do I find out when it is not?
- How do I ship a new model or a new setting without breaking users?
- What does each answer cost, and who is paying?
- What if a GPU, a node or a region fails?
This course answers those for LLM inference specifically, where GPUs are scarce, models are large, and a request can last many seconds.
#The six areas
- 1Architectureworkload, capacity, GPUs, gateways
- 2Kubernetespackaging, scheduling GPUs, networking
- 3Scalingqueueing, autoscaling, cold starts, batch, regions
- 4Observabilitylogs, metrics, traces, SLOs
- 5Operationsreliability, rollouts, incident habits
- 6DeliveryCI for models, multi-tenancy, client code, FinOps
| Area | Question | Lessons | Time |
|---|---|---|---|
| Architecture | What are we building, and how big? | 5 | about 5 hours |
| Kubernetes | How do we run it on a cluster? | 8 | about 9 hours |
| Scaling | How does it grow and shrink? | 7 | about 8 hours |
| Observability | How do we see what is happening? | 7 | about 8 hours |
| Operations | How do we keep it reliable? | 5 | about 5 hours |
| Delivery | How do we change it safely and pay for it? | 5 | about 4 hours |
#A sensible order
You do not have to read the areas as listed on the course page. A path that follows how a project unfolds:
- Architecture first: the workload decides everything else.
- Kubernetes, if you will run on a cluster (skip it if you use a managed service).
- Observability, early, because every later step needs it.
- Scaling and Operations together.
- Delivery last, as it builds on all of the above.
#What you can skip
- Not using Kubernetes? Read only the architecture, observability and delivery areas, plus the lessons in Scaling that are platform-neutral.
- Single-model internal tool? Focus on architecture, observability and reliability patterns.
- Already run production services? Read the LLM-specific lessons (GPU scheduling, cold starts, queueing for long requests, FinOps).
#What you will be able to do
- Size a deployment from a workload and cost it.
- Package and run vLLM-style serving on a cluster with GPUs.
- Set autoscaling policies that respect cold starts and long requests.
- Instrument a service, define SLOs and alert on them.
- Roll out models with canaries and roll back safely.
- Control cost and attribute it to tenants.
#Prerequisites
- Local LLM Inference runtimes and serving (the course): you have run vLLM or llama.cpp and called an OpenAI-compatible API.
- Basic Docker and Linux, refreshed in the next lesson.
- You do not need to have run Kubernetes before; that area teaches it.
#Exercise
A team will launch a chat feature for 5,000 daily users with 2,000-token prompts and 300-token answers. Which area do you start with, and what is the first thing you would ask them?
Answer: Architecture, and the first thing is the workload: how many requests per second at peak, prompt and output lengths, and the latency target. Without those, no one can size GPUs or autoscaling.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. What separates a production service from a demo?
2. Which area should you decide first?
3. Why does the course put observability before rollouts?
Found this useful?
Send it to someone who is learning the same thing.