Serving AI in Production

Turn a working model into a service: SLOs, capacity planning, cost per token, hardware choices and gateways.

1. 1From laptop to production: what actually changesFree13 min
2. 2Capacity planning and cost per token1 diagramFree15 min
3. 3Choosing GPUs and deployment targetsPro14 min
4. 4Gateways, routing and failover1 diagramPro14 min

GPUs on Kubernetes, packaging models, production vLLM manifests, scheduling and multi-tenancy, streaming networking, serving platforms with GitOps, and a hands-on cluster lab.

1. 1Kubernetes for LLM engineers: the mental modelFree16 min
2. 2GPUs on Kubernetes: drivers, device plugin, sharing and healthPro17 min
3. 3Packaging models: images, weights and fast startupPro15 min
4. 4Deploying vLLM on Kubernetes: a production manifest, line by linePro19 min
5. 5Scheduling, node pools and multi-tenancy for GPUsPro16 min
6. 6Networking, ingress and streaming for LLM trafficPro15 min
7. 7Serving platforms and GitOps: KServe, KubeRay, llm-d, Helm and ArgoPro14 min
8. 8Lab: a GPU-style cluster on your laptopLabPro75 min

Queueing maths, autoscaling policies, cold starts, batch workloads and multi-region capacity, with a simulator lab.

1. 1Queueing theory for LLM capacityFree17 min
2. 2Autoscaling policies in depthPro18 min
3. 3Cold starts and fast model loadingPro17 min
4. 4Batch and offline inference at scalePro16 min
5. 5Multi-region serving and failoverPro17 min
6. 6GPU capacity management across cloudsPro16 min
7. 7Lab: an autoscaling simulatorLabPro60 min

Structured logs, log pipelines, Prometheus and Grafana for GPU inference, SLO alerting and OpenTelemetry tracing for RAG and agents, with a full observability-stack lab.

1. 1Structured logging for LLM servicesFree16 min
2. 2Log pipelines, cost and retentionPro16 min
3. 3Metrics and dashboards for GPU inferencePro17 min
4. 4SLOs, error budgets and alerting that people trustPro17 min
5. 5Distributed tracing with OpenTelemetryPro18 min
6. 6Tracing RAG pipelines and agentsPro14 min
7. 7Lab: build an observability stackLabPro75 min

Autoscaling, observability, safe rollouts, reliability patterns and a monitored deployment lab.

1. 1Autoscaling LLM workloads1 diagramFree14 min
2. 2Observability for LLM servicesFree14 min
3. 3Rollouts and model versioningPro14 min
4. 4Reliability patterns: limits, backpressure and load sheddingPro14 min
5. 5Lab: ship a monitored deploymentLabPro60 min

CI and canaries for models, incident response and postmortems, multi-tenancy and billing, client-side retries and timeouts, and unit-cost FinOps.

1. 1CI for models: tests, canaries, shadow traffic and rollbackFree15 min
2. 2Incident response, game days and blameless postmortemsPro14 min
3. 3Multi-tenancy, quotas and usage billingPro14 min
4. 4Client code: timeouts, retries, streaming and SDK designPro14 min
5. 5FinOps for LLM inference: unit cost, utilisation, spot and reservedPro15 min