Kubernetes for LLM engineers: the mental model

If you can run vLLM in a Docker container, Kubernetes is the system that runs many such containers across many machines, restarts them when they fail, replaces them during upgrades, routes traffic to the healthy ones and adds more when demand grows. It is the standard substrate for production LLM serving (production architecture). It is also large, with an overwhelming vocabulary. This lesson gives you the 20 percent that explains 90 percent of what you will do, with a focus on GPU inference.

#The one idea: desired state and controllers

You do not tell Kubernetes how to run things. You declare the desired state ("three replicas of this container, each with one GPU, reachable at this address") and controllers continuously compare reality with that declaration and fix the difference. A pod crashes: the controller starts another. A node dies: pods are rescheduled elsewhere. You change the image tag: the controller performs a rolling update.

This has a practical consequence: you edit files and apply them; you rarely run containers by hand. Treat manifests as code: review them, version them, test them (rollouts).

#The objects you need

Cluster
 ├─ Nodes (machines; some have GPUs)
 └─ Namespaces (logical groups: team-a, llm-prod, monitoring)
     ├─ Deployment ──manages──▶ ReplicaSet ──creates──▶ Pods (your vLLM container)
     ├─ Service ──routes to──▶ ready Pods (stable address)
     ├─ Ingress / Gateway ──exposes──▶ Service to the outside world
     ├─ ConfigMap / Secret ──inject──▶ settings and credentials
     └─ PersistentVolumeClaim ──mounts──▶ storage (weights, caches)

#Anatomy of a manifest

Every object is a YAML document with the same four top-level parts: apiVersion, kind, metadata, spec.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm
  namespace: llm-prod
  labels: { app: llm }
spec:
  replicas: 2
  selector:
    matchLabels: { app: llm }            # which pods this Deployment owns
  template:                              # the pod template
    metadata:
      labels: { app: llm }               # must match the selector
    spec:
      containers:
        - name: server
          image: vllm/vllm-openai:<pinned-tag>
          args: ["--model", "/models/qwen2.5-7b", "--max-model-len", "16384"]
          ports: [{ containerPort: 8000 }]
          resources:
            limits: { nvidia.com/gpu: 1 }

Read it from the inside out: a container with an image, arguments and a GPU limit, inside a pod template, inside a Deployment that keeps two copies running. Labels and selectors are how objects find each other: the Service selects pods by label, the Deployment owns pods by label.

#What is different about GPU workloads

Kubernetes was designed for CPU microservices. Several assumptions break with LLM servers:

Much of the rest of this track is about handling these differences: GPU drivers and the device plugin (next lesson), packaging weights (packaging), safe manifests (vLLM on Kubernetes), scheduling (scheduling) and networking for streams (networking).

#Resources: requests, limits and GPUs

Each container declares what it needs:

resources:
  requests: { cpu: "4", memory: 16Gi }       # what the scheduler reserves for placement
  limits:   { memory: 16Gi, nvidia.com/gpu: 1 }   # hard caps; GPU is an "extended resource"

• Requests decide placement: the scheduler only puts the pod on a node with that much unreserved capacity.
• Limits are enforced at runtime. Exceeding a memory limit kills the container (OOMKilled). CPU limits throttle.
• GPUs are extended resources advertised by a device plugin. You request them in limits (a request is implied). You cannot request half a GPU unless GPU sharing (time-slicing or MIG) is configured.
• GPU memory is not a Kubernetes resource. The scheduler does not know how much VRAM your model needs. Two pods will not share a GPU by default, but if you enable sharing, nothing stops them from overcommitting VRAM and crashing each other (GPUs on Kubernetes).
• Host memory matters too: loading a 40 GB model can need tens of GB of CPU RAM during startup. Size memory requests accordingly or pods are OOMKilled while loading.

#Probes: how Kubernetes knows your server is OK

For LLM servers: use a startup probe with a long allowance (for example 30 attempts × 10 s = 5 minutes) so a slow model load is not mistaken for a crash; make readiness mean "can serve a real request" (not just "port open"); keep liveness lenient so a busy but healthy server is not killed during a long generation (reliability patterns).

startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 10, failureThreshold: 60 }
readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 5,  failureThreshold: 3 }
livenessProbe:  { httpGet: { path: /health, port: 8000 }, periodSeconds: 20, failureThreshold: 6 }

#Rolling updates and rollbacks

Change the pod template (a new image tag, new args) and the Deployment replaces pods gradually:

kubectl -n llm-prod apply -f llm-deployment.yaml
kubectl -n llm-prod rollout status deployment/llm        # watch progress
kubectl -n llm-prod rollout undo deployment/llm          # go back to the previous version
kubectl -n llm-prod rollout history deployment/llm

Remember the GPU twist: the default strategy creates an extra pod before removing an old one (maxSurge: 25%). If no spare GPU exists, the new pod stays Pending and the rollout stalls. The fix, covered in the vLLM lesson, is to set maxSurge: 0 and maxUnavailable: 1 (or keep spare GPUs).

#The kubectl commands you will actually use

# see what exists
kubectl get nodes -o wide
kubectl get pods -n llm-prod -o wide                    # which node each pod is on
kubectl get deploy,svc,hpa,pdb -n llm-prod

# why is it not working?
kubectl describe pod <pod> -n llm-prod                   # events: scheduling failures, probe failures, OOMKilled
kubectl logs <pod> -n llm-prod --tail=200 -f
kubectl logs <pod> -n llm-prod --previous                # logs of the crashed container
kubectl get events -n llm-prod --sort-by=.lastTimestamp

# poke around
kubectl exec -it <pod> -n llm-prod -- nvidia-smi
kubectl port-forward svc/llm 8000:8000 -n llm-prod       # test the Service from your laptop

# change things
kubectl apply -f manifest.yaml
kubectl scale deployment/llm --replicas=3 -n llm-prod
kubectl rollout restart deployment/llm -n llm-prod
kubectl delete pod <pod> -n llm-prod                     # the Deployment recreates it

# GPU view
kubectl describe node <gpu-node> | grep -A8 "Allocatable"
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

Debugging flow for a pod that will not start: get pods (status) → describe pod (events at the bottom) → logs (and --previous) → exec if it runs. Common statuses: Pending (cannot be scheduled: no GPU, taints, quota), ContainerCreating (pulling image, mounting volumes), CrashLoopBackOff (crashes repeatedly: read the logs), OOMKilled (memory limit), ImagePullBackOff (bad tag or registry credentials).

#Scheduling in one paragraph

The scheduler places each Pending pod on a node that (1) has enough unreserved resources, (2) satisfies node selectors and affinity (for example "only nodes with an H100"), (3) tolerates the node's taints (GPU nodes are usually tainted so ordinary pods stay off them), and (4) fits spread rules (do not put all replicas on one node). You steer placement with labels, taints and tolerations (scheduling and multi-tenancy).

#Managed or self-hosted?

Kubernetes earns its complexity when you have several models, several teams, autoscaling needs, multiple environments or a platform group to run it. For a single model on one or two GPU servers, it can be overkill.

#Common mistakes

1. Using latest image tags, so pods differ between restarts.
2. No resource requests, so memory-hungry loads OOM their neighbours.
3. Liveness probes that kill a model during loading.
4. Default rolling update on a full GPU pool, stalling on Pending pods.
5. Running as root with a writable root filesystem and broad permissions (stack security).
6. Editing live objects with kubectl edit instead of changing files in git.
7. Mixing prod and experiments in one namespace with no quotas.
8. Assuming Kubernetes knows about GPU memory.

#Exercise

A Deployment asks for 3 replicas, each with nvidia.com/gpu: 2. The cluster has 4 GPU nodes with 4 GPUs each; 6 GPUs are already used by other pods.

1. How many GPUs does the Deployment need, and how many are free?
2. How many replicas can run, and what do the others show?
3. During a rolling update with the default surge of 25 percent, what could go wrong?

Answers:

1. Needs 3 × 2 = 6 GPUs; the cluster has 16 − 6 = 10 free (if they are spread so that two free GPUs exist on at least one node per replica; a replica needs both GPUs on the same node).
2. All three can run if three nodes each have at least 2 free GPUs; otherwise fewer. Replicas that cannot be placed stay Pending with an event like Insufficient nvidia.com/gpu.
3. The extra surge pod needs another 2 GPUs. If none are free, it stays Pending and the rollout waits. Set maxSurge: 0 with maxUnavailable: 1 (and size your capacity and PodDisruptionBudget for it) or keep a spare node.

#Going deeper

• Read the manifest above again and annotate every field in your own words.
• Continue with GPUs on Kubernetes, then try everything in the hands-on lab.
• Compare with the non-Kubernetes route in Docker and Compose deployment.