Kubernetes for GPU Inference
GPUs on Kubernetes, packaging models, production vLLM manifests, scheduling and multi-tenancy, streaming networking, serving platforms with GitOps, and a hands-on cluster lab.
0 of 8 complete
- ▶ Up next16 min+50 XPKubernetes for LLM engineers: the mental modelThe Kubernetes concepts that matter for model serving (pods, deployments, services, nodes, probes, resources…
- Pro17 min+50 XPGPUs on Kubernetes: drivers, device plugin, sharing and healthHow a GPU becomes schedulable: drivers, the container toolkit, the NVIDIA device plugin and GPU Operator, extended…
- Pro15 min+50 XPPackaging models: images, weights and fast startupWhere model weights should live: baked into the image, downloaded at start, or mounted from a shared volume. Image…
- Pro19 min+50 XPDeploying vLLM on Kubernetes: a production manifest, line by lineA complete Deployment, Service, PodDisruptionBudget and HPA hook for vLLM: GPU limits, shared memory, probes sized…
- Pro16 min+50 XPScheduling, node pools and multi-tenancy for GPUsDesigning GPU node pools, priority classes and preemption, headroom with pause pods, quotas and LimitRanges…
- Pro15 min+50 XPNetworking, ingress and streaming for LLM trafficWhy LLM traffic breaks default load balancing and timeouts: long-lived SSE streams, connection-based balancing…
- Pro14 min+50 XPServing platforms and GitOps: KServe, KubeRay, llm-d, Helm and ArgoWhen raw Deployments stop being enough: what KServe, KubeRay, llm-d and Dynamo-style platforms add, how to choose…
- LabPro75 min+100 XPLab: a GPU-style cluster on your laptopCreate a local kind cluster, advertise a fake nvidia.com/gpu extended resource, taint a 'GPU' node, deploy a CPU…