Serving & APIs
OpenAI-compatible endpoints, Docker, concurrency, load testing and production hardening.
0 of 4 complete
- ▶ Up next12 min+50 XPThe OpenAI-compatible APITalk to any local runtime with the OpenAI SDK: chat, streaming (and what the stream looks like), sampling parameters…
- Pro12 min+50 XPDocker and Compose deploymentContainerize llama.cpp and vLLM with GPU access, health checks, persistent model volumes, a TLS reverse proxy, log…
- Pro13 min+50 XPConcurrency, throughput and load testingWhy batching raises throughput, how slots and KV memory limit concurrency, how to load test honestly (open vs closed…
- LabPro45 min+100 XPLab: a hardened production endpointPut an authenticated, TLS-terminated, rate-limited and monitored llama.cpp server behind Caddy with systemd, then…