Prompt and prefix caching
Reuse the KV cache across requests so long system prompts, tool definitions and documents are processed once: how matching works, how to structure prompts for hits, per-runtime setup, cache-aware RAG, and how to prove it is working.
If 90% of every request is identical (a system prompt, tool schemas, a long reference document), recomputing it each time wastes seconds of prefill per request. Prefix caching keeps the KV cache for the common prefix and processes only what is new. For long-prompt workloads it is often the largest latency win available, and it costs nothing in quality.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.