Local LLM Inference

Tokens, context windows, prefill vs decode and sampling: the mental model behind every local LLM.

1. 1What is local LLM inference?Free10 min
2. 2Tokens and context windows2 diagramsFree11 min
3. 3How generation works: prefill, decode and the KV cache2 diagramsFree12 min
4. 4Sampling parameters that matter1 diagramFree12 min

VRAM math, memory bandwidth, Apple Silicon and how to pick a machine that fits the models you want.

1. 1VRAM math: will this model fit?Free12 min
2. 2Memory bandwidth: the speed limit of decoding1 diagramFree11 min
3. 3Apple Silicon for local LLMsFree11 min
4. 4Hardware buying guide by memory tierPro14 min

GGUF, K-quants, AWQ, GPTQ, EXL2 and KV-cache quantization, and what each costs in quality.

1. 1Quantization basics1 diagramFree12 min
2. 2GGUF and the K-quant familyFree12 min
3. 3AWQ, GPTQ, EXL2 and FP8: GPU-first formatsPro22 min
4. 4KV-cache quantization1 diagramPro20 min
5. 5Lab: choose a quant for your hardwareLabPro45 min

Ollama, llama.cpp, vLLM and MLX: what each is for and how to run them properly.

1. 1The runtime landscape: which engine when?1 diagramFree12 min
2. 2Ollama quickstart and ModelfilesFree12 min
3. 3llama.cpp: build, run and serveFree13 min
4. 4vLLM for production servingPro24 min
5. 5MLX on Apple SiliconPro11 min

GPU offload, batching, flash attention, speculative decoding and prompt caching.

1. 1GPU offload, threads and batch sizes1 diagramFree12 min
2. 2Flash attention and long-context performancePro20 min
3. 3Speculative decoding1 diagramPro22 min
4. 4Prompt and prefix caching1 diagramPro20 min
5. 5Lab: benchmark your rig and compare with theoryLabPro40 min

OpenAI-compatible endpoints, Docker, concurrency, load testing and production hardening.

1. 1The OpenAI-compatible APIFree12 min
2. 2Docker and Compose deploymentPro12 min
3. 3Concurrency, throughput and load testing1 diagramPro13 min
4. 4Lab: a hardened production endpointLabPro45 min

Local RAG, structured output, function calling and agents on top of your own model.

1. 1Local RAG from scratch1 diagramFree16 min
2. 2Structured output: JSON schemas and grammarsPro12 min
3. 3Function calling (tools) with local models1 diagramPro14 min
4. 4Building local agents that don't fall apartPro14 min
5. 5RAG in depth: chunking, hybrid search, fusion, reranking and permissionsPro20 min
6. 6Agents: loops, memory, planning and the Model Context ProtocolPro18 min
7. 7Prompting that works on small local modelsPro14 min

Small models, WebGPU in the browser, phones and NPUs.

1. 1Small models: what 0.5B-4B can really doFree12 min
2. 2LLMs in the browser with WebGPUFree12 min
3. 3Phones, NPUs and embedded devicesPro13 min

Measure speed and quality honestly, pick the right model, and handle licenses and privacy.

1. 1Measuring speed correctlyFree12 min
2. 2Measuring quality: perplexity, KL divergence and task evalsPro14 min
3. 3Model licenses, provenance and privacyFree12 min
4. 4A framework for choosing the right modelPro13 min