Glossary

43 terms every local LLM user runs into.

AWQ: Activation-aware Weight Quantization. A 4-bit method that protects the weight channels that matter most according to activations. Learn more →
Batch size (-b / -ub): Number of prompt tokens processed together. In llama.cpp, -b is the logical batch and -ub the physical micro-batch. Learn more →
Bits per weight (bpw): Average storage per parameter after quantization, including scale metadata. Q4_K_M is about 4.85 bpw. Learn more →
Chat template: The text format (special tokens and role markers) a model was trained with. Using the wrong one produces poor or garbled output.
Context window: Maximum number of tokens (prompt plus output) the model can attend to at once. Learn more →
Continuous batching: Scheduling that lets new requests join an in-flight batch at every decode step, greatly improving server throughput. Learn more →
Decode: The token-by-token generation phase. Memory-bandwidth-bound. Learn more →
Dense model: A model that uses all its parameters for every token (as opposed to Mixture of Experts).
EXL2 / EXL3: Quantization formats for ExLlama with variable bits per weight, optimized for NVIDIA GPUs.
Flash attention: An attention algorithm that avoids materializing the full score matrix, reducing memory use and traffic. Learn more →
FP8: 8-bit floating point supported natively on recent NVIDIA GPUs; near-lossless compared to FP16 with half the memory.
GBNF: llama.cpp's grammar format for constraining output to a formal grammar (such as JSON). Learn more →
GGUF: Single-file model format used by llama.cpp, Ollama and LM Studio. Contains tensors, tokenizer and metadata. Learn more →
GPTQ: Post-training quantization method using second-order information to minimize layer-wise error.
GQA (grouped-query attention): Attention variant where several query heads share one key/value head, shrinking the KV cache.
imatrix: Importance matrix: calibration statistics used to quantize more carefully, especially at low bit widths.
Inference: Running a trained model to produce output, as opposed to training it.
K-quants: llama.cpp quantization family (Q2_K to Q6_K) using super-blocks with quantized scales. Learn more →
KL divergence: A measure of how different two probability distributions are; used to compare quantized and original model outputs. Learn more →
KV cache: Stored attention keys and values for all earlier tokens so they need not be recomputed. Grows with context length. Learn more →
Layer offload (-ngl): How many transformer layers run on the GPU; the rest run on CPU. Learn more →
LoRA: Low-rank adaptation: a lightweight fine-tuning method that trains small adapter matrices.
Memory bandwidth: How fast memory can be read (GB/s). The main limit on decode speed. Learn more →
min-p: Sampling filter that drops tokens whose probability is below a fraction of the top token's probability. Learn more →
MLX: Apple's machine-learning framework for Apple Silicon. Learn more →
mmap: Memory-mapping a file so the OS loads model pages on demand; makes startup fast and lets the page cache be shared.
MoE (Mixture of Experts): Architecture with many expert sub-networks, only a few of which are active per token. Needs memory for all experts, computes with few.
PagedAttention: vLLM technique that stores the KV cache in fixed-size pages to avoid fragmentation. Learn more →
Perplexity: A measure of how well a model predicts text (lower is better). Used to compare quantizations. Learn more →
Prefill: Processing the prompt in parallel before generation starts. Compute-bound; determines time to first token.
Prefix caching: Reusing the KV cache of a shared prompt prefix across requests. Learn more →
Quantization: Storing weights in fewer bits to reduce size and increase speed at some quality cost. Learn more →
RAG: Retrieval-augmented generation: fetch relevant documents and include them in the prompt. Learn more →
Sampling: Choosing the next token from the model's probability distribution (temperature, top-p, etc.).
Slot: In llama-server, an independent sequence with its own share of the context and KV cache. Learn more →
Speculative decoding: A small draft model proposes tokens which the large model verifies in one pass, producing several tokens per step. Learn more →
Temperature: Scales the randomness of sampling; 0 is greedy. Learn more →
Tensor parallelism: Splitting each layer's matrices across multiple GPUs.
Token: The unit of text a model processes; about 4 characters or 0.75 English words. Learn more →
TTFT: Time to first token: the delay before output starts appearing. Dominated by prefill.
Unified memory: Architecture (Apple Silicon) where CPU and GPU share one memory pool. Learn more →
VRAM: Video memory on a GPU. Must hold the weights, KV cache and runtime buffers. Learn more →
WebGPU: Browser API for GPU compute; enables in-browser LLM inference. Learn more →