VRAM math: will this model fit?

"Will this model fit on my GPU?" is the first question in local inference, and it has a clean answer you can compute in a minute. Total memory is roughly:

total = weights + KV cache + runtime overhead

Three terms, each with a formula. Learn them once and you can size any model, quant and context.

#1. Weights

weights_bytes = parameters × bits_per_weight / 8

Quantised formats store scale values alongside the weights, so use the effective bits per weight, not the headline number:

Example: Llama 3.1 8B at Q4_K_M = 8.03e9 × 4.85 / 8 = 4.87 GB, which is about 4.5 GiB. Llama 3.1 70B at Q4_K_M is about 40 GiB.

GB versus GiB: a gigabyte is 10⁹ bytes, a gibibyte is 2³⁰ (1.074 × 10⁹). Memory tools differ in which they show, and the gap is 7 percent. Pick one unit and stay with it. GPU spec sheets usually mean GiB-ish capacity, while model files are listed in GB.

Rule of thumb: 4-bit weights need about 0.55 to 0.6 GB per billion parameters, 8-bit about 1.05, 16-bit 2. See quantization basics for what each format costs in quality.

#2. KV cache

Every token in the context stores a key and a value vector per layer. That memory grows with context and with concurrent sequences:

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element
kv_total           = kv_bytes_per_token × context_tokens × parallel_sequences

Llama 3.1 8B (32 layers, 8 KV heads, head dim 128, 16-bit): 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token. An 8K context costs 1 GiB, 32K costs 4 GiB, and 128K costs 16 GiB.

KV size per token for a few real models (16-bit cache):

Models with grouped-query attention (GQA) have few KV heads, which is why modern models have much smaller caches than older ones. Notice Phi-3 mini: a 3.8B model whose cache per token is three times larger than an 8B Llama's, because it has 32 KV heads versus 8. A small model is not automatically a cheap-context model.

You can read these numbers from a model's config.json (num_hidden_layers, num_key_value_heads, hidden_size / num_attention_heads). If num_key_value_heads is absent, the model uses full multi-head attention and the KV head count equals num_attention_heads.

Concurrency multiplies it: four users, each with an 8K window, on Llama 3.1 8B is 4 × 1 GiB = 4 GiB. In llama.cpp, -c 32768 -np 4 gives four slots of 8,192 tokens each, and all of that memory is reserved up front.

#3. Overhead

Compute buffers for the current batch, the CUDA or Metal context, scratch space and allocator fragmentation: budget roughly 0.5 to 1.5 GiB, more at long context, large batch sizes (-b, -ub) or with multiple GPUs. If your GPU also drives your display, the desktop takes another 0.3 to 1 GiB or more (browsers with hardware acceleration are notorious).

#Putting it together

Llama 3.1 8B, Q4_K_M, 8K context, one user:

weights  4.5 GiB
KV       1.0 GiB   (8,192 × 128 KiB)
overhead 0.8 GiB
total    6.3 GiB

It fits an 8 GB GPU with about 1.5 GiB to spare, but at 32K context the cache alone is 4 GiB and the total is about 9.3 GiB, which does not fit.

A fit check you can reuse:

Aim for at least 10 to 15 percent headroom. A model that fits to the last megabyte will fail the moment the context grows, a second tab opens or the driver allocates a buffer.

Use the VRAM calculator to do this for any model, quant and context, and the hardware checker to compare against real devices.

#Other things that take memory

• Multimodal models add a vision encoder (hundreds of MB to a few GB) and image tokens in the context.
• Embedding and reranker models used in RAG need their own memory (local RAG).
• LoRA adapters add a little each.
• Draft models for speculative decoding add the draft model's weights and cache (speculative decoding).
• Mixture-of-experts models need memory for all experts even though only a few are active per token.
• Multiple servers on one GPU each carry their own overhead.

#When it does not fit

Options, roughly from least to most painful:

1. Quantize the KV cache (8-bit roughly halves it, with little loss): see KV-cache quantization.
2. Shrink the context to what you actually use. Most tasks need under 8K.
3. Choose a smaller quant (Q4_K_M instead of Q5 or Q6). Below about 3 bits, quality drops quickly.
4. Reduce concurrency, or shorten per-user context.
5. Pick a smaller or more efficient model (one with GQA and a modern architecture).
6. Offload some layers to CPU RAM. It works, but every offloaded layer runs at system-memory speed, so decode slows a lot (GPU offload).
7. Add memory: a bigger card, a second card, a Mac with more unified memory (buying guide).

#Common mistakes

1. Sizing by file size alone, forgetting the KV cache and overhead.
2. Using the headline "24 GB" as if all of it were free.
3. Mixing GB and GiB.
4. Ignoring parallel slots. -np 8 with a large window multiplies cache memory.
5. Assuming a long-context model should be run at its maximum context.
6. Forgetting that MoE needs all experts resident.

#Exercise

You want to run Qwen2.5 32B at Q4_K_M (about 32.8B parameters) on a 24 GiB GPU that also drives your display (assume 1 GiB used by the desktop). Context 16K, one user. KV is 256 KiB per token.

1. How big are the weights and the KV cache?
2. Does it fit with 1.2 GiB of overhead?
3. What would you change?

Answers:

1. Weights: 32.8e9 × 4.85 / 8 = 19.9 GB ≈ 18.5 GiB. KV: 16,384 × 256 KiB = 4 GiB.
2. Total 18.5 + 4 + 1.2 = 23.7 GiB, against about 23 GiB available. It does not fit.
3. Quantize the KV cache to 8-bit (saves about 2 GiB, new total 21.7 GiB, a thin margin), or cut the context to 8K (saves 2 GiB), or move the display to integrated graphics to free the desktop's memory. Doing the KV quantization and a 12K context would leave comfortable headroom.