How much VRAM do you need to run an LLM?
A worked method for sizing memory: weights, the KV cache and runtime overhead, with a Llama 3.1 8B example at every quantisation and a rule of thumb you can do in your head.
· 3 min read
"Will it fit?" is the first question about any local model, and it has a precise answer. Memory use has three parts: the weights, the KV cache that grows with conversation length, and a small runtime overhead. Add them up and compare with what your GPU (or unified memory) can give a model.
#1. Weights
Weights take parameters times bits per weight, divided by eight to get bytes:
A quantised file does not use exactly 4 or 8 bits per weight, because each small block also stores a scale. Q4_K_M averages about 4.85 bits per weight and Q8_0 about 8.5.
For Llama 3.1 8B (8.03 billion parameters):
| Format | Bits per weight | Weights |
|---|---|---|
| F16 | 16 | 14.96 GiB |
| Q8_0 | 8.5 | 7.95 GiB |
| Q4_K_M | 4.85 | 4.53 GiB |
#2. KV cache
Every token already processed keeps a key and a value vector in every layer, so the model need not recompute them. Its size per token is:
Llama 3.1 8B has 32 layers, 8 KV heads (grouped-query attention) and a head dimension of 128. At 16-bit that is 131,072 bytes per token, or 128 MB per 1,000 tokens. An 8,192-token conversation holds 1 GiB; a full 131,072-token context would hold 16 GiB, more than the Q4 weights several times over.
Models without grouped-query attention are far hungrier. Phi-3 mini has 32 KV heads of dimension 96 across 32 layers, so it needs about 393 KB per token, three times as much as Llama 3.1 8B despite being a smaller model.
#3. Overhead
Compute buffers and the runtime itself take roughly half a gigabyte plus a little more for bigger models and longer contexts. Treat it as 0.5 to 1 GiB.
#Putting it together
For Llama 3.1 8B at an 8,192-token context:
| Quantisation | Total memory |
|---|---|
| F16 | 16.8 GiB |
| Q8_0 | 9.8 GiB |
| Q4_K_M | 6.4 GiB |
| Q3_K_M | 5.5 GiB |
So a 12 GB card runs Q8_0 comfortably, and an 8 GB card runs Q4_K_M with room to spare. You can check any combination with the VRAM calculator or look up a specific pair, such as Llama 3.1 8B on an RTX 3060 12GB.
#A rule of thumb
For a quick mental estimate at 4-bit: about 0.6 GiB per billion parameters, plus 1 to 2 GiB. An 8B model is around 6 GiB, a 14B around 10 GiB, and a 70B about 44 GiB. That last number is why a 70B model does not fit on a single 24 GB card, and why it needs a Mac with plenty of unified memory, two GPUs, or CPU offload (can you run a 70B model at home?).
#What is left over is not all yours
Two practical corrections:
- Not all memory is available. A desktop GPU running your display loses a little, and an Apple Silicon Mac only lets the GPU use about two thirds to three quarters of unified memory by default.
- Leave headroom. Running at 98 percent of capacity invites out-of-memory errors when a long prompt arrives. Aim for 90 percent or less.
#If it does not fit
In order of how much quality they cost: shorten the context, quantise the KV cache to 8 bits, step down one quantisation level, choose a smaller model. Dropping below 4 bits hurts quality quickly, so a smaller model at Q4 usually beats a larger one squeezed to Q2 (choosing a quantisation).
The lessons VRAM math and KV-cache quantisation go through this with exercises.
Want the full picture?
The lessons teach this step by step with animated diagrams, quizzes and hands-on labs. Start with What is local inference?, or try the VRAM calculator and Can I run it? checker.