Can I run Llama 3.1 70B on CPU only, dual-channel DDR5 (32GB)?
Not comfortably: Llama 3.1 70B only runs on CPU only, dual-channel DDR5 (32GB) with Q2_K by offloading part of the model to system RAM, which is much slower (often a few tokens/s). Smaller models or lower quantisations fit entirely in memory.
Memory by quantisation (8,192-token context)
| Quantisation | Memory | Verdict | Est. speed |
|---|---|---|---|
| F16Unquantized reference | 136.0 GiB | Does not fit | — |
| Q8_0Near-lossless | 74.4 GiB | Does not fit | — |
| Q6_KVery close to F16 | 58.5 GiB | Needs CPU offload | — |
| Q5_K_MGreat quality/size | 51.3 GiB | Needs CPU offload | — |
| Q4_K_MBest default for most | 44.4 GiB | Needs CPU offload | — |
| IQ4_XSSmaller 4-bit (imatrix) | 39.5 GiB | Needs CPU offload | — |
| Q3_K_MNoticeable loss | 36.7 GiB | Needs CPU offload | — |
| Q2_KLast resort; big loss | 32.1 GiB | Needs CPU offload | — |
CPU only, dual-channel DDR5 (32GB) offers about 32 GiB of memory (80% usable for a model) at 80 GB/s. Speed is a bandwidth-bound estimate of single-stream generation at 65% efficiency; real results vary by runtime, drivers and settings, so measure on your machine.
How this is calculated
Weights ≈ parameters × bits per weight ÷ 8. KV cache ≈ 2 × 80 layers × 8 KV heads × 128 head dim × 2 bytes per token. Decode speed ≈ efficiency × memory bandwidth ÷ bytes read per token. Learn the maths in VRAM math and memory bandwidth, or try the VRAM calculator.
Llama 3.1 70B on other hardware
Similar models on CPU only, dual-channel DDR5 (32GB)
Running slowly or out of memory? See CUDA out of memory and the quantisation lesson.