Can you run a 70B model at home? What hardware works

Seventy-billion-parameter models such as Llama 3.1 70B are a big quality step up from 8B ones, and they are the first size where "can I run it?" gets hard. The short answer: yes, but not on a single typical gaming GPU.

#The number to beat

At Q4_K_M, Llama 3.1 70B has weights of about 39.8 GiB. With an 8,192-token KV cache and runtime overhead, the total is about 44.4 GiB. Dropping to Q3_K_M brings it to 36.7 GiB and Q2_K to 32.1 GiB, but quality falls off quickly below 4 bits (why).

So you need roughly 48 GB or more of fast memory available to the model. Our estimates assume a bit of headroom, and about a quarter of an Apple Silicon Mac's memory is not available to the GPU by default.

#What works

Single GPUs. An RTX 3090, 4090 (24 GB) or 7900 XTX (24 GB) holds only about 22 GiB, so the model would need to be split between GPU and system RAM. An RTX 5090 (32 GB) is closer but still short at 4-bit. Offloading roughly half the model to RAM makes generation much slower, because the part in system RAM is limited by RAM bandwidth. See Llama 3.1 70B on an RTX 4090 for the exact figures.

Two GPUs. Two 24 GB cards give 48 GB between them, which holds the 4-bit model with a short context. Splitting layers across cards gives capacity but not extra speed (multi-GPU placement).

Apple Silicon. Unified memory is the easiest route. An M4 Max with 128 GB or an M3 Ultra with 192 GB holds the model with plenty of room, and the M3 Ultra's 819 GB/s of bandwidth puts a 4-bit 70B at roughly 12 tokens per second by our estimate (an M4 Max is about 8). A 64 GB Mac fits it but with little spare memory, because only about 75 percent is available to the GPU by default (Llama 3.1 70B on an M3 Max 64GB).

CPU only. A machine with 64 GB or more of DDR5 can load the model, but at about 80 GB/s the speed is only about 1 token per second. That is usable for batch jobs and not for chat.

#The cost-effective routes

#Before you buy

1. Check whether you need 70B. A well-quantised 14B or 32B model handles many tasks well, runs several times faster, and fits on one GPU. Use the Can I run it? pages to compare.
2. Count context. A 70B model's KV cache is 320 KB per token at 16-bit (80 layers, 8 KV heads), so a 32K context adds about 10 GiB on top of the weights. Quantising the cache to 8 bits halves it.
3. Measure, then decide. The buying guide by budget covers the trade-offs, and the self-host vs API calculator shows when buying hardware beats paying per token.

For many people the best answer is to run an 8B to 32B model locally for private or offline work and call a hosted model for the occasional task that really needs 70B.