Can I run Gemma 2 9B on NVIDIA RTX 5090 32GB?
Yes: Gemma 2 9B runs on NVIDIA RTX 5090 32GB with F16 at roughly 55 tokens/s. Estimated memory 20.7 GiB at 8,192 tokens of context.
Memory by quantisation (8,192-token context)
| Quantisation | Memory | Verdict | Est. speed |
|---|---|---|---|
| F16Unquantized reference | 20.7 GiB | Fits | ~55 tok/s |
| Q8_0Near-lossless | 12.6 GiB | Fits | ~92 tok/s |
| Q6_KVery close to F16 | 10.5 GiB | Fits | ~112 tok/s |
| Q5_K_MGreat quality/size | 9.6 GiB | Fits | ~124 tok/s |
| Q4_K_MBest default for most | 8.7 GiB | Fits | ~138 tok/s |
| IQ4_XSSmaller 4-bit (imatrix) | 8.0 GiB | Fits | ~151 tok/s |
| Q3_K_MNoticeable loss | 7.7 GiB | Fits | ~159 tok/s |
| Q2_KLast resort; big loss | 7.1 GiB | Fits | ~174 tok/s |
NVIDIA RTX 5090 32GB offers about 32 GiB of memory (94% usable for a model) at 1792 GB/s. Speed is a bandwidth-bound estimate of single-stream generation at 65% efficiency; real results vary by runtime, drivers and settings, so measure on your machine.
How this is calculated
Weights ≈ parameters × bits per weight ÷ 8. KV cache ≈ 2 × 42 layers × 8 KV heads × 256 head dim × 2 bytes per token. Decode speed ≈ efficiency × memory bandwidth ÷ bytes read per token. Learn the maths in VRAM math and memory bandwidth, or try the VRAM calculator.
Gemma 2 9B on other hardware
Similar models on NVIDIA RTX 5090 32GB
Running slowly or out of memory? See CUDA out of memory and the quantisation lesson.