Will it run? Hardware checker
Choose your hardware and a model to get a fit verdict and an estimated tokens-per-second figure.
Verdict
It fits, but headroom is thin. Reduce context or quantize the KV cache for safety.
Needs ~21.8 GiB; NVIDIA RTX 4090 24GB offers ~23 GiB usable of 24 GiB.
Estimated decode speed
~31 tok/s
1008 GB/s x 65% efficiency / bytes read per token (theoretical estimate).
Best quant that fits
Q4_K_M
Highest-quality quant that fits at this context on this device.
Estimates use published bandwidth and typical runtime efficiency. Measure your own system with the benchmark lab.
Learn more