Speed estimates matrix
Compare devices for a given model, quant and context. These are model-based estimates, not measurements: decode speed is predicted from published memory bandwidth at ~65% efficiency ( how it works). Real results vary with runtime, drivers and settings, so measure yours with the benchmark lab.
| Hardware | Memory | Bandwidth | Fits? | Est. decode tok/s |
|---|---|---|---|---|
| NVIDIA RTX 5090 32GB | 32 GB | 1792 GB/s | Yes | ~227 |
| NVIDIA RTX 4090 24GB | 24 GB | 1008 GB/s | Yes | ~128 |
| AMD RX 7900 XTX 24GB | 24 GB | 960 GB/s | Yes | ~121 |
| NVIDIA RTX 3090 24GB | 24 GB | 936 GB/s | Yes | ~118 |
| Apple M3 Ultra (192GB) | 192 GB | 819 GB/s | Yes | ~104 |
| Apple M4 Max (128GB) | 128 GB | 546 GB/s | Yes | ~69 |
| NVIDIA RTX 4070 12GB | 12 GB | 504 GB/s | Yes | ~64 |
| Apple M3 Max (64GB) | 64 GB | 400 GB/s | Yes | ~51 |
| NVIDIA RTX 3060 12GB | 12 GB | 360 GB/s | Yes | ~46 |
| NVIDIA RTX 4060 Ti 16GB | 16 GB | 288 GB/s | Yes | ~36 |
| Apple M4 Pro (48GB) | 48 GB | 273 GB/s | Yes | ~35 |
| Apple M3 Pro (36GB) | 36 GB | 150 GB/s | Yes | ~19 |
| Apple M2 (16GB) | 16 GB | 100 GB/s | Yes | ~13 |
| CPU only, dual-channel DDR5 (32GB) | 32 GB | 80 GB/s | Yes | ~10 |
| CPU only, dual-channel DDR4 (16GB) | 16 GB | 45 GB/s | Yes | ~6 |