CUDA out of memory (OOM) when loading a model
Symptom
Errors like CUDA error: out of memory, failed to allocate CUDA buffer or the process is killed at load.
Likely causes
- Weights + KV cache + overhead exceed VRAM
- Context size (-c) or parallel slots too large
- Another process (browser, another model) is using VRAM
How to fix it
- Estimate memory with the VRAM calculator at your real context and slot count.
- Lower -c, reduce -np (slots), or choose a smaller quant.
- Quantize the KV cache (--cache-type-k q8_0 --cache-type-v q8_0 with flash attention).
- Offload fewer layers (-ngl) as a last resort; speed will drop.
- Check what else uses the GPU with nvidia-smi and close it.
bash
nvidia-smi
llama-server -m model.gguf -ngl 99 -c 8192 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn onStill stuck? Check the lessons or measure first with the benchmark lab.