Troubleshooting wiki

CUDA out of memory (OOM) when loading a model

Errors like CUDA error: out of memory, failed to allocate CUDA buffer or the process is killed at load.

Model runs on CPU instead of the GPU

Very slow generation (a few tokens/s on a big GPU); GPU utilization stays near 0%.

Extremely slow generation on a Mac

Tokens trickle out, the machine lags, Activity Monitor shows heavy swap or red memory pressure.

Gibberish, odd symbols or ignoring instructions

Output contains special tokens, repeats role names, ignores the system prompt or switches language randomly.

Model repeats itself in a loop

The same phrase or sentence repeats until the token limit.

Model forgets earlier messages or long documents

Long prompts are partly ignored; the model acts as if the beginning of the chat never happened.

error: unknown model architecture

llama.cpp refuses to load a new model with unknown model architecture: '...'.

vLLM fails at startup with an out-of-memory or KV cache error

No available memory for the cache blocks or the model needs more KV memory than available at the requested --max-model-len.

Address already in use (port 11434 / 8080)

Server fails to start: bind: address already in use.

Ollama keeps unloading the model (slow first response)

Every request after a few minutes of idle time has a long delay while the model reloads.

Tokens/second drops after a few minutes

Fast at first, then noticeably slower during a long chat or sustained load.

Invalid or truncated JSON from the model

Parsing fails; output has trailing text, markdown fences or is cut off.

Model takes a very long time to load

Startup takes minutes; first token is delayed after loading.

Function/tool calls are ignored or malformed

The model answers in prose instead of calling the tool, or arguments are broken.

GPU not detected on Windows or inside Docker

Runtime falls back to CPU; containers can't see the GPU.