Tuning a local server: parallel slots, keep-alive, queues and context shift

A server that feels fast to you alone can feel slow the moment a second tab, a coding agent and a teammate join. The settings below control how the server shares one model between requests. They matter as much as the choice of quant.

#Slots: how many conversations at once

llama.cpp's server and Ollama both keep several slots. Each slot holds one conversation's KV cache and can generate in parallel with the others.

llama-server -m MODEL.gguf -ngl 99 -c 32768 -np 4   # four slots

Two things to remember:

1. The context is divided across slots. -c 32768 -np 4 gives each conversation about 8192 tokens, not 32768. If you need long context per user, raise -c in proportion.
2. Every slot costs cache memory whether it is busy or not. Four slots with 8K each use as much cache as one slot with 32K (cache size).

In Ollama the same ideas are environment variables:

OLLAMA_NUM_PARALLEL=4        # concurrent requests per loaded model
OLLAMA_MAX_LOADED_MODELS=2   # how many different models stay in memory
OLLAMA_MAX_QUEUE=512         # how many waiting requests before it returns an error
OLLAMA_CONTEXT_LENGTH=8192   # default context, if your version supports it

Defaults and availability change between versions, so check the docs for yours.

#Choosing the number of slots

#Continuous batching

When two requests are generating, the GPU does not run them one after the other. At every step it decodes one token for each active slot in a single batch. Because decode is limited by reading the weights, reading them once to serve four slots costs almost the same as serving one: that is why total throughput rises with slots while each stream slows only a little.

llama-server enables continuous batching by default; the flag exists to turn it off, which you almost never want. You can see it working by running a load test and watching the per-request speed fall slowly while the total rises (load testing).

The honest trade: more parallelism raises total tokens per second and lowers each user's speed. Pick the slot count for the number of people who actually use it at the same time.

#Keep-alive: avoid the reload

A model that is not in memory has to be read from disk. For a 20 GB model that is several seconds, and the user sees it as a slow first reply.

• Ollama unloads a model after a few minutes of idle time by default. Set OLLAMA_KEEP_ALIVE=1h (or -1 to keep it forever), or send keep_alive in a request. keep_alive: 0 unloads immediately.
• llama-server keeps the model loaded for as long as it runs. If you launch it on demand, expect a load time on every start.
• With several models, OLLAMA_MAX_LOADED_MODELS decides how many stay in memory at once. If your VRAM cannot hold two, the second request makes the first unload: a model-swap storm that looks like random slowness.

Tip: if your tool sends a tiny warm-up request on start, the first real request is fast.

#Queues and timeouts

When all slots are busy, new requests wait. Two settings stop a bad day from becoming an outage:

• A queue limit (Ollama's OLLAMA_MAX_QUEUE) makes the server answer "busy" instead of piling up work.
• A client timeout shorter than the longest reasonable response, so a stuck request does not hold a slot for ever.

Cap max tokens in your client. An agent loop that forgets to stop can occupy a slot for minutes.

#Context shift and overflow

When a conversation fills its slot, something has to give:

• Context shift drops the oldest tokens (while keeping a prefix you choose) and carries on. llama.cpp has options for it (see --help for the names in your build, such as the keep and context-shift flags). It prevents a hard error but can silently drop your system prompt or early instructions.
• Stop with an error is safer for applications that cannot tolerate lost instructions.
• Summarise and restart (your app does it) keeps the meaning and frees the cache (handling overflow).

If your system prompt matters, make sure it is in the part you keep, or avoid shifting at all and manage history in your application.

#Prompt reuse across requests

If many requests start with the same system prompt or document, keep that prefix cached instead of re-processing it each time: llama-server has prompt-cache reuse options, and prefix caching explains the effect. With slots, the server tries to give a returning conversation the same slot so its cache is still there. Constant slot churn (many different users bouncing between few slots) throws that benefit away.

#A starting configuration

For a small team on one 24 GB GPU running an 8B model:

llama-server -m MODEL.gguf -ngl 99 -fa on \
  -c 32768 -np 4 -b 2048 -ub 512 \
  --host 127.0.0.1 --port 8080

Then measure (the lab): total tokens per second with 1, 2 and 4 concurrent clients, time to first token under load, and whether any conversation hits its 8K limit. Adjust one thing at a time. Flag spellings differ by version (for example, flash attention takes -fa on in newer builds and plain -fa in older ones), so check --help.

#Common mistakes

• Raising slots without raising context, then wondering why long chats get cut.
• Leaving defaults on a shared machine, so a single agent loop starves everyone.
• Binding to 0.0.0.0 without a proxy or authentication (serving basics).
• Two models, one GPU. The server swaps them in and out and every request pays a load.
• No max-tokens cap. One runaway response holds a slot.

#Exercise

A 12 GB GPU runs an 8B Q4 model (4.9 GB) with a KV cache of about 0.125 MiB per token in f16. You want four slots with at least 4096 tokens each.

1. What total context do you set?
2. How much cache memory does it use, and does it fit?

Answers: -c 16384 -np 4, since 4 × 4096 = 16384. Cache is 16384 × 0.125 MiB = 2048 MiB = 2 GiB. With weights 4.9 GB and some overhead, about 7.5 GiB in total, so it fits in 12 GB with room to spare. You could raise the context to give each slot 8K.