GPU offload, threads and batch sizes

When a model does not fit entirely in GPU memory, runtimes can split it: some layers on the GPU, the rest on the CPU. It works, which is why people use it, but it is a rescue plan with a steep cost. This lesson shows exactly how steep, how to tune what remains, and how the batch settings affect prompt processing.

#Offload layers with -ngl

-ngl N places N transformer layers (and their KV cache) on the GPU. -ngl 99 means "all". Rules:

• Fully on GPU is by far the fastest. Partial offload is a rescue plan: every token still has to pass through the CPU-resident layers at CPU memory speed.
• Offload as many layers as fit with your KV cache. If you hit out-of-memory, lower -ngl or the context.
• llama-server logs offloaded 33/33 layers to GPU: verify it. A log that says 0 layers means a CPU-only build or a missing flag (llama.cpp).

#Why partial offload costs so much

Decode time per token is the time to stream the weights through whichever memory they sit in:

time per token ≈ GPU_bytes / GPU_bandwidth + CPU_bytes / CPU_bandwidth

Worked example. An 8B Q4 model is 4.9 GB in 33 layers, about 0.15 GB per layer. The GPU has 450 GB/s and system RAM has 60 GB/s.

Leaving just 15 percent of the layers (5 of 33) on the CPU halves the speed. Leaving 27 percent cuts it to about a third. The time is additive, and the slow memory has a long time per byte, so a small share dominates. The practical lesson: if the model almost fits, fix the fit (a smaller quant, a shorter context, a quantised KV cache) rather than offloading.

Real numbers are somewhat worse than this idealised model, because layers on different devices also add synchronisation and data-transfer costs.

#When partial offload makes sense

• The model genuinely cannot fit and you accept slow, usable speed (for example a 70B on a 24 GB card for occasional use).
• Batch or offline jobs where speed is secondary.
• Prefill still benefits: prompt processing is compute-bound, so even partial GPU offload speeds up the prompt considerably.

#MoE models

Mixture-of-experts models have huge expert tensors but small per-token compute. Recent llama.cpp builds can keep attention and shared layers on the GPU and the experts on the CPU (see --cpu-moe and --n-cpu-moe, or -ot tensor overrides), which can make very large MoE models usable on modest GPUs: only the few active experts are read from system RAM per token, so the CPU part of the formula stays small. Flag names evolve; check --help.

Worked idea: a 120B-class MoE with about 5B active parameters. The dense parts (attention, embeddings, shared layers) on the GPU, the 100+ GB of experts in 128 GB of system RAM. Each token reads a small slice of experts from RAM, so speeds of tens of tokens per second are possible on a machine where a dense 120B would crawl. You need a lot of RAM, ideally fast and with all channels populated (memory bandwidth).

#Threads

llama-cli -m model.gguf -t 8      # set to PHYSICAL cores; hyperthreads rarely help

For GPU-resident models, threads barely matter: the GPU does the work, and the CPU only feeds it. For CPU inference or partial offload, thread count matters, but only up to the point where memory bandwidth saturates. Too many threads can hurt through contention and scheduling. Hybrid CPUs with performance and efficiency cores often do best with the performance-core count; test a few values:

llama-bench -m model.gguf -ngl 0 -p 128 -n 32 -t 4,6,8,12

#Batch sizes

• -b (logical batch, default 2048): maximum tokens submitted per decode call.
• -ub (physical batch, default 512): the chunk actually processed at once; bigger can speed up prompt processing but uses more memory for compute buffers.

llama-server -m model.gguf -ngl 99 -b 2048 -ub 1024

These control prefill (prompt processing). A 4,000-token prompt is processed in chunks of -ub tokens. Larger chunks keep the GPU busier and amortise overheads, with diminishing returns and higher memory use. Try 512, 1024 and 2048 and look at the pp number in llama-bench.

#Batching is two different things

1. Prompt batching (-b, -ub): how one prompt's tokens are chunked.
2. Request batching (-np slots, continuous batching): how many separate conversations share each decode step, so that the weights are read once for all of them (concurrency).

The second is what raises total throughput for many users; the first only changes prompt-processing speed.

#Memory options

• --mlock asks the OS to keep the weights in RAM (no paging out). Useful on servers with enough RAM.
• --no-mmap loads the whole file into memory instead of memory-mapping it. Slower start, sometimes more predictable.
• Memory-mapping (the default) lets the OS load pages on demand and share them between processes, so a second process loading the same file is nearly free.

#Multiple GPUs

--tensor-split 3,1 puts layers on GPUs in proportion, and -sm layer or -sm row choose the split mode. Layer splitting mostly adds capacity rather than speed: one GPU works while the others wait (buying guide).

#Measure, don't guess

llama-bench -m model.gguf -ngl 0,20,33 -p 512 -n 128

This sweeps GPU layer counts and prints pp/tg speeds for each. Plot tg against -ngl and you will see the curve from the table above on your own hardware. See the benchmark lab.

#Troubleshooting

#Common mistakes

1. Offloading a few layers to CPU "to make it fit" without realising the cost.
2. Using all logical threads on CPU inference.
3. Setting huge batch sizes that waste memory without helping.
4. Confusing prompt batch size with parallel requests.
5. Not reading the load log, so you never notice that the GPU is unused.

#Exercise

A 14B model at Q4_K_M is 8.9 GB in 41 layers. Your GPU has 400 GB/s and 8 GiB free, and system RAM is 70 GB/s. Suppose 30 layers fit on the GPU.

1. How many GB are on each side?
2. What is the speed compared with a hypothetical full fit?

Answers:

1. About 0.217 GB per layer: GPU 30 × 0.217 = 6.5 GB, CPU 11 × 0.217 = 2.4 GB.
2. Offloaded time 6.5/400 + 2.4/70 = 16.3 + 34.3 = 50.6 ms (about 20 tok/s), against a full fit 8.9/400 = 22 ms (about 45 tok/s). The offloaded layers (27 percent of the model) cost more than twice the time. A smaller quant or an 8-bit KV cache would often be a better fix.