The tuning playbook: diagnose first, change one thing, keep a log

You now have many knobs. Without a method, tuning turns into random flag changes that you cannot reproduce. This lesson is the method, with a decision path from symptom to technique. Everything in it links back to a lesson with the detail.

#Step 0: pick the number that matters

"Make it faster" can mean different things. Choose one:

You care about	Metric	Phase
How long until the first word	Time to first token	Prefill
How quickly text streams	Tokens per second per user	Decode
How many users at once	Total tokens per second at acceptable latency	Batching
Whether it runs at all	Fits in memory, loads, stays stable	Memory
Cost and noise	Watts, temperature	Platform

#Step 1: get a baseline

Write down, before you change anything:

Hardware, driver and runtime versions.
The model file, quantisation, context size, slot count and every non-default flag.
Prefill and decode speed at your real prompt length (the lab).
Peak VRAM or RAM, and temperature after a ten-minute run.
A handful of fixed prompts, with their outputs saved, to check quality later.

If you cannot reproduce your baseline, you cannot tune.

#Step 2: find the bottleneck

1
Something feels slow
2
Which phase?first word, or streaming speed, or many users
3
Check the logare all layers on the GPU, is flash attention on
4
Compare with theorybandwidth divided by bytes per token
5
Pick one technique
6
Change one thing
7
Measure againsame benchmark, several runs
8
Keep it, or revert

#Symptom to cause to technique

Symptom	Likely cause	Try	Lesson
Does not load, or out of memory	Weights plus cache exceed memory	Smaller quant, shorter context, quantised KV cache, fewer slots	Quantisation, KV cache
Log shows fewer layers on the GPU than the model has	It does not fit	Make it fit before anything else	Offload
Decode far below the bandwidth estimate	Partial offload, CPU fallback, wrong build, throttling	Check the log, build for your hardware, check clocks	Platform
Decode fine at short context, slow at long	Cache reads grow with length	Flash attention, quantised cache, shorter prompts	Long context
Long wait before the first word	Prefill on a long prompt	Prefix reuse, shorter prompts, batch sizes	Prefill, Caching
Fast alone, slow with several users	Too few slots, or queueing	Slots, context per slot, continuous batching	Server tuning
First request after a pause is slow	Model unloaded	Keep-alive, warm-up request	Server tuning
Fast at first, slower after minutes	Thermal or power throttling	Cooling, power limit	Platform
Big mixture-of-experts model too large for the GPU	Experts do not fit	Offload experts only, not dense layers	Placement
Two GPUs, no speed-up	Layer split gives capacity, not speed	Expect it, or use tensor parallel with a fast link	Placement
Output is repetitive or text-heavy and decode is the limit	Few tokens per step	Speculative decoding	Speculative decoding

#Step 3: change one thing

One setting, one benchmark. If you must try a combination, test each part alone first, then the combination.

Rules of thumb that save time:

Memory first. Anything that does not fit is slow or broken, so fix fit before speed.
Biggest lever first. Moving from partial to full offload beats any flag. Fixing throttling beats thread counts.
Then the cheap ones: flash attention, batch sizes, threads, keep-alive.
Then the quality-costing ones: lower quant, quantised cache, speculative decoding with a draft. Check quality after each.
Last, hardware changes: more RAM channels, a bigger GPU, a different runtime.

#Step 4: measure properly

Warm up first; the first run includes loading and cache building.
Repeat at least three times and look at the spread. A "gain" smaller than the spread is noise.
Use the same prompts and the same context length every time.
Run long enough to include heating for any change you plan to keep.
Check quality against your saved outputs, or a small evaluation set, especially after quantisation or cache changes.
Measure under the load you expect. A single-user benchmark says little about a team.

#Step 5: keep a tuning log

A plain table is enough, and you will thank yourself in a month:

Date	Change	Metric before	Metric after	Quality	Keep?
14 Nov	`-ub 512` to `1024`	prefill 1800 t/s	2100 t/s	same	yes
14 Nov	KV cache f16 to q8_0	8K ctx, 7.1 GB	7.1 GB at 16K ctx	same on 20 prompts	yes
14 Nov	power limit 300 W to 240 W	decode 82 t/s, 74 °C	79 t/s, 66 °C	same	yes

(The numbers above are an example of the format, not results to expect.)

Record the versions too: a runtime update can change the best settings, and the log tells you when to re-test.

#A default recipe

If you want a short order of operations for a new machine:

Pick a model and quant that fit with headroom (VRAM math, VRAM calculator).
Confirm all layers are offloaded and flash attention is on.
Set the context to what you need, and the slot count to the number of simultaneous users.
Check the machine for throttling on a ten-minute run.
Measure prefill and decode at your real prompt size.
Only then try the extras: prefix reuse, speculative decoding, a quantised cache, a different runtime.
Write down what you chose and why.

#Common mistakes

Tuning without a baseline.
Copying someone else's flags from a forum for different hardware and model.
Trusting a single run.
Optimising the wrong phase (decode when the problem is prefill).
Giving up quality silently. A faster setup that gives worse answers is not an improvement.
Never re-testing after upgrading the runtime or driver.

#Exercise

Your server shows these numbers for an 8B Q4 model (4.9 GB) on a GPU with 450 GB/s of memory bandwidth: decode 31 tokens per second, and the log says "offloaded 24/33 layers to GPU". The context is 16K and nothing else is using the card.

What does theory predict for full offload, and how does it compare?
List the first two changes you would try, in order.

Answers: full offload predicts about 450 / 4.9 ≈ 90 tokens per second (real results are a bit lower), so 31 is about a third of the potential. The cause is that 9 layers sit on the CPU. First, make it fit: quantise the KV cache (q8_0) or shorten the context so all 33 layers fit, then check the log shows 33/33. Second, if it still does not fit, step down one quant level and re-measure speed and quality.