Sampling parameters that matter

At each step the model outputs a score for every token in its vocabulary (often 32,000 to 200,000 of them). Sampling is the rule that turns those scores into one chosen token. Different rules give very different behaviour from the same model: focused or varied, precise or imaginative, stable or chaotic. A few settings do most of the work.

#From scores to probabilities

The raw scores are called logits. The softmax function turns them into probabilities that sum to 1. Every sampling setting is either a change to the logits before softmax (temperature, penalties) or a filter on the probabilities after it (top-k, top-p, min-p).

Take a toy vocabulary with logits [2.0, 1.0, 0.1] for tokens A, B and C.

#Temperature

Temperature divides the logits before softmax.

• 0 is greedy decoding (always the most likely token). Use it for repeatable, factual tasks.
• 0.2 to 0.5 is focused with a little variation.
• 0.7 to 1.0 is the natural range for conversation and writing.
• 1.2 and above increasingly produces odd or incoherent text.

Lower temperature does not make a model more correct; it makes it more consistent. A wrong answer at temperature 0 is wrong every time.

#Filters: top-k, top-p and min-p

Filters remove unlikely tokens before sampling, which prevents the occasional absurd pick from the long tail of the vocabulary.

• top-k: sample only among the k most likely tokens (40 is typical). Simple, but ignores how confident the model is: if the model is certain, k=40 still keeps 39 pointless candidates; if it is uncertain, k=40 may cut good ones.
• top-p (nucleus): keep the smallest set of tokens whose probabilities add up to p (0.9 to 0.95 is typical). It adapts: a confident step keeps two tokens, an uncertain step keeps dozens.
• min-p: drop tokens whose probability is below min_p × the top token's probability (0.05 is a popular default). It adapts too, and often behaves better than top-p at higher temperatures, because it scales with the model's confidence.

Worked example. Probabilities after temperature: [0.50, 0.25, 0.12, 0.08, 0.03, 0.02].

• top-k = 3 keeps the first three (renormalised: 57%, 29%, 14%).
• top-p = 0.9 keeps the smallest prefix summing to at least 0.9: 0.50 + 0.25 + 0.12 + 0.08 = 0.95, so four tokens.
• min-p = 0.1 has threshold 0.1 × 0.50 = 0.05, so it keeps tokens above 0.05: the first four.

If the top token were 0.95 instead, min-p 0.1 would drop everything under 0.095: the model's confidence tightens the filter automatically.

#Order of operations

Runtimes apply samplers in a defined order (it differs by runtime and is often configurable). A common llama.cpp order: penalties, then top-k, top-p, min-p, then temperature, then pick. The details matter at the margins: temperature applied before or after the filters changes what the filters see. Use the runtime's defaults unless you have a reason, and check its documentation if you combine many samplers.

#Repetition and frequency penalties

• repeat penalty (llama.cpp, typical 1.05 to 1.15): reduces the probability of tokens that appeared recently.
• frequency and presence penalties (OpenAI-style APIs, typically 0 to 1): penalise tokens in proportion to how often (frequency) or whether (presence) they have appeared.

Keep them mild. Strong penalties damage coherence, because legitimate repetition is normal (names, code identifiers, the word "the"). They also hurt tasks like extraction where the same token must recur.

#Other controls

• max tokens: a hard cap on the reply length. Set it deliberately: too small truncates answers mid-sentence, unlimited can ramble.
• stop sequences: strings that end generation (useful when the model should not continue into a made-up "User:" turn).
• seed: fix it for reproducible output with the same settings and the same runtime version. Different hardware or runtime versions can still produce different text.
• logit bias: boost or ban specific tokens (for example to forbid a word).
• mirostat (llama.cpp): an adaptive method that targets a given perplexity; rarely needed with min-p available.
• grammar and JSON-schema constraints: remove invalid tokens entirely, which is stronger than any sampling setting for structured output (structured output).

#Starting points

#Practical advice

• Use the model card's recommended settings. Reasoning and instruction-tuned models are often tuned for specific values (some explicitly recommend against greedy decoding).
• Change one knob at a time and compare on the same prompts. Temperature changes everything at once.
• For quality comparisons, use temperature 0 so differences come from the model, not luck (measuring quality).
• Test several samples, not one: a single output at temperature 0.8 tells you little.
• Loops ("the the the") usually mean temperature is too low, the context is truncated, or the chat template is wrong, not that you need a bigger penalty. See the fix.
• Gibberish at the end of long outputs often means the context window overflowed.

# llama.cpp example
llama-cli -m model.gguf --temp 0.7 --top-p 0.9 --min-p 0.05 --repeat-penalty 1.1 --seed 42

# OpenAI-compatible request (llama-server, vLLM, Ollama's /v1)
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "local", "temperature": 0.7, "top_p": 0.9, "max_tokens": 300, "seed": 42,
  "messages": [{"role": "user", "content": "Give me three taglines for a bakery."}]}'

Not every runtime accepts every parameter through the OpenAI-style endpoint (min-p and repeat penalty are often runtime-specific extras), so check the documentation of the one you use.

#Debugging with sampling

#Exercise

You are building a feature that classifies support tickets into five labels and writes a one-sentence summary.

1. What settings would you use for the classification step and the summary step?
2. How would you guarantee a valid label?

Answers:

1. Classification: temperature 0, with a constrained output limited to the five labels. Summary: temperature around 0.3, top-p 0.95, a stop sequence and a modest max_tokens, since you want faithful, consistent wording rather than creativity.
2. Use a grammar or JSON-schema enum so the model can only emit one of the five labels (structured output), and still validate the value in code.