Tokens and context windows

Models do not read characters or words. They read tokens: chunks of text produced by a tokenizer (usually byte-pair encoding, BPE). Common words are a single token; rare words, code, numbers and non-English text split into more. Everything about speed, memory and cost in LLM systems is measured in tokens, so it pays to understand them well.

"Local inference is fun"  ->  ["Local", " inference", " is", " fun"]   (4 tokens)
"Quantization"            ->  ["Quant", "ization"]                    (2 tokens)
"2026-03-03"              ->  ["202", "6", "-", "03", "-", "03"]      (6 tokens; varies by model)

Notice that the leading space belongs to the token ( inference), and that the splits above are illustrative: every tokenizer cuts differently.

#How a tokenizer is built

A BPE tokenizer is trained on a large text sample by repeatedly merging the most frequent adjacent pairs of symbols. It starts from bytes (so any text can be encoded) and builds a vocabulary of typically 32,000 to 200,000 pieces. Frequent English words become single tokens; rare strings fall back to smaller pieces, down to single bytes if needed.

Consequences you will notice:

• The same text has different token counts in different models. A Llama tokenizer, a Qwen tokenizer and a Mistral tokenizer disagree. Always count with the model you actually run.
• Whitespace and capitalisation matter. Hello, hello and HELLO can be different tokens.
• Numbers split awkwardly. Many tokenizers cut numbers into short digit groups, one reason arithmetic on long numbers is error-prone for LLMs.
• Special tokens exist for structure: beginning and end of text, role markers (<|im_start|>user), end-of-turn. They are part of the chat template (chat templates).

#Rules of thumb (English prose)

• 1 token is about 4 characters or 0.75 words.
• 1,000 tokens is roughly 750 words, or 1.5 pages of text.
• A typical email is 150 to 400 tokens; a long article 2,000 to 5,000; a 300-page book around 100,000.

These shift noticeably outside plain English:

This matters for cost and speed: a Hindi or Japanese user may get a smaller effective context window and slower responses from the same model than an English user. Newer tokenizers with larger vocabularies reduce the gap.

#Counting tokens properly

Use the model's own tokenizer:

# llama.cpp server
curl -s localhost:8080/tokenize -d '{"content":"Local inference is fun"}'

# Python, with the Hugging Face tokenizer for the model you run
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
print(len(tok.encode("Local inference is fun")))

Count the whole prompt as the model sees it, including the system message, the chat template's special tokens and every previous turn. The text you typed is only part of it.

#The context window

The context window is the maximum number of tokens the model can attend to at once, and it must hold everything: system prompt, chat history, retrieved documents, your question and the reply being generated.

context_size >= prompt_tokens + max_new_tokens

A worked budget for a 8,192-token window:

When the history keeps growing, something has to give: drop old turns, summarise them, or retrieve less. This is the central design problem in chat and RAG applications (local RAG, agents).

#Why context costs memory

Each token in context stores key and value vectors for every layer: the KV cache. Its size is:

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_value

For Llama 3.1 8B (32 layers, 8 KV heads, 128 dimensions) with 16-bit values:

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token

So an 8,192-token context uses 8,192 × 128 KiB = 1 GiB, and 128K tokens would use 16 GiB, more than three times the 4-bit weights of the model. Larger models cost more per token (a 70B can need 320 KiB per token or more). Concurrent users multiply it.

Longer context also slows decoding, because every new token must read the whole cache, and slows prefill, which grows with prompt length. See VRAM math for the full memory budget, and the Pro lessons on KV-cache quantization and long-context performance for ways to reduce it.

#Chat templates and special tokens

Chat models are trained on text wrapped in a specific format, for example:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello!<|im_end|>
<|im_start|>assistant

This wrapping is the chat template, and different model families use different ones. If you feed a model text in the wrong format, it still produces output, but worse: it may ramble, ignore the system prompt, or never stop. Runtimes normally apply the right template for you (Ollama does automatically; llama.cpp uses the template stored in the model file, and --jinja enables the model's full template). If a model behaves strangely, suspect the template first.

#When you overflow

Runtimes handle a full window differently: some truncate the oldest tokens, some shift the window (discarding old tokens and keeping the rest), and some error out. The dangerous case is silent truncation: the system prompt sits at the start, so it is the first thing lost, and the model appears to "forget the instructions".

Defences:

• Set the context size explicitly. Defaults are often 2,048 or 4,096 even for models that support much more (-c 8192 in llama.cpp, num_ctx in Ollama).
• Measure your prompt and reserve space for the reply.
• Keep the system prompt short and put critical instructions where they will not be truncated; some runtimes can pin the first tokens.
• Summarise history instead of dropping it blindly.
• Log prompt token counts in applications so you notice growth.

Tip: a model that supports 128K tokens does not mean you should run 128K. Memory and speed both degrade with length, and quality often drops for facts buried in the middle ("lost in the middle"). Use the smallest window that fits your task.

#Why long context is not free quality

Even when everything fits:

• Attention dilutes. The model has more to look at, and relevant details can be missed, especially in the middle of a long input.
• Distractors hurt. Stuffing ten loosely related documents in the prompt often gives worse answers than the three best ones (testing RAG).
• Cost and latency grow with every token you add.
• Claimed window is not effective window. Test retrieval of facts at different positions in your actual prompts ("needle in a haystack" style) before relying on a long context.

#Exercise

You run Llama 3.1 8B (128 KiB of KV per token in 16-bit) with a 16,384-token window and expect 4 concurrent users.

1. How much KV memory do you need in total?
2. A teammate asks for a 128K window "just in case". What would that cost for one user, and what is your answer?

Answers:

1. 16,384 × 128 KiB = 2 GiB per user, so 8 GiB for four users (if each fills its window).
2. 131,072 × 128 KiB = 16 GiB for a single user. Unless you have a concrete task that needs that much context, keep the window small, and use retrieval or summarisation for long documents.