How vision-language models work, and what images cost

Vision-language models (VLMs) answer questions about images, read documents and charts, and describe scenes. Architecturally they are a language model with an eye attached.

#The pipeline

image → resize/tile → patches → image encoder (vision transformer) → patch embeddings
      → projector (MLP or cross-attention) → "image tokens" in the LLM's embedding space
text prompt → text tokens ┐
                          ├─ one sequence → decoder LLM → answer tokens
image tokens ─────────────┘

1. Image encoder: usually a Vision Transformer (ViT) trained contrastively (CLIP/SigLIP-style) so its features align with text. The image is cut into square patches (for example 14×14 pixels), each becomes a vector, and a transformer produces contextual patch embeddings.
2. Projector: maps vision features to the language model's width (a small MLP is common; some designs use cross-attention or a resampler that compresses to a fixed number of tokens).
3. Language model: a normal decoder. Image embeddings occupy positions in the context, so every later token can attend to them. Training stages typically: align the projector, then instruction-tune on image-text data.

#Counting image tokens

Patch arithmetic is exact: tokens = (height / patch) × (width / patch).

Design variants: fixed resolution (always resize to e.g. 336; fast but poor for small text), tiling / any-resolution (split a large image into tiles plus a thumbnail; more tokens, better detail), native resolution (the encoder handles variable sizes), and token pooling/merging (compress neighbouring patches to cut tokens, e.g. 2×2 merging gives 4× fewer).

#Serving consequences

Visual tokens are prompt tokens (roofline):

• Prefill compute scales with tokens: a 6,144-token image costs like a 6,144-token prompt, plus the encoder's own forward pass.
• KV cache: those tokens occupy cache for the whole conversation (KV size). For a 131 KB/token model, 6,144 tokens ≈ 0.8 GB per image.
• Encoder cost is paid once per image; cache the embeddings if the same image is reused (caching).
• Decoding speed is unchanged per token.
• Batching: variable image sizes make batches uneven; engines handle encoder and LLM stages separately.
• Pre-processing: decoding and resizing images on the CPU can bottleneck throughput; use a worker pool.
• Pricing: hosted APIs bill image inputs by tile/token count; compute it before choosing resolution.

#What VLMs are good and bad at

Good: describing scenes, reading clear text and UI screenshots, charts at moderate complexity, extracting fields from documents, answering questions grounded in an image.

Weak spots to test for:

• Counting many small objects, precise spatial relations, fine-grained measurements.
• Small or dense text when the resolution is too low.
• Hallucination: confidently describing details that are not present.
• Prompt injection through images: text inside an image can carry instructions (prompt injection).
• Sensitive content and privacy: faces, IDs and personal data in uploads (PII).
• Orientation, rotation, low light, and compression artifacts.

Evaluate on your images with your own metrics: field-level accuracy for documents, human-reviewed samples for descriptions (evals). Compare with a specialised OCR/layout pipeline: for clean documents, OCR + LLM can be cheaper and more accurate than a VLM alone.

#Running a VLM locally

Local runtimes load the language model and a separate vision projector file (often named mmproj in GGUF setups) (runtimes). Memory = LLM + encoder + projector + KV cache including image tokens. Start with small VLMs (2–8B) and a fixed or tiled resolution; measure latency per image.

#Exercises

1. Compute visual tokens for a 1024×768 image at patch 16 and at patch 14 (round down).
2. A VLM with 131 KB/token KV cache gets 5 images at 2,304 tokens each. How much KV memory do the images use?
3. Name two ways to cut visual-token cost with little accuracy loss on documents.
4. Why might an OCR + LLM pipeline beat a VLM on invoices?

Answers: (1) Patch 16: 64×48 = 3,072. Patch 14: 73×54 = 3,942. (2) 5 × 2,304 × 131,072 = 1.51 GB. (3) Crop to the relevant region; render at the lowest resolution that keeps text legible; pool/merge tokens; cache encoder outputs. (4) OCR is exact on printed text, cheap, and lets you validate fields programmatically, while a VLM may hallucinate digits.

#Next

Sound: speech recognition and synthesis.