Images as numbers: pixels, channels and patches

#An image is a grid of numbers

A grayscale image is a grid of brightness values, often 0 to 255. A colour image has three grids (channels): red, green and blue. So a 224 × 224 colour image holds 224 × 224 × 3 = 150,528 numbers.

#Patches

A vision transformer (and most vision-language models) cuts the image into a grid of equal patches and turns each patch into one vector. The sequence of patch vectors is what the language model attends to, alongside text tokens.

H = W = 8
img = [[(r * W + c) % 256 for c in range(W)] for r in range(H)]   # a fake 8x8 grayscale image
P = 4                                                              # patch size
patches = []
for r in range(0, H, P):
    for c in range(0, W, P):
        patch = [img[r + i][c + j] for i in range(P) for j in range(P)]
        patches.append(patch)
print("patches:", len(patches), "each with", len(patches[0]), "numbers")
# patches: 4 each with 16 numbers

for size, p in ((224, 14), (336, 14), (1024, 14)):
    n = (size // p) ** 2
    print(f"{size}x{size} image, {p}x{p} patches -> {n} patches")
# 224x224 image, 14x14 patches -> 256 patches
# 336x336 image, 14x14 patches -> 576 patches
# 1024x1024 image, 14x14 patches -> 5329 patches

Each patch becomes roughly one token before any merging or compression that a particular model applies. So an image costs hundreds to thousands of tokens, depending on resolution and the model.

#What this means for cost

• Context. A few high-resolution images can fill a context window that held a whole document.
• Prefill time. Tokens must be processed before the first output word, so time to first token rises with image size (prefill).
• Memory. The KV cache grows with these extra tokens.
• Price. Hosted APIs bill images in tokens or tiles.

Techniques that bound the cost: resize to the model's native resolution, tile large images into fixed-size crops, crop to the region that matters, and models that merge neighbouring patches into fewer tokens (how vision-language models work).

#Resolution and what the model can see

Resizing a page of small text to 224 pixels wide makes the text unreadable. Models read text in images only if the resolution is high enough for the characters, which is why document pipelines either use a higher-resolution model or run OCR first and feed the text instead. The right choice depends on the task, and testing on your own images beats guessing.

#Exercise

1. How many patches does a 448 × 448 image make with 14 × 14 patches?
2. A pipeline sends 6 pages as 1,024 × 1,024 images. Estimate the image tokens, and say two ways to cut them.

Answers: 1. 448 / 14 = 32, so 32 × 32 = 1,024 patches. 2. About 6 × 5,329 ≈ 32,000 tokens. Cut them by resizing to the model's native size or running OCR on text-heavy pages and sending text, and by sending only the pages or regions that matter.

Next: audio as numbers.