Small models: what 0.5B-4B can really do

Models in the 0.5 to 4B range (Llama 3.2 1B and 3B, Qwen2.5 0.5 to 3B, Gemma 2B-class, SmolLM2, Phi mini models) run on phones, Raspberry-Pi-class boards, laptops without GPUs and in browsers. At Q4 they take only 0.4 to 2.5 GB. They will not replace a frontier model, but for the right task they are fast, cheap, private and good enough, which makes them one of the most useful tools in the box.

#How small is small?

#Where they shine

• Classification, routing, tagging: "is this email spam?", "which department?", "what language?".
• Extraction into a schema, particularly with constrained decoding, which removes formatting failures.
• Summarising short text, rewriting, autocomplete and grammar fixes.
• On-device privacy: nothing leaves the phone or browser (phones and NPUs).
• Speculative-decoding drafters for larger models (speculative decoding).
• Embeddings and rerankers: small specialised models are the backbone of RAG (local RAG).
• Fine-tuning targets: a small model fine-tuned on a narrow task can beat a much larger general model on that task, at a fraction of the cost.
• Latency-critical steps: a 1B model answers in tens of milliseconds, which makes it usable inside interactive features.

#Where they struggle

• Multi-step reasoning, maths and long-context recall. Errors compound across steps (agents).
• Rare or specific knowledge: with few parameters they store little, so they hallucinate facts confidently.
• Complex instructions with many constraints, and many-tool agents.
• Robustness: small changes in prompt wording can change answers more than with larger models.
• Non-English and low-resource languages, which are usually weaker.
• Code beyond simple snippets.

A reliable rule: if a competent person could do the task from the text in front of them in a few seconds, a small model might. If it needs broad knowledge or careful multi-step thought, use a bigger one.

#Speed expectations

Because decode is bandwidth-bound (memory bandwidth), a 1B Q4 model (about 0.8 GB) is about 6 times faster than an 8B (about 4.9 GB) on the same device: often 100+ tokens/s on a modern laptop and 15 to 40 on a recent phone, though these vary widely with chip, runtime and thermals. Prompt processing is also fast, and the KV cache is small.

That speed unlocks designs a larger model cannot afford: run it on every keystroke, on every incoming message, or on ten items in parallel.

#Picking one

1. Define the task and write 20 real examples with the expected output. Small models are very task-dependent.
2. Try 2 to 3 sizes at Q4_K_M (or Q6 for the smallest); stop at the smallest that passes your bar.
3. Try 2 to 3 families. Differences between families are often larger than differences between sizes.
4. Check the licence and the context you actually need (licences).
5. Test on-device if that is your target, warm and on battery.
6. If quality is just short, try these before a bigger model:
   • a better prompt with 2 to 5 examples,
   • a higher quant (small models lose more from quantisation than big ones),
   • constrained output so format errors vanish,
   • splitting the task into simpler steps,
   • retrieval for the facts,
   • fine-tuning on a few hundred examples.

#Prompting small models

• Be explicit and short. One instruction per step; avoid long rule lists.
• Show examples (few-shot): two or three input-output pairs often matter more than more instructions.
• Specify the output format exactly, and enforce it with a schema or grammar.
• Put the instruction last, close to where the model begins answering.
• Use low temperature (0 to 0.3) for classification and extraction.
• Keep the context small: small models are distracted by irrelevant text.
• Check the chat template. A mismatched template hurts small models more than large ones.

#Fine-tuning for a narrow task

Fine-tuning teaches style, format and task behaviour, not new facts. With a few hundred to a few thousand good examples, LoRA fine-tuning of a 1 to 3B model on a single GPU or a Mac can turn a mediocre generalist into a precise specialist (MLX LoRA). Keep a held-out test set, and compare against the base model with a good prompt, since the base with few-shot examples is the real baseline (measuring quality). Be careful with training data that contains personal information (data leakage).

#Using small models next to big ones

Often the best system uses both:

• Router: a small model classifies the request and sends easy ones to itself and hard ones to a larger model (gateways).
• Pre- and post-processing: a small model extracts fields or cleans text before an expensive model reasons about it.
• Guard: a small classifier screens inputs or outputs for safety or scope (red-team lab).
• Draft model: proposes tokens that a big model verifies, speeding it up.
• Cascade: try the small model first, check confidence or a validator, and escalate on failure.

#Where to run them

#Common mistakes

1. Testing on three prompts and declaring a small model "good" or "bad".
2. Expecting factual recall without retrieval.
3. Over-quantising a tiny model.
4. Long, complicated prompts that confuse it.
5. Ignoring the chat template.
6. Skipping a stronger baseline (a bigger model) so you never know what you are giving up.

#Exercise

You need an on-device feature that tags each incoming support message as one of six categories and extracts an order number if present.

1. Which model size would you try first, and how would you guarantee valid output?
2. How would you decide it is good enough?

Answers:

1. Start with a 1B model at Q6 or Q4 and, if needed, a 3B. Use a JSON schema with an enum for the category and an optional order-number string, with a constrained decoder, so output is always valid.
2. Build 100 real labelled messages (including tricky ones), measure accuracy per category and order-number exact match, and compare 1B against 3B. Pick the smallest meeting your target, and check speed and battery on a mid-range phone (evaluation harness).