When to fine-tune (and when not to)
A decision framework across prompting, retrieval, fine-tuning and distillation: what each changes, cost and risk, the evidence you need before training, a baseline-first process, and a checklist with worked examples.
14 min·Verified 2026-11+50 XP
Fine-tuning is attractive because it feels like "making the model yours". It is also one of the easier ways to spend weeks on something a better prompt would have solved.
#What each lever changes
| Lever | What it changes | Updates | Typical cost | Best for |
|---|---|---|---|---|
| Prompting (instructions, examples) | Behaviour for one request | Instantly | Tokens | Format, tone, simple tasks |
| Retrieval (RAG) | The facts available in context | Instantly (re-index) | Retrieval infra + longer prompts | Changing, private or long-tail knowledge; citations |
| Tools / structured output | What the system can do/guarantee | With code | Engineering | Actions, exact computation, schema-valid output |
| Fine-tuning (SFT/LoRA) | The model's default behaviour | On retraining | Data + GPU hours + evaluation | Consistent style/format, domain phrasing, specialised tasks, smaller/faster models |
| Preference tuning | Which outputs are preferred | On retraining | Preference data | Quality, safety style, helpfulness trade-offs |
| Distillation | Capability transferred to a small model | On retraining | Teacher outputs + training | Cheaper inference for a narrow task |
| Continued pre-training | Domain language and knowledge | On retraining | Large corpora, large compute | Specialised domains with lots of text |
#Decision framework
text
1. Define the task and a success metric; build an evaluation set (50–500 real examples).
2. Baseline: strong prompt + few-shot + the best model you can afford. Measure.
3. Is the gap about missing KNOWLEDGE? → add retrieval/tools; re-measure.
4. Is the gap about FORMAT/BEHAVIOUR/STYLE? → try structured output, better examples, then consider fine-tuning.
5. Is the problem COST or LATENCY (the big model works)? → distil/fine-tune a small model on the big model's verified outputs.
6. Is the gap about REASONING depth? → inference-time techniques or a stronger model first.
7. Fine-tune only if the expected gain justifies data, training, evaluation and maintenance costs.#Good reasons to fine-tune
- Consistent format or style that prompts achieve only ~90% of the time and where the remaining failures are costly.
- A narrow, high-volume task where a small tuned model matches a large one at a fraction of the cost and latency (classification, extraction, routing, tagging).
- Domain language or jargon the base model handles poorly.
- Prompt length reduction: moving long instructions or many examples into the weights (saves tokens at scale).
- Tone and brand voice at scale.
- Tool-use or schema adherence for a specific API, when models fail often.
- Privacy or deployment constraints requiring a small on-device model.
#Bad reasons
- "We want it to know our documents." (Use retrieval; fine-tuning teaches style, not reliable fact recall.)
- "It makes mistakes." (Diagnose the mistakes first; many are retrieval, prompt or evaluation problems.)
- "We have GPUs."
- "Because competitors do."
- Fine-tuning to fix safety issues without evaluation (can make things worse).
#Costs people forget
- Data: collecting, cleaning, labelling and reviewing examples is usually the largest cost (data preparation).
- Evaluation: you need the evaluation anyway; a fine-tune without one is a gamble (evals).
- Regression risk: fine-tuning can degrade other abilities and safety behaviour.
- Maintenance: every base-model upgrade, schema change or policy change may require retraining and re-evaluation.
- Serving: adapters, versioning and rollouts (serving adapters).
- Licences: check the base model's licence for derivative models and the terms of any teacher model used for data.
#Worked examples
| Situation | Choice | Why |
|---|---|---|
| Support bot over a changing help centre | RAG + prompt | Facts change; need citations |
| Extract 12 fields from invoices at 5M docs/month, a large model works but is expensive | Distil into a small fine-tuned model | Narrow, stable, high volume, easy to verify |
| Marketing copy in a specific brand voice | Few-shot prompt first; LoRA if still inconsistent | Style is learnable from examples |
| Model refuses valid medical-device questions | Prompt/system policy first; evaluate; preference tuning only with safety review | Behaviour issue with safety implications |
| "Teach it our codebase" | Retrieval over code + tools; fine-tune only for house style | Code changes constantly |
#Checklist before training
- Evaluation set and metric defined; baselines measured (prompt, few-shot, RAG, stronger model).
- The failure analysis shows behaviour/format (not knowledge) problems, or a cost/latency target.
- Data source, licence and privacy reviewed.
- A held-out test set that is never trained on.
- A plan for regression checks (general abilities, safety).
- A rollout and rollback plan.
#Exercises
- For your own project, fill the decision framework and decide with evidence.
- A task has 70% accuracy with a good prompt. Error analysis shows 80% of failures are missing facts. What next?
- Estimate the break-even: a fine-tuned 8B replaces a large hosted model costing $4 per million tokens with self-hosting at $0.50 per million tokens, at 2 billion tokens per month. Monthly saving? What else must be counted?
Answers: (2) Add or improve retrieval/tools; fine-tuning will not reliably supply missing facts. (3) Saving 2,000 × (4 − 0.5) = $7,000/month before training, data, evaluation and operations costs, so compute payback including a one-off cost and maintenance.
#Next
Prepare data you can trust: data preparation and synthetic data.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. A model must answer questions about your company's product docs, which change weekly. What is the best first approach?
2. Which problem is fine-tuning especially good at?
3. What should you do before investing in fine-tuning?