RAG, fine-tuning or prompting? A decision guide

When a language model does not do what you need, three tools are on the table: change the prompt, add retrieval (RAG), or fine-tune the model. Teams often jump to the most expensive one first. A short decision process saves weeks.

What each tool actually changes

• Prompting changes what you ask. Instructions, examples, a required output format. No training, instant to try, easy to undo.
• Retrieval-augmented generation (RAG) changes what the model knows at answer time. You fetch relevant passages from your own documents and place them in the prompt, so the answer can be grounded in current, private information.
• Fine-tuning changes how the model behaves. You continue training on examples so it adopts a style, format or skill more reliably, without needing a long prompt each time.

The most useful sentence in this whole topic: RAG adds knowledge; fine-tuning shapes behaviour. Mixing them up is the commonest expensive mistake.

A decision path you can follow

1. Start with a clear prompt and a handful of examples. Many "the model can't do this" problems are really "the model was not told clearly". Add a system instruction, two or three worked examples, and a defined output format.
2. If the answer needs facts the model cannot have (your policies, yesterday's data, private documents), add retrieval. No amount of prompting can make a model recall a document it never saw.
3. If the model has the knowledge but responds in the wrong shape, tone or format, reliably, and prompting cannot fix it, consider fine-tuning.
4. If you need both (private facts and a consistent house style), combine them: retrieval for the facts, fine-tuning or a strong prompt for the style.

Quick comparison

Examples to calibrate your instinct

• "Answer questions from our 300-page employee handbook." Facts, private, changing: RAG.
• "Always reply as a JSON object with these five fields." Format problem: prompting with a schema and examples, then validate the output in code. Fine-tune only if that stays unreliable at scale.
• "Write in our brand voice across thousands of emails." Consistent style: try a good prompt with examples first; fine-tuning if the prompt becomes long and still drifts.
• "Classify support tickets into 12 categories." Try few-shot prompting; consider fine-tuning a smaller model once you have enough labelled examples and volume.

Why fine-tuning is not a way to add knowledge

It is tempting to "teach the model our documents" by fine-tuning on them. Models tend to pick up style and patterns from that, and recall specific facts unreliably. They also cannot tell you where an answer came from, and every change to the documents means another training run. Retrieval keeps facts in a place you can inspect, update and cite.

What it costs to be wrong

• Prompting is nearly free to try. Iterate.
• RAG needs a pipeline (chunking, embeddings, an index, evaluation) but it is ordinary engineering.
• Fine-tuning needs good, consistent training data, an evaluation set, and a plan to redo it when the base model or your needs change. Do it when there is a measured gap that cheaper tools failed to close.

Common mistakes

1. Fine-tuning to fix a problem that a clearer prompt would have solved.
2. Building RAG when the "documents" would fit in the prompt. A short, stable reference can simply be included.
3. Skipping evaluation: no fixed set of test questions, so nobody can tell whether a change helped.
4. Treating the three as exclusive. Production systems commonly use a good prompt and retrieval, and sometimes a fine-tuned model as well.

A checklist

• Have we written down the failure we are trying to fix, with three real examples?
• Did a better prompt and few-shot examples fail first?
• Does the task need knowledge the model cannot have?
• Is the problem behaviour (format, tone) rather than knowledge?
• Do we have an evaluation set to prove the change helped?