How to choose between RAG and fine-tuning

Start with what you need to change

Retrieval-augmented generation (RAG) and fine-tuning are often presented as rivals. They change different things. RAG changes what the model can see at the moment it answers. Fine-tuning changes how the model behaves by adjusting its weights. Most confusion comes from using one to solve the other's problem.

If the model gives wrong answers because it does not have the facts (last month's policy, your product catalogue, a customer's record), no amount of fine-tuning will reliably fix it. Facts go stale, and weights are a poor place to store them. If the model has the facts but writes in the wrong tone, ignores your output format or fumbles a narrow task, retrieval will not help. That is a behaviour problem.

Five questions that usually decide it

1. Do the facts change? If the answer depends on documents that are updated weekly, use RAG. You update the index, not the model.
2. Must answers be traceable? RAG can show which passages an answer used. A fine-tuned model cannot point to where it "learned" something.
3. Is the problem style, format or a narrow skill? Consistent JSON, a house tone, a classification label: fine-tuning is a good fit.
4. Is the data private and access-controlled? RAG lets you filter by permission at query time. Baking restricted data into weights is hard to undo.
5. What are latency and cost limits? Retrieval adds a step and more prompt tokens. A small fine-tuned model can replace a large prompted one for a narrow job.

Try the cheap option first

The order that wastes the least time is usually:

• Prompting with clear instructions and a few examples.
• RAG, if the gap is knowledge.
• Fine-tuning, if the gap is behaviour and you have enough good examples.

Each step costs more to build and to change. Move on only when you can show, on a test set, that the previous step is not enough.

They combine well

The choice is not exclusive. A published Microsoft Research study on agricultural question answering found that fine-tuning and retrieval each improved accuracy and that the gains added up when the two were combined. A common real pattern is to fine-tune for domain vocabulary and answer format, then retrieve the facts.

A short test before you commit

Build a set of 50 to 100 real questions with agreed good answers. Run the same set through each approach and score them the same way:

• Did the answer contain the right facts?
• Was it in the right format and tone?
• Was it grounded in a source?
• What did it cost and how long did it take?

Small differences on a small set are noise, so look for clear gaps. If two options are close, pick the one that is cheaper to update.

Common mistakes

• Fine-tuning to add facts. The model may repeat them confidently and still get details wrong.
• RAG without evaluation. Poor chunking or ranking looks like a model problem. Measure retrieval separately from generation.
• Too little tuning data. A few dozen inconsistent examples teach inconsistency.
• Skipping the baseline. Without a prompted baseline you cannot tell whether either approach helped.

Quick summary