Retrieval or fine-tuning

The question, stated precisely

You want a model to produce outputs it does not currently produce. Two mechanisms are available, and they act on different things:

• Retrieval changes what the model has in front of it at query time. The weights are untouched. Facts are supplied as input tokens, fetched per request from an index you control.
• Fine-tuning changes the weights. It shifts behaviour (output format, register, domain vocabulary, refusal patterns, how a task is decomposed) by showing the model many examples of the behaviour you want.

Stated that way the decision is often already made, because most requirements are clearly one or the other. The reason it feels hard is that both can raise a score on a superficial evaluation, so a narrow test does not distinguish them.

Three things are not being decided here.

Not prompting. A long system prompt with a few examples is the cheapest option and frequently sufficient. Neither mechanism below should be considered until prompting has been tried and its ceiling identified, because both are ways of buying something prompting cannot give you.

Not the cost model. The two have opposite cost shapes. Retrieval is variable cost per query; fine-tuning is fixed cost per training run plus, on some platforms, an inference premium. That arithmetic is worked through in cost of retrieval versus fine-tuning. This page is about which mechanism answers your requirement; that page is about what each one costs once you know.

Not whether to pick only one. They compose, and composing them is often correct. The decision path below ends there on purpose.

The factors that genuinely decide it

1. Is the requirement knowledge or behaviour?

This is the first question and it settles most cases. Ask what a reviewer would point at if the output were wrong.

The failure to separate these is what produces the fine-tune-on-facts mistake. Facts encoded in weights cannot be cited, cannot be corrected individually, and cannot be shown to have changed.

2. How often does the knowledge change?

If the requirement is knowledge, churn decides the mechanism almost on its own, because the two absorb change completely differently. A retrieval index absorbs a policy change by re-embedding the changed documents: minutes, scoped to what changed. A fine-tuned model absorbs it by re-training, re-evaluating and re-deploying: a full cycle, per change, triggered by your business rather than by your roadmap.

The useful threshold is not a frequency but a comparison: if the knowledge changes more often than you are willing to run a training run and a full evaluation, fine-tuning cannot hold it. For most organisational knowledge (prices, policies, product catalogues, personnel, contracts), that condition is met immediately.

3. Do you need attribution?

If an answer must be traceable to a source document, retrieval is not merely preferable, it is the only mechanism that provides it. A fine-tuned model cannot show where a claim came from, because the claim is distributed across weights. Provenance was one of the explicitly stated motivations for retrieval-augmented generation in the original paper, which named providing provenance for decisions and updating world knowledge as open problems for parametric models (Lewis et al., 2020).

Attribution is often a hard requirement without being written down as one. It appears as “the reviewer needs to check it”, “we have to show the customer the clause”, or a regulatory obligation to evidence a decision. Any of those settles the decision.

4. Query volume

Volume does not decide which mechanism works; it decides whether the working one is affordable and fast enough, and it is the one factor that can override the answer from factors 1 to 3.

Retrieval’s cost and latency are per query. High volume multiplies both, and the dominant term is not the vector search but the retrieved chunks arriving as input tokens on every generation call. Fine-tuning’s training cost is unaffected by volume, and if it lets you delete a long few-shot prompt it removes input tokens from every future request permanently.

So at very high volume with genuinely static knowledge, the arithmetic can favour moving knowledge into weights despite everything above. That is one cell of the space, not the general case, and it is worked out with numbers on the economics page. At high volume the more useful lever is usually to retrieve fewer, better chunks: reranking, tighter chunking, a lower k. That reduces the dominant cost term without changing mechanism.

The differences that do not decide it

• Which one scores better on a benchmark. They are answering different questions, so a single aggregate score compares them on the dimension where they overlap and hides the one where they do not. Build an evaluation that separates factual accuracy from format and register conformance, and the comparison stops being confusing.
• Which one is more fashionable. Both have been through a hype cycle. Neither position tells you what your requirement is.
• Model size. Reaching for a larger base model is a third option, not a way of choosing between these two.
• Whether the data is “enough” for fine-tuning. A real constraint, but it comes after the decision. If the requirement is knowledge, no quantity of training data makes fine-tuning the right mechanism.
• Context window size. A large window reduces the pressure on retrieval precision. It does not remove the need for retrieval, because you still have to choose what goes in the window, and filling it indiscriminately costs input tokens on every request and degrades precision.

A decision path

1. If prompting with a small number of examples meets the requirement, do that. Neither mechanism is justified yet.
2. If the requirement is knowledge that changes, or must be attributable, or both, use retrieval. This covers most enterprise cases. See retrieval-augmented generation for the mechanism and the RAG pipeline for the components.
3. If the requirement is behaviour (format, register, vocabulary, task decomposition) and prompting has hit its ceiling, fine-tune. Build the evaluation harness first, because a fine-tune’s failure mode is a quality shift with no error attached, and without a harness you cannot tell whether a re-tune regressed.
4. If the requirement is both, do both, in that order: retrieval for the facts because they churn and need citing, fine-tuning for the form because it is stable and expensive to specify in a prompt. This is the common end state for a mature system, and the costs compose rather than compound.
5. If the requirement is knowledge, the knowledge is genuinely static, volume is high, and the knowledge does not need attribution, model both options with real numbers before choosing. All four conditions must hold. If any one fails, return to step 2.

Note what step 3 implies about ordering when you are doing both. Fine-tune on examples that already contain retrieved context, so the model learns to use retrieved passages instead of answering from memory. Fine-tuning first and adding retrieval later produces a model that has been trained to trust its weights over its input, which is the opposite of what you want.

What would change the answer

• Churn appears where you assumed there was none. The static-knowledge case is the fragile one. A single regulatory change or product launch can convert it into the churning case, and if the knowledge is in weights the correction is a training run. Re-examine the assumption whenever the domain changes, not on a schedule.
• Attribution becomes a requirement. A new compliance obligation, an audit finding, or a customer asking to see the source moves the answer to retrieval regardless of the previous arithmetic. There is no retrofit that gives a fine-tuned model provenance.
• Volume grows by an order of magnitude. Revisit the cost model, but exhaust the retrieval levers (reranking, chunk size, k, prompt caching placement) before revisiting the mechanism.
• The platform changes under you. A fine-tune is anchored to a base model version; when the base is retired you re-tune, and providers do retire and restrict fine-tuning platforms. That is platform risk you take on with the weights and not with the index. The economics page documents a current example.
• Retrieval quality plateaus below the requirement. If recall is the binding constraint, the fix is usually in the retrieval stack (chunking, embedding model, hybrid keyword-plus-vector search, reranking) and not a switch to fine-tuning. Only if the failure is that the model cannot use well-retrieved context does tuning become the answer, and then you are tuning behaviour, which is consistent with factor 1.
• The base model improves. A stronger base model can remove the need for a behaviour fine-tune entirely. This is the most common reason a fine-tune becomes dead weight, and it is worth re-testing the prompted baseline against a new base before re-tuning.

Common wrong reasons

Fine-tuning to “teach the model our data”. This is the central error. It produces a model that is confidently out of date, cannot cite a source, and requires a full training run to correct a single changed sentence - fixed cost, no auditability, and a re-training obligation driven by external events. If the phrase “teach it our documents” appears in the requirement, the mechanism is retrieval.

Fine-tuning because retrieval “adds latency and complexity”. Both objections are true and neither is decisive. Retrieval adds a hop and an index to operate; fine-tuning adds a training pipeline, an evaluation harness and a version matrix, which is more operational surface, just less visible at request time. Compare the two burdens, not just the one in the request path.

Choosing retrieval for a formatting problem. If the complaint is that the JSON is the wrong shape or the tone is wrong, no amount of retrieved context fixes it reliably. Retrieving more only adds tokens. Use structured output constraints, or tune.

Choosing retrieval because it is the default answer. Retrieval has a real failure mode: wrong or missing chunks, and recall regressions that are silent because nothing errors. If you adopt it you own an index that must be monitored, backed up and re-embedded whenever the embedding model changes. Adopt it for a reason, and instrument recall.

Fine-tuning without an evaluation harness because the first result looked good. A fine-tune has no error signal. It shifts a distribution, and the shift includes behaviour you did not intend to change. Without a harness you find that out from users. Building it is a project and it is not optional.

Treating the choice as permanent. Both are reversible at different costs. A retrieval index can be re-embedded or replaced; a fine-tuned model can be retired in favour of a prompt on a newer base. Deciding once and never revisiting is a larger error than choosing the second-best mechanism now.

Related

• Retrieval-augmented generation - what retrieval is and how it fails
• RAG pipeline - the components you take on with retrieval
• Cost of retrieval versus fine-tuning - the arithmetic
• “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” - what it actually said - the paper the term comes from
• pgvector - vector search in a database you may already run