Cost of Retrieval Versus Fine-Tuning

Retrieval and fine-tuning are usually presented as alternative answers to the same question: how do I get the model to know about my data. Judged on quality that framing is reasonable, and the usual conclusion is sound: retrieval for facts, fine-tuning for form and behaviour. Judged on cost they are not alternatives at all, because they have opposite shapes.

Retrieval is a variable cost. You pay on every query: an embedding call, a vector search, and the retrieved tokens as extra input on the generation. Volume drives the bill.

Fine-tuning is a fixed cost. You pay per training run, in proportion to dataset size and epochs, and then you pay again every time your data changes enough to invalidate the model. Change frequency drives the bill, and on some platforms the resulting model also costs more per token to serve.

So the deciding variables are query volume and knowledge churn, and the answer flips as either moves. Prices below were observed on 22 August 2026 and each links to its source. This page is analysis: the prices are facts, the framework for combining them is interpretation.

What retrieval costs, item by item

The line that carries the cost is not the one people budget for. Embedding is cheap: at $0.02 per million tokens, embedding a query costs almost nothing, and even a substantial corpus is a one-off in the low tens of dollars. The expensive part of retrieval is the generation it feeds. Retrieved chunks are input tokens on the model call, priced at the model’s input rate: $1.25 per million for gpt-5.1 and $2.00 per million for Claude Sonnet 5 (both observed 22 August 2026). Retrieving eight chunks instead of three triples that component with no change to any retrieval price.

This is the single most useful cost lever in a retrieval pipeline and it is not a pricing decision at all: retrieve fewer, better chunks. Reranking, tighter chunking and a lower k reduce the input tokens on every subsequent query, permanently. See retrieval-augmented generation for the quality side of that trade.

Two further cost characteristics of retrieval that per-token tables miss.

Retrieval defeats prompt caching if you put it in the wrong place. Retrieved chunks vary per query, so they are volatile content. Placed before the stable system prompt they invalidate the cacheable prefix entirely; placed after it they do not. Anthropic prices cache reads at 0.1× the base input rate and 5-minute cache writes at 1.25×, so the difference between correct and incorrect ordering is close to an order of magnitude on the stable portion of the prompt. This is covered in detail in context window economics.

The index is infrastructure you now operate. Whether that is a managed tool priced per call and per GB-day, or pgvector in a PostgreSQL instance you already run, there is an index to build, monitor, back up and re-embed when you change embedding model. The pgvector README documents the storage arithmetic for its vector type as 4 × dimensions + 8 bytes, which makes the storage side small and predictable. Index build time, maintenance_work_mem pressure and recall tuning are the operational work, and recall regressions are silent. If you already run PostgreSQL, pgvector is the option that adds the least new operational surface, which is a stronger argument in a small team than any per-unit price.

What fine-tuning costs, item by item

Four things in that table are worth reading carefully.

Training tokens are not dataset tokens. The billed token count equals the size of the training dataset multiplied by the epoch count, so three epochs triple the bill. Epoch count is a hyperparameter you will change while tuning, so the cost of a tuning programme is a multiple of the cost of one run.

The per-token training rate varies by orders of magnitude across model families. On Vertex AI, observed 22 August 2026, supervised fine-tuning is $0.01 per million training tokens for Gemini 3.5 Flash and $6.72 per million for Llama 3.3 70B. Whether fine-tuning is “cheap” is not a property of fine-tuning; it is a property of which model you tune.

Fine-tuned inference can cost more per token than base inference. Google states that from Gemini 3 onward, tuned model endpoint prediction price is 1.5 times the base model, while older Gemini models keep the base price. OpenAI lists fine-tuned gpt-4.1 at $3.00 input and $12.00 output against $2.00 and $8.00 for base gpt-4.1 (observed 22 August 2026). So fine-tuning is not purely a fixed cost on those platforms. It can raise the variable cost too, which inverts the usual mental model.

Some training is billed by time rather than by token. Reinforcement fine-tuning of o4-mini is listed at $100.00 per hour (observed 22 August 2026), and the same page notes that tokens used for model grading in reinforcement fine-tuning are billed at that model’s per-token rate. An hourly training rate with a token-billed grader has no upper bound you can compute in advance.

The versioning burden, which is the real cost

The published prices above are the easy part. The obligation you take on with a fine-tuned model is that it is now a build artefact with a lifecycle.

• Knowledge changes invalidate it. A retrieval index absorbs a policy change by re-embedding the changed documents. A fine-tuned model absorbs it by re-training. That is a full training run, a fresh evaluation, and a deployment, per change.
• Base model changes invalidate it. Your fine-tune is anchored to a base model version. When that base is retired you re-tune. Anthropic lists several models as retired or retired except on partner platforms, which is the same pattern applied to base models.
• The platform itself may change under you. OpenAI states that it is winding down the fine-tuning platform, that the platform is no longer accessible to new users, that existing users can create training jobs for the coming months, and that fine-tuned models remain available for inference until their base models are deprecated. That is as clear a statement of platform risk as a vendor is likely to publish, and it is the single strongest argument for keeping the knowledge in an index you control instead of in weights held by a provider.
• You need an evaluation harness before you start. Without one you cannot tell whether a re-tune regressed. Building it is a project, and it is not optional, because a fine-tune’s failure mode is a quality shift with no error attached.

None of that appears in a per-token price. In a small team it is usually larger than the compute.

Putting the two side by side

Note the row on latency, because it cuts the other way from the rest. Google lists “lower inference latency and cost due to shorter prompts” among the benefits of tuning. If fine-tuning lets you delete a long few-shot system prompt, it removes input tokens from every future request - a permanent variable-cost saving bought with a fixed cost. That is the one case where the fixed-cost option reduces your marginal cost. Check for it explicitly, because it is the strongest economic argument for fine-tuning and the one most often left unquantified.

A decision rule you can apply

Express both options as monthly cost and compare directly.

retrieval_monthly =
      queries × (query_embedding_tokens / 1e6) × embedding_rate
    + queries × retrieval_call_cost
    + queries × (retrieved_tokens / 1e6) × model_input_rate
    + index_storage_monthly
    + index_operations_monthly

finetune_monthly =
      (dataset_tokens × epochs / 1e6) × training_rate × retrains_per_month
    + queries × (prompt_tokens_saved / 1e6) × model_input_rate × (−1)
    + queries × (input_tokens / 1e6) × (tuned_input_rate − base_input_rate)
    + queries × (output_tokens / 1e6) × (tuned_output_rate − base_output_rate)
    + eval_and_pipeline_monthly

The negative term is deliberate: it is the prompt-shortening saving, and it is the only term that improves as volume grows. Everything else about fine-tuning either stays flat or gets worse.

The structure, not the numbers, gives four regimes:

The high-volume, low-churn cell is the only one where fine-tuning is the cheaper cost shape by a clear margin. Even there the deciding factor is usually the prompt-shortening saving rather than the training price.

They are not mutually exclusive, and that is often the right answer

Fine-tune for form (output structure, domain vocabulary, tone, refusal behaviour) because those are stable and expensive to specify in a prompt. Retrieve for facts, because those change and need to be citable. The costs then compose: you pay the training cost once for behaviour that does not churn, and the per-query retrieval cost for content that does. You also get the prompt-shortening saving on the part you moved into the weights.

The failure mode to avoid is fine-tuning on facts. It produces a model that is confidently out of date, cannot cite a source, and requires a full training run to correct a single changed sentence. That is the worst cost shape available: fixed cost, no auditability, and a re-training obligation triggered by your business, not by your roadmap.

Related

• Retrieval-augmented generation - the retrieval architecture itself
• pgvector - vector search inside PostgreSQL you already operate
• Context window economics - where retrieved tokens land
• Inference cost structure - the meters both options run on
• Build versus buy - whether to own the serving tier at all
• /ai - the wider section