Inference Cost Structure

Inference is not a licence. It is a metered utility, and the meter runs on every request for as long as the feature exists. A model you evaluated once for a few dollars becomes a line item that scales with your traffic, with the length of your prompts, and with how chatty your users are. Nothing about the initial evaluation tells you what the feature costs in production, because the unit of cost is the token and the number of tokens is a property of your product design, not of the model.

The meter has a shape worth understanding: what is billed, why the two halves of a request are priced differently, which discounted tiers vendors actually publish, and why “cost per request” is the wrong number to put in a spreadsheet.

All prices below were observed on 22 August 2026 and each row links to the vendor page it came from. Pricing in this market changes on the order of weeks, so treat the figures as a snapshot of structure. They are not a quote.

The cost is recurring because the work is recurring

A transformer generates output autoregressively: it processes the whole prompt once (the prefill), then produces one token at a time, each step attending over everything generated so far (the decode). There is no result to amortise. Two identical requests do the same arithmetic twice unless something explicitly caches the intermediate state, and even then the cache is charged for.

That is the whole reason inference behaves unlike traditional software cost. Compilation is paid once per build. Indexing is paid once per document. Inference is paid once per request, and a feature that serves ten thousand requests a day pays it ten thousand times a day.

Self-hosting does not remove the recurrence, it only changes the denominator: you rent GPU time by the hour whether requests arrive or not, so the cost per request becomes a function of utilisation. That trade is the subject of Build versus buy and GPU capacity and utilisation.

Input and output are separate meters

Every major vendor bills the prompt and the completion at different rates, in units of tokens, conventionally quoted per million tokens.

Two things matter more than any individual number in that table.

Output is priced above input, consistently. Across every row above, the output rate is a multiple of the input rate: five times for the three Claude models, eight times for gpt-5.1 and gpt-5-nano, six times for Gemini 3.1 Pro Preview. The mechanical reason is the difference between prefill and decode. Prefill processes the entire prompt in parallel, so it saturates the GPU’s matrix units and the per-token cost is low. Decode produces one token per forward pass over the whole model, is bound by memory bandwidth rather than arithmetic, and cannot be parallelised within a single sequence. The vLLM tuning documentation reflects this asymmetry directly: its scheduler prioritises decode requests and batches all pending decodes before scheduling any prefill, because decode is the scarce resource.

The practical consequence: a feature that reads a long document and answers in one sentence is cheap. A feature that reads one sentence and writes a long document is not. Reasoning models make this sharper still, because internal reasoning tokens are billed as output. Vertex AI labels the relevant line “Text output (response and reasoning)” on its pricing page.

Ratios differ by vendor, so per-token comparison depends on your traffic mix. Comparing two models on input price alone is meaningless unless your output:input ratio is fixed. The right comparison is a blended rate weighted by your own measured token counts:

blended_cost_per_request
  = (input_tokens  × input_rate)
  + (output_tokens × output_rate)

Measure input_tokens and output_tokens from your own traffic. Every vendor returns them in the response usage object; there is no reason to estimate them.

Cached input is a third meter

Repeated prefixes (a system prompt, a tool schema, a document the user is asking successive questions about) can be served from a cache of the model’s intermediate state instead of recomputed. Vendors publish this as a distinct, lower input rate, and in some cases as a separate charge for writing to the cache.

The structural claim, which is what you should design against: a cache read costs a fraction of a fresh input token, and where a write premium exists the cache only pays for itself after enough reads. Anthropic states the break-even explicitly: one cache read for the five-minute duration, two reads for the one-hour duration. Google charges cache storage as a separate per-token hourly rate, which means an idle cache costs money even with no reads at all.

That single sentence is the whole design rule. Caching is worthwhile when the same prefix is read back several times inside the cache lifetime, and is a net loss when a prefix is written once and never reused. The details of prefix stability are in Context window economics.

Batch and latency tiers

Where a vendor can schedule your work for later, it can fill idle capacity, and it prices accordingly.

The pattern across vendors is symmetric: relaxing the latency requirement buys a discount, and tightening it carries a premium. Anthropic notes that batch and caching discounts combine, and that fast-mode pricing is excluded from the Batch API. The tiers are therefore not freely composable, and you have to read the interaction rules, not just the headline rates.

Batch is only available to you if the work is genuinely asynchronous. Nightly enrichment, backfilling classifications, evaluation runs and document summarisation qualify. Anything a user is waiting on does not. This is a product constraint masquerading as a pricing choice: the discount is real, but it is only reachable if you have already designed the feature to tolerate delay.

Two further modifiers appear on the same pages and are easy to miss when modelling cost. Regional or data-residency routing carries a documented premium: a 1.1× multiplier on all token categories for US-only inference on Claude 4.6 and later (Anthropic), and a 10% uplift on OpenAI data residency endpoints for eligible models released on or after 5 March 2026. Tokenisation also changes: Anthropic documents that Claude 4.7 and later use a newer tokenizer producing more tokens for the same text, which raises the cost of an unchanged prompt on a model upgrade. That is a cost regression with no code change behind it, and it will not appear in any of your own metrics except the bill.

Tool calls are separately metered

Vendor-hosted tools are billed on top of tokens. Billing is usually per call, not per token, and the tokens the tool returns are billed at the model’s normal input rate.

The embedding rates are worth noting because they are the cheapest thing on the page and are often where a retrieval pipeline’s costs are assumed to live. They are not. See Cost of retrieval versus fine-tuning.

Why cost per request is the wrong unit

An agentic feature does not make one request. It makes a request, inspects the result, calls a tool, appends the tool output to the conversation and requests again. Each of those steps re-sends the accumulated context as input tokens. A “single user action” can be a dozen billed requests with monotonically growing prompts.

Retries make this worse in a way that is invisible in a per-request average. Consider the failure modes that generate billed work with no user-visible output:

• Validation failure. The model returns malformed JSON, your parser rejects it, you retry. Both attempts are billed in full, including the output tokens of the discarded one.
• Timeout with work completed server-side. Your client gives up; the tokens were still generated.
• Guardrail rejection. A moderation or policy check discards a completion after generation.
• Refusal loops. A model declines, the harness rephrases and asks again.

Cost per successful outcome is the number that matters:

cost_per_outcome = (sum of all billed tokens for the attempt chain) / (successful outcomes)

If one attempt in five fails and is retried once, the token cost per outcome is 20% above the per-request cost, and none of your dashboards will say so unless you attribute token usage to the outcome instead of the call. Instrument at the level of the user-visible action: total input tokens, total output tokens, total attempts, and whether the action succeeded.

One useful billing detail supports this instrumentation. Vertex AI states that you are charged only for requests returning a 200 response code, and that 4xx and 5xx responses are not charged for input or output. That protects you from infrastructure-level failures. It does not protect you from a request that succeeds at the HTTP level and fails at the application level, which is the far more common case and the one that actually inflates cost per outcome.

What to do with this

Four measurements, taken from production traffic and not estimated:

1. Input and output token counts per user-visible action, not per API call.
2. Attempts per successful outcome, including validation and guardrail rejections.
3. Cache hit rate on your repeated prefixes, if you use a cached tier.
4. The fraction of your workload that could tolerate batch latency.

With those four numbers, the vendor pricing pages give you a cost model you can defend. Without them, any figure you produce is a guess about your own product, dressed up in someone else’s published rate.

What this does not price

It describes how vendors meter hosted inference. It does not tell you which model to choose - capability per pound is a quality judgement, not a pricing one, and no table above says anything about output quality. For the decision about whether to buy tokens at all, see Build versus buy. For the broader AI section, see /ai.