Context Window Economics

A context window is usually presented as a capability: this model can read a million tokens. It is more useful to read it as a budget line. Every token you place in the context is billed on the request that carries it, and the context is rebuilt on every request because the API is stateless. There is no “loading” a document once. You send it again, and you pay again.

That single fact reframes most prompt engineering decisions as cost decisions. A system prompt is not configuration; it is a per-request surcharge. Few-shot examples are not free accuracy; they are input tokens multiplied by your request volume. Conversation history is not memory; it is a monotonically growing bill.

Prices below were observed on 22 August 2026 and each links to its source.

Two cost curves, not one

There are two distinct things that grow as context grows, and conflating them causes bad reasoning.

What you are billed grows linearly in context length. Vendors charge per input token. Doubling the prompt doubles the input charge, subject to the tier boundaries discussed below. That is the curve on your invoice.

What it costs to compute grows faster than linearly for the attention component. In standard scaled dot-product attention, every token attends to every preceding token, so the attention work for a sequence of length n scales with n² while the feed-forward work scales with n. For short sequences the linear term dominates and attention is a rounding error; as n grows the quadratic term takes over. Modern implementations reduce the memory traffic of this computation substantially, but they do not change its asymptotic shape.

The gap between those two curves is absorbed by the vendor at short context lengths and passed back to you at long ones, which is exactly why long-context price tiers exist. If you self-host, you feel the quadratic term directly as reduced throughput instead of a higher rate. The same context growth that raises your API bill linearly lowers your GPU’s tokens per second more than linearly. That is a utilisation problem wearing a cost mask.

There is a third scaling effect that matters more in practice than the attention arithmetic: KV cache memory. Each token in the context contributes key and value tensors that must be held in GPU memory for the whole generation, and that memory grows linearly in context length per concurrent request. Longer contexts therefore reduce how many requests fit concurrently, which reduces batch size, which reduces throughput. The vLLM tuning documentation states this trade directly, recommending you decrease max_num_seqs or max_num_batched_tokens to reduce KV cache pressure when requests are being preempted. That is, serve fewer requests at once so the long ones fit.

Long-context price tiers

Some vendors charge a higher per-token rate once a request crosses a context threshold. This is not a surcharge on the tokens above the line; on the pages below it is a different rate for the whole request.

Two design consequences follow.

First, where a threshold exists, a request that drifts across it does not get slightly more expensive; the rate for the whole request changes. On the Vertex AI page the input rate for Gemini 3.1 Pro Preview above 200K input tokens is double the rate below it, and the output rate rises as well even though the output length has not changed. A feature whose typical request sits just under a threshold has a cost distribution with a cliff in it, and the mean will not tell you that.

Second, the structure is vendor-specific and it moves. Anthropic explicitly documents on its pricing page that Claude 4.6 and later include the full 1M-token window at standard pricing, with prompt caching and batch discounts applying at standard rates across the whole window. That is a materially different cost profile for a long-context workload than a tiered model, and it is the kind of thing worth re-checking on a 90-day cycle instead of assuming.

The system prompt is paid on every call

This is the most common avoidable cost in a production LLM feature, and the arithmetic is trivial:

monthly_system_prompt_cost =
    system_prompt_tokens × requests_per_month / 1e6 × input_rate

Nothing about that expression depends on whether the system prompt was useful on any given request. A tool schema you include unconditionally is billed on requests that use no tools. Few-shot examples for edge cases are billed on the requests that hit the common case. A style guide is billed on the request where the user asked for a single number.

The mitigations, in rough order of effectiveness:

• Cache the prefix. Covered below; this is the intended answer and it is well supported.
• Make the prompt conditional. Send the tool schema only when tools are plausible; send the formatting examples only for the request types that need them. This fragments your cacheable prefix, so it trades against the previous point rather than composing with it.
• Route to a cheaper model for the requests that do not need the long prompt. OpenAI lists gpt-5-nano at $0.05 per million input tokens against gpt-5.1 at $1.25 (OpenAI pricing, observed 22 August 2026), so classification and routing work can be an order of magnitude cheaper per input token than the model that handles the hard cases.
• Shorten it. Prompts accumulate instructions the way configuration files accumulate flags. Most long system prompts contain instructions added to fix a bug that no longer reproduces. Nobody removes them because nobody measures them.

Instrument the system prompt as a distinct token count in your telemetry, separate from user content and retrieved content. Otherwise it is invisible, and invisible costs do not get optimised.

Prompt caching: what it does and does not buy

Caching stores the model’s computed state for a prefix so that a later request sharing that prefix can skip the prefill for it. Vendors expose this as a cheaper input rate, sometimes with a write premium.

Three properties of these schemes decide whether caching helps you.

There is a break-even in reads, not in time. Anthropic states it explicitly on its pricing page: with a write at 1.25× and a read at 0.1× of base input, the five-minute cache pays off after one cache read, and the one-hour cache (written at 2×) pays off after two. Below that threshold you have paid a premium for nothing.

Google’s model charges for storage as well as reads. The Vertex AI page prices explicit context cache storage per token per hour, which means an idle cache accrues cost with no requests against it. That changes the calculus for a cache you populate speculatively.

The cache is a prefix cache, so ordering is load-bearing. Cache reuse works from the start of the prompt up to the first difference. Put the volatile content after the stable content: the user’s message, the current timestamp, retrieved chunks that vary per query. A single dynamic token near the top of the prompt invalidates everything after it. The vLLM automatic prefix caching design describes the same block-level prefix matching for self-hosted serving, so the discipline transfers: stable prefix first, variable content last, and do not interleave.

That last point conflicts with a common instinct to interpolate the current date, the user’s name or a session identifier into the system prompt. Doing so gives every user a private cache entry and, if the value changes per request, no cache reuse at all.

Conversation history compounds

In a multi-turn conversation, each turn re-sends every previous turn. If turn k adds t tokens of user message and r tokens of model response, the input tokens billed on turn n are the sum of everything before it. Total input tokens across a conversation of n turns therefore grow with the square of the number of turns for a constant per-turn size, not linearly.

That is the compounding effect, and it is why cost per conversation is a much better metric than cost per request:

# constant s tokens added per turn
input_tokens_on_turn_n  = system_prompt + s × (n − 1)
total_input_over_n_turns = n × system_prompt + s × n × (n − 1) / 2

The same effect drives agentic loops, where the “turns” are tool calls rather than human messages and there may be dozens of them per user action. Each tool result is appended and re-sent on every subsequent step. An agent that makes ten tool calls does not cost ten times a single call; it costs substantially more. Prompt caching is what stops it from being ruinous, because the growing prefix is exactly the shape caching is designed for.

Bounding strategies, with their trade-offs stated:

There is no free option. Every one of them either drops information, spends tokens to compress information, or adds infrastructure.

Two things that silently change your token count

Tokenizer changes on model upgrade. Claude 4.7 and later ship a revised tokenizer that inflates token counts by roughly 30% for identical input, though the actual increase varies with the content mix (Claude pricing, observed 22 August 2026). An identical prompt therefore costs more on the newer model at the same headline per-token rate. This is a cost regression with no diff behind it.

Non-text modalities. Images, audio and video are converted into tokens, and the conversion rate is not intuitive. The Vertex AI pricing page prices input as “text, image, video, audio” against a single per-token rate, and its provisioned throughput worked example shows an input audio token burning down at 7 tokens against 1 for an input text token. Multimodal context is expensive per unit of apparent content. A feature that attaches a screenshot to every request has a very different cost profile from one that attaches a paragraph.

The rule this all reduces to

Treat the context window as a budget you must justify per request, not as headroom. For each block you put in the prompt, you should be able to say: how many tokens, on what fraction of requests, and is it in the stable prefix or after it. If you cannot answer those three questions for every block, you do not know what your feature costs.

Related

• Inference cost structure - the underlying meters
• GPU capacity and utilisation - what long context does to throughput
• Build versus buy - where these costs land under each model
• Retrieval-augmented generation - bounding context by retrieving instead of stuffing
• Context engineering - the technique of deciding what goes into the window, as against what it costs