GPU Capacity and Utilisation

A hosted API charges for tokens. A GPU charges for time. That difference is the whole of self-hosted inference economics, because it means your cost per token is not a price you were quoted. It is a quotient you produce:

cost_per_token = gpu_hourly_rate / tokens_produced_per_hour

The numerator is published. The denominator is yours, and it varies by more than an order of magnitude depending on batching, sequence lengths and arrival pattern. Two teams renting identical hardware at identical rates can have wildly different unit costs, and the difference is almost entirely utilisation.

Prices below were observed on 22 August 2026 and each links to its source. This page is analysis: the prices and the documented engine behaviour are facts, the framework is interpretation.

The numerator: what accelerator time costs

Two observations about this table that matter more than the individual figures.

The single-GPU rate is higher per GPU than the eight-GPU rate: $4.29 against $3.99 for H100 SXM (Lambda GPU cloud, observed 22 August 2026). Small deployments pay a premium per unit of capacity, which works against the case for a modest self-hosted footprint.

Regional variation is published and significant. The same p5.48xlarge reservation is listed at $41.528 per hour in US East (N. Virginia) and $37.76 per hour in Europe (London) and Asia Pacific (Tokyo) (EC2 Capacity Blocks pricing, observed 22 August 2026). If your data-residency constraints allow choice of region, that is a real lever, and if they do not, it is a cost you should attribute to compliance, not to compute.

The denominator: why “it fits” is not the question

The instinct when sizing is to check whether the weights fit in GPU memory. That check is necessary and nowhere near sufficient, because at serving time GPU memory is split three ways:

1. Model weights. Fixed, known from parameter count and precision.
2. Activation and workspace memory. Roughly fixed per configuration.
3. KV cache. Everything left over, and this is what determines how many requests you can serve concurrently.

The KV cache holds the key and value tensors for every token in every in-flight sequence. Its size grows with the number of concurrent sequences multiplied by their lengths. Whatever memory the weights do not consume becomes your concurrency budget, and concurrency is throughput.

This is why a model that comfortably fits can still be throughput-bound. Fitting the weights into 80 GB with 4 GB spare gives you a working model with almost no room for concurrent requests. You will serve one or two sequences at a time, the GPU’s arithmetic units will sit mostly idle while decode waits on memory bandwidth, and your cost per token will be catastrophic. The hardware is not slow. You bought a memory allocation instead of a serving capacity.

vLLM exposes this budget directly. Its tuning documentation describes gpu_memory_utilization as the percentage of GPU memory vLLM pre-allocates for the cache, and lists increasing it as the first remedy when requests are being preempted for lack of KV cache space. The same page documents --kv-cache-memory, which lets you skip startup memory profiling by passing back the value vLLM logged on a previous boot, and it is candid about the trade: set the value too low and you leave batch capacity on the table; set it too high and the server either refuses to start or crashes later when a co-tenant changes the free memory available. The value is only valid on the same GPU with the same initial free memory.

That sentence is the whole KV cache sizing problem in one line. Too small and you are paying for a GPU you cannot fill. Too large and the server fails to start, or fails later when a co-tenant changes the free memory available.

Batching is the mechanism, and it has a ceiling

Decode is memory-bandwidth bound: producing one token requires reading the model’s weights from GPU memory, and that read costs far more time than the arithmetic. If you run one sequence, you pay the full weight read for one token. If you run sixty-four sequences in the same forward pass, you pay the same weight read for sixty-four tokens. Throughput per GPU rises close to linearly with batch size until some other resource binds.

Continuous batching (admitting and retiring sequences from the running batch each step instead of waiting for a fixed batch to complete) is what makes this practical for interactive traffic. The controls, per the vLLM tuning documentation, are:

The ceiling shows up as preemption. When KV cache space runs out, vLLM preempts requests and recomputes them later; the tuning page reproduces the warning it logs (“Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space”) and states that preemption and recomputation “can adversely affect end-to-end latency”. It also notes that in vLLM V1 the default changed from SWAP to RECOMPUTE, because recomputing from the stored prompt is cheaper than swapping KV blocks to host memory and back in the V1 architecture.

Recomputation is the diagnostic worth watching, because it is work you pay for twice and it does not appear as an error. The same page notes that preemption counts are exposed as Prometheus metrics, and that setting disable_log_stats=False logs the cumulative count. A cluster with a rising preemption counter is a cluster whose effective cost per token is drifting upwards for reasons no dashboard about GPU utilisation percentage will show you.

Prefill and decode compete, and the scheduler picks a side

A batch step can either process new prompts (prefill) or generate next tokens for running sequences (decode). These have opposite performance characteristics: prefill is compute-bound and parallel across the prompt, decode is bandwidth-bound and strictly sequential per sequence.

vLLM’s answer is chunked prefill, which the tuning documentation states is enabled by default whenever possible in V1. With it enabled, the scheduler prioritises completing in-flight generations over starting new ones: all pending decode requests are scheduled first, then any remaining token budget is given to prefill work, chunking a prefill that does not fit. The same page recommends setting max_num_batched_tokens > 8192 for throughput, particularly for smaller models on large GPUs, and notes that when chunked prefill is disabled max_num_batched_tokens must exceed max_model_len or the server may crash at start-up.

The economics of that default: prioritising decode protects inter-token latency for users already receiving a response, at the cost of time-to-first-token for users still waiting. Which side you want is a product decision, and it directly changes your achievable throughput and therefore your cost per token. There is no configuration that is cheap and low-latency simultaneously; there is only a choice about where the latency lands.

Prefix caching converts repetition into capacity

If many requests share a prefix (a system prompt, a document, a conversation history), the KV state for that prefix can be computed once and reused. vLLM implements this as automatic prefix caching, and the design document describes the block-level matching that makes it work.

Economically this is a utilisation multiplier, not a discount. A cached prefix costs no prefill compute and, more importantly, the cache blocks are shared, not duplicated per sequence, so more sequences fit in the same KV cache and your batch size goes up. The condition is the same as for hosted prompt caching: the shared content must sit at the start of the prompt, before anything that varies per request. See context window economics for the prompt structure this implies.

Utilisation arithmetic

Cost per token is inversely proportional to utilisation, which is worth writing out because the consequence is larger than it feels:

effective_cost_per_1M_tokens
  = gpu_hourly_rate × 1e6
    / (sustained_tokens_per_second × 3600 × utilisation)

A node at one-fifth utilisation costs five times per token what the same node costs fully loaded. No price on any vendor page changed. This is why the interesting comparison against a hosted API is never the hourly rate. It is the hourly rate divided by realised throughput, and realised throughput is the number you have to measure. See build versus buy for that comparison in full.

Three traffic properties destroy utilisation, and all three are common:

• Diurnal load. Capacity sized for the peak idles through the trough. A single-time-zone interactive product has a peak-to-trough ratio that no amount of tuning fixes.
• Redundancy. A second node for availability is, by design, idle capacity. It halves utilisation at a given load.
• Model proliferation. Each distinct model needs its own weights resident, so five models across five nodes each at low load is the worst case: five sets of weights consuming memory that could have been KV cache for one.

The corresponding remedies are backfill (batch work that can absorb troughs), consolidation (fewer models, or LoRA adapters over a shared base), and honest peak sizing.

Reserved against on-demand capacity

Once you accept that you are buying time, the question becomes how far ahead you buy it. The published spread is wide.

The Vertex AI rows show the shape of the trade with a vendor’s own numbers: the hourly rate falls as the commitment lengthens, and the gap is the price of keeping your options open. Google also notes on its Compute Engine pricing page that Spot prices are variable and can change up to once a day, and that Spot instances are recommended only for fault-tolerant workloads. Spot capacity is therefore usable for batch inference and evaluation runs, and not for anything with a latency SLO attached to a user.

AWS adds a caution worth quoting for anyone building a multi-quarter model: its Capacity Blocks pricing page states that reservation prices are updated regularly based on trends in supply and demand, that the current prices were scheduled to be updated next in October 2026, and that a block is charged at the prevailing rate at the time of purchase. Accelerator capacity is priced like a scarce commodity because it is one.

Interconnect matters only if you need it

Multi-GPU serving turns the network into part of the cost model. AWS states on its P5 instance page that P5 and P5e instances support up to 3,200 Gbps of networking using second-generation Elastic Fabric Adapter, and that the P5en generation on Nitro v5 reduces latency by up to 35% over P5 (Amazon EC2 P5 instances). That infrastructure exists for distributed training and for models too large to fit on one accelerator.

If your model fits on a single GPU, you are paying for interconnect you are not using. If it does not, tensor parallelism makes you dependent on it - and, per the vLLM tuning documentation, exposes you to the synchronisation overhead it warns about. Match the instance shape to the model, not to the specification sheet.

What to measure

Four numbers, none of which is “GPU utilisation percentage” as reported by the driver - that metric goes to 100% during an inefficient single-sequence decode and tells you nothing about cost.

1. Sustained output tokens per second per GPU, at your real sequence lengths and at the batch size your latency SLO permits.
2. Realised utilisation over a week, not a peak hour.
3. Preemption count, from the metrics the vLLM tuning page describes. Rising preemption means you are paying twice for the same tokens.
4. Prefix cache hit rate, which converts directly into effective batch size.

With those, gpu_hourly_rate / tokens_per_hour becomes a defensible figure you can compare against a vendor’s per-token price. Without them, self-hosted cost per token is not a measurement, it is a hope.

Related

• Build versus buy - the comparison this feeds
• Context window economics - why long prompts cost throughput
• Inference cost structure - how hosted vendors meter
• /ai - the wider section