Inference
Running a trained model to produce output, as distinct from training it. The phase where cost, latency and throughput matter operationally.
Inference cost scales with token count, model size and whether reasoning is involved. Batching, caching (KV cache), quantisation and speculative decoding are all techniques to reduce it. For hosted models the cost is per-token pricing; for self-hosted models it is hardware and energy.
Explained in depth
Context is a per-request budget, not a capability. Attention cost scaling, long-context price tiers, prompt caching, and why chat history compounds.
Inference serving - the engine and the control planeHow production LLM inference is actually served - the engine (vLLM) that optimises one replica's execution, the control plane (llm-d) that decides where each request lands, and why prefill and decode are split onto different hardware.