Inference serving - the engine and the control plane

Serving an LLM used to mean “run the model and answer requests”. It does not, at any scale that matters. Production inference has quietly split into two systems that get discussed as one: the engine that makes a single replica fast, and the control plane that decides which replica should run a given request at all. Treating them as the same problem is how teams end up with a fleet that is individually fast and collectively slow.

The request is two different problems

Every inference request is two phases with opposite bottlenecks:

• Prefill - processing the prompt once, in parallel, to build the key-value (KV) cache. This is compute-bound: it wants FLOPs, and a long prompt is one heavy burst.
• Decode - generating the response token by token, where each step re-reads the whole KV cache. This is memory-bandwidth-bound: it wants memory speed, and it runs for as many steps as there are output tokens.

They want different things from the hardware - one is a sprint, the other is a marathon - which is the entire case for the modern serving architecture.

The engine layer: vLLM and friends

The engine optimises what one replica does with a batch of requests. vLLM is the reference here - out of Berkeley’s Sky Computing Lab and now one of the most active open-source AI projects - and its name mechanisms are the load-bearing ones:

• PagedAttention manages attention memory in pages rather than contiguous blocks, which is what makes high-throughput batching possible without wasting GPU memory.
• Continuous batching inserts new requests into a running batch instead of waiting for it to finish, which is what turns a latency-averaging server into a throughput one.
• Prefix caching reuses the KV cache when requests share a prompt prefix - the mechanism the control plane later exploits.

Engines like vLLM support hundreds of model architectures and handle the quantisation, parallelism (tensor, pipeline, expert, context) and structured-output machinery that a production deployment accumulates. The engine is where “how fast can one replica go” gets answered, and it is largely a solved layer.

The control plane layer: llm-d and the scheduler problem

Once there is more than one replica, a harder question appears: not “how fast can this pod run a request” but “which pod should run it, and under what policy”. That is the control plane’s job, and llm-d is its clearest current implementation - a Kubernetes-native layer that sits in front of the fleet.

The control plane’s real mechanism is its endpoint picker (the EPP), which integrates with a proxy over ext-proc and scores each candidate pod before placing the request:

• KV-cache locality - does this pod already hold the request’s prefix in its cache? Routing to it skips the prefill entirely, which is where the large latency and cost wins live.
• Queue depth and utilisation - which pod is least loaded right now.
• Priority and SLO - which requests get scheduled first under pressure.

The key detail is that the cache signal is real, not heuristic: the model servers publish their KV-cache state over an event bus, and the router builds an index over which prefixes are resident where - so “prefix-cache-aware routing” means routing to the pod that actually holds the cache, not the one that might.

Disaggregation - the split that makes the control plane necessary

The reason the control plane exists as a separate layer at all is that prefill and decode can run on different pods. Disaggregated serving separates them so each phase runs on hardware suited to it - a compute-heavy prefill pool, a bandwidth-heavy decode pool - and the request’s KV cache is handed from one to the other, typically over RDMA.

The topologies form a spectrum:

• No disaggregation - one pod does everything. Fine for short prompts and low scale.
• P/D - prefill and decode on separate workers; the default split, and the one that removes the worst interference, where a long prefill blocks ongoing decodes.
• E/PD and E/P/D - splitting multimodal encoding (images, audio) off as its own stage, which matters once requests carry more than text.

The honest caveat is the seam: splitting the phases turns a local cache hand-off into a network operation, and disaggregation only pays when the transfer is genuinely fast - the llm-d docs say plainly that without RDMA the fallback to TCP is inefficient and suits only testing. The win is real; so is the network bill.

Where it breaks

• A router that guesses is worse than a router that knows. The value of the control plane is measured in cache hits; a router that only balances load treats a pod holding your prefix the same as a cold one, and you pay the prefill twice.
• Disaggregation adds a failure mode, not just a split. The KV transfer between prefill and decode is a new dependency - a slow link or a downed decode worker mid-request is a request that never completes, and the retry logic is the deployment’s problem, not the protocol’s.
• The engine and the control plane version independently. vLLM moves fast and llm-d moves fast, and they are separate projects - a feature one exposes is not automatically a signal the other can read.
• It is still one model. The whole stack - engine, control plane, disaggregation - serves one model’s weights at a time; it does not magically make a model smarter, only faster to run.

What this is not for

• Serving one model to one user - the control plane earns nothing until there is a fleet to route across and a cache hit worth hunting.
• A problem that is really a model problem - serving optimises the cost and latency of a model’s output, not the quality of it.
• Anything you can buy managed - if your inference volume is small enough for a hosted API, the economics of running your own engine and control plane do not favour building it.