vLLM

How production LLM inference is actually served - the engine (vLLM) that optimises one replica's execution, the control plane (llm-d) that decides where each request lands, and why prefill and decode are split onto different hardware.

Read the vLLM guide

Where vLLM sits

How this component relates to the rest of the stack. Items without a link are not covered here yet.

Depends on

Must be present for this to run

Linux

Deployed with

Commonly run alongside

Kubernetes

Used by

Builds on this

llm-d