AI
Disaggregated serving
Splitting the prefill and decode phases of an inference request onto different workers, so each runs on hardware suited to its own bottleneck.
Prefill is compute-bound; decode is memory-bandwidth-bound. Running them on separate pools lets each scale independently and stops a long prefill from blocking ongoing decodes - but it turns the KV-cache hand-off into a network operation, which only pays when the transfer (usually RDMA) is faster than recomputing the prefix.