AI
KV cache
The cached key and value tensors from a prompt's attention computation, kept so that generating each next token does not recompute the whole prefix.
The KV cache is why prefill and decode are different problems: prefill builds it once in a compute-bound burst, then decode reads it back token by token in a memory-bandwidth-bound loop. It is also the resource a modern inference router exploits - routing a request to the pod that already holds its prefix skips the prefill entirely.