The serving layer
Running the model & the world · 2023
The idea
Section titled “The idea”Prefix caching, paged KV memory, continuous batching, eviction: the deployment economics of remembering. Paged attention treats the KV cache like virtual memory - serving is mostly cache management.
The paper’s own motivating number: naive KV-cache allocation was wasting 60-80% of reserved memory to fragmentation and over-provisioning. PagedAttention borrows the OS’s own answer - fixed-size pages, a page table per sequence, allocate on demand - and vLLM, the system this paper shipped, is now the reference implementation most open-model serving stacks either run or copy.
The serving trick behind every chat product: the server pre-fills the conversation tokens up through the assistant’s turn-start marker, then runs plain autoregressive sampling. Serving a chat reply is mechanically identical to continuing a document - only the conditioning differs (ch. 1:08).
The story, animated
Section titled “The story, animated”A shared, bounded memory pool - pages claimed, shared, freed, and evicted - then the same chat-reply mechanism, laid bare.
Where it sits
Section titled “Where it sits”Worth having first: KV cache
Deeper, hands-on: Continuous batching, from the scheduler’s side · The token bucket
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).