Knowledge · Concept
The key-value cache: what it stores, what it costs and what it limits
What a model's key-value cache holds during generation, how its memory grows with model shape and context length, and how that limits concurrent requests. Separates it from prompt (prefix) caching and from application-level response caching.
Fast understanding
Definition
The key-value cache, or KV cache, holds the attention keys and values that a transformer has already computed for the tokens of one request, so that generating the next token does not recompute them. In every layer, attention turns each token into a key vector and a value vector, and each new token reads the keys and values of all the tokens before it. In a decoder-only model those earlier keys and values do not change while the answer is being generated, so they can be computed once, stored and reused.
Three different things get called a cache in LLM serving, and mixing them up leads to wrong estimates. This entry is about the first.
| Name | Where it lives | What it holds | Lifetime |
|---|---|---|---|
| KV cache | The serving engine, in accelerator memory | Keys and values for the tokens of one in-flight request | Until that request finishes, unless the engine keeps it for reuse |
| Prefix (prompt) cache | A provider feature or a serving-engine feature | The processed state of a prompt prefix that later requests repeat | Minutes or longer, depending on the provider and settings |
| Response cache | Your application | Finished answers | Whatever your time-to-live and versioning rules say |
Serving within a latency and cost budget covers the second and third. The prefix cache reuses work that was once held in the first, which is why the names overlap, but they are billed, sized and invalidated by different rules.
Why it matters
The cache is why concurrency, and not only raw compute, limits a self-hosted deployment. Batching raises throughput by advancing many requests per step, and every request in the batch holds its own cache, so the batch size that fits is set by this memory. The serving lesson makes that point for continuous batching.
It also explains why long inputs cost more than their token price suggests. A long prompt costs prefill time once and then occupies cache memory for as long as the request runs, which lowers how many other requests the same hardware can serve.
With a hosted API the memory is the provider's problem and appears in your bill as input and output token prices and context limits. The reasoning still helps, because it tells you which levers move cost and which do not: shortening the context and reusing a stable prefix both reduce the work per request, and neither is something the application can do by tuning the cache itself.
Explanation
How it works
During prefill, the model processes the whole prompt in parallel and stores a key and a value for every prompt token in every layer. During decode, each step processes only the one new token: it computes that token's key and value, appends them to the stored ones, and attends over everything stored so far. Without the cache, every decode step would have to reprocess the entire sequence. The phases are described in Anatomy of a model request.
What the cache costs is memory, and the amount follows from the model's shape:
bytes per token = 2 x layers x key-value heads x head dimension x bytes per value
cache for one request = bytes per token x tokens held (prompt plus generated so far)
The 2 counts the key and the value. In standard multi-head attention the number of key-value heads times the head dimension equals the model's hidden size. The vLLM paper uses that form for a 13-billion-parameter OPT model: 2 x 5,120 (hidden size) x 40 (layers) x 2 bytes (FP16) is about 800 KB for one token, and a request of 2,048 tokens can hold as much as 1.6 GB (Kwon and others, 2023, Section 3).
Two consequences are worth separating. The cache grows linearly with the tokens a request holds, and it is held for the whole life of the request. And every decode step reads all of the request's stored keys and values, so the work of one step grows with context length. How much that shows up in per-token latency depends on the model, the hardware and the implementation.
Model designers can shrink the formula by using fewer key-value heads than query heads. Grouped-query attention uses an intermediate number of key-value heads, with multi-query attention (one key-value head) as the extreme. The authors of the grouped-query attention paper report quality close to multi-head attention at speed comparable to multi-query attention (Ainslie and others, 2023). For a team the head count is a fixed property of the model. For a team hosting an open-weight model, read its configuration to find the layer count, key-value head count and head dimension, instead of assuming the multi-head formula. For a hosted API, use architecture details only when the provider makes them available.
How the engine allocates the cache matters as much as its size. Reserving a contiguous slab for the longest possible output wastes memory on tokens that never arrive. The vLLM paper measured existing systems at the time and reported that only 20.4 to 38.2 percent of KV cache memory held actual token states, with the rest lost to reservation, internal fragmentation and external fragmentation. Its answer, PagedAttention, splits each request's cache into fixed-size blocks that need not be contiguous and tracks them with a table (Kwon and others, 2023, Sections 1, 3.1 and 4). Those percentages describe the systems compared in that paper, not every engine today. The design point that carries over is that wasted cache memory directly lowers how many requests can run at once.
Trade-offs
| Lever | Effect on cache memory | Cost or risk |
|---|---|---|
| Fewer key-value heads (grouped-query or multi-query attention) | Smaller bytes per token | A property of the model, chosen at design time. Reported quality is close to multi-head for grouped-query, which you should still confirm for your task |
| Storing keys and values at lower precision | Fewer bytes per value in the formula | Possible quality loss. This is a technique to test on your own evaluation set, not a free saving |
| Shorter contexts (trimming or summarising history, retrieving fewer passages) | Fewer tokens held per request | Information the model no longer sees |
| Block-based allocation | Less memory lost to fragmentation and over-reservation | A property of the serving engine. Choosing an engine that does this is a deployment decision |
| Capping concurrency or context length per request | Keeps the total inside the budget | Queueing or rejected requests at peak |
None of these changes the model's answers except the ones that change what the model sees or stores, and those need the evaluation work described in Evaluating output you can't unit-test.
Deeper reference
Example
A constructed example. The model below is hypothetical, with numbers chosen to make the arithmetic visible. It is not a measurement of any real model or accelerator.
Assume 32 layers, 8 key-value heads, a head dimension of 128 and 2 bytes per value:
bytes per token = 2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB
Assume 20 GiB of accelerator memory remains for the cache after the model weights and runtime overhead. That holds 20 GiB / 128 KiB = 163,840 tokens in total, shared by every in-flight request. If requests average 4,000 tokens (prompt plus answer), about 40 can be in flight at once.
Now suppose the same model had 32 key-value heads, as a multi-head design would. Bytes per token become 512 KiB, the budget holds 40,960 tokens, and only about 10 requests of 4,000 tokens fit. The only change was the head count, and the concurrency ceiling fell by a factor of four.
Context length has the same leverage. Moving the average request from 4,000 to 16,000 tokens in the first configuration cuts the ceiling from about 40 requests to about 10. A feature that stuffs long retrieved context into every prompt is therefore also a capacity decision.
Failure modes
- Capacity planned from the weights alone. The model fits, so the deployment looks fine, and then it runs out of memory or queues badly when several long requests arrive together. Budget the cache explicitly.
- Planning from the average context length. Memory is consumed by the requests that are in flight, and a small share of very long requests can use a large share of it. Size from the context-length distribution, including its tail, not its mean.
- Applying the multi-head formula to every model. The key-value head count can be much smaller than the query head count, and some architectures change how attention state is kept. Read the model's configuration where it is available.
- Treating the three caches as one. A prefix-cache hit changes billing and prefill work. It does not by itself give a request more room in the context window or remove the cache memory that request needs while it runs.
- Copying the example figures. The sizes above come from one 2023 paper's model and from a hypothetical configuration. Compute yours from the real model.
Interview considerations
If a design involves hosting a model, add the cache to the memory budget next to the weights, and state the three inputs: the model's layers, key-value heads and head dimension, the bytes per value, and the expected context-length distribution. The result is a rough ceiling on concurrent requests, which then feeds the batching and queueing discussion. If the model is behind a hosted API, say that the provider owns this and move on to the levers you do control.
Sample wording for a candidate: "For a self-hosted model I'd budget accelerator memory as weights plus runtime overhead plus a KV cache. The cache costs two times layers times key-value heads times head dimension times bytes per value for every token a request holds, so I'd check the model's configuration, size for the long end of our context lengths, and treat that as the cap on concurrent requests."
Related Lessons
Follow one request through tokenization, the context window, prefill and decode to see which steps set cost and latency. Opens with who this Track is for, what it assumes and what it covers.
Compare the main levers: batching, caching, streaming and routing between models. Learn which lever suits which workload and what each one gives up.