Concept
The key-value cache: what it stores, what it costs and what it limits
What a model's key-value cache holds during generation, how its memory grows with model shape and context length, and how that limits concurrent requests. Separates it from prompt (prefix) caching and from application-level response caching.
Topics: kv-cacheinferencememory