The KV cache: why LLM serving is a memory problem

When people first serve a large language model, they expect the bottleneck to be arithmetic. Often it is memory. A large share of that memory, and a large share of what limits how many users you can serve at once, is something called the KV cache. Understanding it explains many practical decisions about batching, context length and cost.

How generation works

A transformer produces text one token at a time. To produce the next token, each layer's attention mechanism looks back at all the previous tokens. For each earlier token it needs two things, called the key and the value.

If the model recomputed those for the entire history at every step, generating a long answer would repeat huge amounts of work. Instead, the keys and values for tokens already processed are stored and reused. That store is the KV cache. Each new token only needs its own key and value computed and appended.

The two phases you will hear about

• Prefill. The model processes your whole prompt at once, building the cache. This phase is compute-heavy and can run in parallel across the prompt tokens.
• Decode. The model generates tokens one at a time, reading the cache and appending to it. This phase is dominated by memory bandwidth: each step must read a lot of stored data to produce a single token.

This is why time to the first token and speed per token behave differently, and why long prompts and long answers stress the system in different ways.

How big is the cache?

The cache grows with:

• the number of tokens in each conversation (prompt plus generated),
• the number of layers and the size of the keys and values in each,
• the number of conversations being served at once (each has its own cache),
• the numeric precision used to store it.

You can estimate it with a simple formula:

def kv_cache_bytes(layers, kv_heads, head_dim, tokens, bytes_per_value=2, batch=1):
    # 2 = one key + one value per token, per layer, per kv head
    return 2 * layers * kv_heads * head_dim * tokens * bytes_per_value * batch

Plug in the numbers for your model and you will see why a long context or a large batch can consume as much memory as the model weights themselves. The model weights are a fixed cost; the cache is a per-user, per-token cost.

Why this shapes system design

• Concurrency is limited by cache memory, not just compute. More simultaneous users means more caches.
• Long contexts are expensive. Doubling the context roughly doubles the cache for that request.
• Batching helps throughput because the model weights are read once for many requests, but each request still needs its own cache.
• Latency and cost trade off against how many requests you pack together.

Techniques that reduce the pressure

• Grouped-query and multi-query attention. Models share keys and values across several attention heads, shrinking the cache. This is a design choice made when the model is built.
• Quantising the cache. Storing keys and values at lower precision saves memory, at a small quality risk that should be tested.
• Paged cache management. Instead of reserving one large contiguous block per request, the cache is split into fixed-size pages that are allocated as needed, which cuts wasted memory and lets more requests fit.
• Prefix caching. If many requests start with the same text (a long system prompt), its cache can be computed once and reused.
• Continuous batching. Requests join and leave the batch at each step instead of waiting for the slowest one, keeping the hardware busy.
• Limiting context. Truncating, summarising or retrieving only what is needed reduces both cost and cache.
• Offloading. Moving less-used cache to slower memory trades speed for capacity.

What to measure

• Memory use split into weights, cache and overhead.
• Time to first token and tokens per second at realistic prompt and output lengths.
• Maximum concurrent requests before latency degrades or memory runs out.
• The effect of each optimisation on output quality, not just speed.

Common mistakes

1. Sizing a deployment from the model weights alone.
2. Testing with short prompts, then serving long ones.
3. Enabling cache quantisation without a quality check.
4. Ignoring that a long shared system prompt can be cached once.

Try it yourself

Pick a model whose configuration you can read (layers, heads, head size). Use the formula to compute cache size for 1,000, 4,000 and 16,000 tokens, then multiply by ten simultaneous users. Compare that total with the memory of a single accelerator. The result usually explains why serving is a memory problem before it is a compute problem.