KV Cache Explained: Why AI Inference Is Becoming a Memory Problem

When a large language model generates the next token in a response, it does not recompute everything from scratch. During autoregressive decoding, it reuses stored Key and Value tensors from the tokens that have already been processed — the "KV cache." That reuse avoids a large amount of redundant work and is a core reason token-by-token generation can be served efficiently.

The trade-off is memory. The cache grows with active sequence length and must share finite accelerator memory with the model's own weights and other runtime state. At long contexts and high concurrency, KV-cache pressure can become one of the main limits on how many active sequences a GPU can serve at once — alongside compute, memory bandwidth, and scheduling.

What Are the K and V in "KV Cache"?

Inside a transformer attention layer, token representations are projected into Query (Q), Key (K), and Value (V) vectors. Queries are compared with Keys to determine attention weights, and those weights are then used to combine the Values. This Q/K/V mechanism is the foundation of the transformer architecture introduced in Vaswani et al.'s "Attention Is All You Need" (NeurIPS 2017).

During autoregressive decoding, the important asymmetry is that the K and V tensors for previously processed tokens can be reused at later steps. The current step needs a new query, but the earlier K/V states do not need to be regenerated. That is why K and V — rather than Q — are the states that serving systems cache.

Why Cache Anything at All?

Language models generate text one token at a time. Each new token needs to attend back over the tokens that came before it — including the prompt and the text already generated within that request.

Once an earlier token has been processed, its Key and Value tensors can be reused at later decoding steps. Recomputing them for every new token would be wasteful, so inference systems calculate them once and store them for reuse, as described in NVIDIA's technical overview of LLM inference optimization. That stored collection of K and V tensors is the KV cache.

Without a KV cache, the model would repeatedly recompute K/V states for the already processed prefix as it generated each new token. With the cache, those prior states are reused and only the new token's states have to be added.

How Big Does a KV Cache Actually Get?

The exact formula depends on the attention architecture. A general per-token approximation is:

2 × num_layers × num_kv_heads × head_dim × bytes_per_element

The factor of 2 accounts for storing both K and V. Across a batch of active sequences, the cache scales with sequence length and the number of active sequences.

For standard multi-head attention (MHA), num_kv_heads is the same as the number of query heads, and num_heads × head_dim is commonly equal to the model's hidden size. In that special case, NVIDIA gives the familiar shorthand:

batch_size × sequence_length × 2 × num_layers × hidden_size × bytes_per_element

That distinction matters because modern Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) deliberately use fewer K/V heads than query heads, reducing the cache footprint.

NVIDIA's example for Llama 2 7B shows the scale clearly. With 32 layers, hidden size 4096, batch size 1, sequence length 4096, and FP16 storage at 2 bytes per element, the KV cache is about 2 GiB — for one request, before the model weights are counted.

That cost grows with the amount of active context being served. Several trends can push it higher without changing the model's parameter count:

• Longer context windows. Google opened Gemini 1.5 Pro's 2-million-token context window to all developers on 2024-06-27. Meta's Llama 4 Scout, released 2025-04-05, advertises a 10-million-token context window. These are vendor-stated maximums, not evidence that every production request actually uses the full window or retains equal quality across it.
• Larger batches. Serving systems batch multiple active sequences to improve accelerator utilization and throughput.
• More concurrent active sequences. Continuous-batching systems may schedule them dynamically rather than treating "concurrency" as a literal fixed batch-size setting, but more simultaneously active sequences still increase aggregate KV-cache demand.

Push any of those up, and KV-cache pressure rises even if the model weights themselves do not change.

Why This Becomes a Serving Bottleneck, Not Just an Accounting Detail

KV cache and model weights compete for the same finite accelerator-memory pool. As the cache footprint per active sequence rises, fewer sequences fit at once. That can reduce the practical batch/concurrency level a server can sustain before it has to evict cache blocks, shorten contexts, offload data, or add more hardware.

Memory management can make that ceiling better or worse. The vLLM paper (Kwon et al., SOSP 2023) documented that then-common serving systems often reserved contiguous KV-cache space around a request's maximum possible length. If the request finished early, some reserved space went unused, while differently sized allocations created additional fragmentation.

PagedAttention, the algorithm introduced with vLLM, greatly reduces that waste by borrowing an idea from operating-system virtual memory. It splits each sequence's KV cache into fixed-size blocks that do not need to occupy one contiguous region of GPU memory, allowing blocks to be allocated as needed and shared more flexibly.

The vLLM paper reported 2–4× throughput improvement over FasterTransformer and Orca at the same level of latency, with larger gains on longer sequences, larger models, and more complex decoding algorithms. That result is specific to the baselines and test conditions in the 2023 paper; it should not be read as a universal multiplier for every modern serving stack.

Shrinking the Cache: Architectural Fixes

One way to reduce KV-cache pressure is to change the attention architecture so the model stores fewer distinct K/V heads.

Standard multi-head attention (MHA) gives each query head its own K/V head. Multi-Query Attention (MQA) goes to the other extreme, using a single K/V head shared across all query heads. Grouped-Query Attention (GQA) sits between those two: multiple query heads share a smaller set of K/V heads.

GQA was formalized in Ainslie et al. (EMNLP 2023), which showed that models uptrained from multi-head checkpoints with about 5% of the original pretraining compute could achieve quality close to full multi-head attention with speed comparable to MQA. The design is used in models such as Llama 2 70B and Mistral 7B. Falcon 40B, by contrast, is commonly cited as an MQA-era design rather than a GQA example.

DeepSeek took a different route with Multi-head Latent Attention (MLA), introduced in the DeepSeek-V2 technical report. Instead of storing full-resolution K/V states for each head, MLA compresses the information into a lower-dimensional latent representation and reconstructs what is needed during attention.

DeepSeek reported that DeepSeek-V2 reduced KV-cache usage by 93.3% and increased maximum generation throughput to 5.76× compared with its earlier DeepSeek 67B model. That is a whole-model comparison between two different generations, not a controlled experiment isolating MLA alone; DeepSeek-V2 also changed other parts of the architecture, including its MoE design. The numbers should therefore be treated as DeepSeek's reported end-to-end comparison, not as proof that "MLA alone" delivers those gains.

Shrinking the Cost: Quantization and Moving the Cache Off the GPU

A second lever is storing KV tensors at lower precision.

NVIDIA TensorRT-LLM supports FP8 KV-cache quantization, and NVIDIA has reported that switching to FP8 KV cache can enable roughly 2–3× larger batch sizes in some workloads by reducing persistent cache memory. That is a workload-dependent result, not a universal guarantee.

The newer direction goes below 8-bit precision. In NVIDIA's 2025 work on NVFP4 KV cache for Blackwell, moving from FP8 to NVFP4 cut KV-cache memory footprint by about 50%. In a Qwen3-Coder-480B-A35B test configuration, NVIDIA reported up to 3× lower time-to-first-token latency through higher effective cache residency and hit rates. Those numbers are specific to the tested model, hardware, and workload, but they show why cache precision has become a first-class serving-system design choice.

The third lever is offloading cache blocks out of GPU memory. NVIDIA's Dynamo 1.0, released in March 2026, supports distributed inference features including KV-aware routing and disaggregated serving. Current Dynamo KV-cache offloading documentation describes moving reusable KV blocks from GPU memory into CPU memory and disk-backed tiers so effective cache capacity can extend beyond GPU HBM.

That does not make offloading free. Moving cache blocks across memory and storage tiers adds transfer latency and bandwidth requirements, so a serving system has to decide which blocks are worth keeping close to the GPU, which can be moved, and which should simply be recomputed.

Where This Is Headed

None of these techniques make the KV cache disappear. They make it smaller, pack it more efficiently, reduce the number of K/V heads that must be stored, or move less time-sensitive blocks away from scarce GPU memory.

The broader pressure is structural: longer active contexts and more simultaneously served sequences increase the amount of reusable attention state a system may need to hold. That is why a faster GPU by itself is not always a complete answer to inference scaling.

AI NodeLab's HBM Explained covers the hardware side of high-bandwidth memory, while The AI Memory Wall Explained looks at the broader compute-versus-memory gap.

The next question is what happens when KV data no longer fits comfortably in GPU HBM. That broader memory-tiering problem is where technologies such as CXL-attached memory and emerging flash-based tiers become relevant — and where this series goes next.