How to Calculate KV Cache Memory: MHA, GQA, and MQA

A hypothetical transformer with 32 layers, eight KV heads, a head dimension of 128, and a 16-bit cache needs 4 GiB of KV data for one sequence containing 32,768 cached tokens. Eight such sequences need 32 GiB before model weights and other allocations enter the budget.

You can reproduce that estimate with a short multiplication. The difficult part is choosing the correct inputs: stored KV heads, cache precision, and the tokens actually retained across resident sequences. This guide works through those choices, then shows where a clean calculation diverges from a serving engine's memory report.

Calculated logical KV payload for a hypothetical full-attention model. Weights and runtime allocations are excluded.

Start with Bytes per Token

For a conventional decoder using full attention, equal key and value dimensions, and the same cache layout in each layer:

Bytes per token = 2 × layers × KV heads × head dimension × bytes per value

The leading two counts the key and value tensors. Each layer keeps its own cache; the formula counts those stored tensors across the model. Hugging Face's cache explanation documents the underlying tensor axes and per-layer storage.

Then multiply by the total number of retained token positions:

KV bytes = bytes per token × total retained tokens across sequences

If B resident sequences each retain T tokens, the last term becomes B × T. For requests of different lengths, add their retained lengths instead. The result is the logical tensor payload for one model replica, before allocation overhead, shared-prefix deduplication, or replication across devices.

For background on why inference stores these tensors, start with KV Cache Explained. Here, the focus is the memory arithmetic.

Which Inputs Belong in the Formula?

Input	What to enter	Common mistake
Layers	Layers retaining the conventional KV tensors being counted	Assuming every layer in a hybrid model uses the same cache
KV heads	Stored key/value head count	Substituting the query-head count
Head dimension	Width of each stored K or V head	Using the model's full hidden size
Bytes per value	Actual cache storage precision	Using weight quantization as the cache precision
Retained tokens	Token positions currently cached or planned for the workload	Treating the advertised context limit as live occupancy
Resident sequences	Sequences whose caches occupy memory	Counting all registered users or queued requests

The simple formula assumes that K and V have equal widths. It also assumes uniform full-attention layers. Architectures with different layer types or compressed cache representations require their actual storage layouts; the exclusions appear later in the guide.

Count the prompt and generated tokens that have already been processed and stored. When planning capacity, leave space for the output still to come. A prompt-only estimate can be correct at the start of a request and inadequate as generation continues.

Worked Example: 4 GiB at 32K Tokens

Use these hypothetical inputs: 32 full-attention layers, eight KV heads, head dimension 128, two bytes per value, and one sequence with 32,768 retained tokens.

2 × 32 × 8 × 128 × 2 = 131,072 bytes per token

That is 128 KiB per token across all layers. Multiplying by the retained length gives:

131,072 × 32,768 = 4,294,967,296 bytes = 4 GiB

Keep units consistent. One GiB is 1,073,741,824 bytes, while one decimal GB is 1,000,000,000 bytes. This example is therefore approximately 4.295 GB, even though it is exactly 4 GiB. NIST's binary-prefix reference explains the distinction.

This estimate describes the cache at the stated length. It does not claim that a GPU with that capacity can run the model: weights, temporary tensors, and runtime allocations still need room.

MHA vs. GQA vs. MQA Under the Same Conditions

Multi-head attention (MHA) has a KV head for each query head. Grouped-query attention (GQA) lets groups of query heads share KV heads. Multi-query attention (MQA) uses one KV head across the query heads. The original GQA paper explains this architectural relationship.

To isolate the cache effect, keep 32 layers, 32 query heads, head dimension 128, two bytes per value, and one 32,768-token sequence. Change only the stored KV-head count in this hypothetical comparison.

Architecture	KV heads	Cache per token	Cache at 32K
MHA	32	512 KiB	16 GiB
GQA	8	128 KiB	4 GiB
MQA	1	16 KiB	0.5 GiB

These are AI NodeLab calculations of logical KV payload, excluding weights and runtime overhead. Here, 32K means exactly 32,768 tokens.

The GQA case retains one-quarter of the MHA payload, a 75% reduction. This ratio follows from eight divided by 32 KV heads. It is not a throughput or model-quality ratio. Nor can an existing checkpoint generally switch among these rows through a harmless runtime setting: its trained attention architecture matters.

Check a Real Configuration: Qwen2.5-7B-Instruct

The official Qwen configuration lists 28 layers, 28 query heads, four KV heads, and a hidden size of 3,584. For this architecture, the head dimension is 3,584 ÷ 28 = 128. Sliding-window attention is disabled in that configuration.

With a chosen BF16 cache at two bytes per value:

2 × 28 × 4 × 128 × 2 = 57,344 bytes per token = 56 KiB

For one sequence retaining 32,768 tokens:

57,344 × 32,768 = 1,879,048,192 bytes = 1.75 GiB

Using the 28 query heads instead would produce 12.25 GiB, seven times the logical cache size. This is why the head-count distinction deserves attention before any GPU sizing decision.

The 1.75 GiB result is calculated from the published architecture and the stated cache precision. It is not a measured serving result. Check the engine's cache dtype separately from the model's weight dtype. Also verify context configuration: the model card describes extra configuration for lengths beyond the default 32,768-token setting.

Longer Context and More Sequences Multiply Together

Return to the hypothetical GQA example with 128 KiB per token. The following table assumes independent sequences, equal retained lengths, no prefix sharing, and a 16-bit cache.

Retained tokens per sequence	1 sequence	8 sequences	32 sequences
8,192	1 GiB	8 GiB	32 GiB
32,768	4 GiB	32 GiB	128 GiB
131,072	16 GiB	128 GiB	512 GiB

All entries are calculated logical KV payloads across one unsharded model replica, excluding weights and other allocations. The last row illustrates memory growth; it does not establish that a particular model supports 131,072 tokens or retains useful quality at that length.

The table helps separate two planning decisions. Longer requests increase the cache required by each sequence, while more resident sequences multiply the aggregate requirement. A capacity estimate should state both.

Requests Usually Have Different Lengths

Suppose three resident sequences in the same hypothetical GQA setup retain 4,096, 8,192, and 16,384 tokens. Their sum is 28,672 tokens, so the logical cache occupies:

128 KiB × 28,672 = 3.5 GiB

Multiplying the longest sequence by three would give 6 GiB. That can represent a conservative equal-length plan, but it is not the current logical occupancy of these three requests. Static allocation or padding may still make actual allocation larger.

Why the Engine Reports a Different Number

A calculation and a memory monitor often measure different things. Before adding an arbitrary overhead percentage, identify which layer of accounting the monitor reports.

Logical KV payload, cache allocation, and total GPU memory are different accounting views; sharing and placement can change GPU residency.

Allocation and Sharing

PagedAttention stores cache in blocks that need not occupy one contiguous region. This can reduce allocation waste, but block rounding and management still differ from an exact count of tensor elements. The PagedAttention paper describes the distinction between useful KV state and memory management.

Compatible requests may also reuse stored prefix blocks. vLLM's prefix-caching documentation explains when a new request can reuse a cached prefix. An independent-sequence calculation counts those tokens repeatedly; physical sharing can change that count. Verify actual sharing before subtracting it from a capacity plan.

Static caches can reserve space beyond the current retained length. Sliding-window or chunked-attention layers can instead stop growing at their configured limits. These behaviors are covered in Transformers cache strategies.

Cache Precision and Device Placement

An eight-bit KV payload uses half the data bytes of a 16-bit payload at identical dimensions. Packed four-bit values would use one-quarter before extra storage. Actual quantized implementations may retain scales, metadata, or higher-precision regions, so payload ratios are not guaranteed allocation ratios.

Weight quantization does not establish the KV-cache format. Check the cache setting, backend support, and quality on your workload; vLLM's quantized-cache guide documents cache-specific formats and scaling.

For multiple GPUs, avoid automatically dividing the total by the device count. Use the layers and KV heads actually stored on each device. vLLM documents KV replication cases when tensor-parallel workers outnumber KV heads.

CPU offload can reduce GPU residency while keeping cache elsewhere, with transfer costs. The placement question connects directly to where HBM, DDR, and CXL fit in an AI server.

Architectures Outside This Formula

Multi-head Latent Attention (MLA) stores a compressed representation, as described in the DeepSeek-V2 report. Hybrid models with recurrent state or different attention types also need separate accounting. Do not force their advertised layer counts into the uniform full-attention formula.

Turn a KV Budget into a Token Budget

First subtract model weights, runtime requirements, and operating headroom from usable device memory. NVIDIA's inference-optimization overview explains the broader memory components. What remains for KV is the input to this calculation.

Assume a hypothetical 20 GiB remains exclusively for logical KV payload after those deductions. Using the 128 KiB-per-token example:

20 × 1,073,741,824 ÷ 131,072 = 163,840 token slots

At 32,768 retained tokens per sequence, that is five sequences. This is an ideal memory-only ceiling under the stated assumptions, not a validated admission limit or a latency guarantee. Block allocation, sharing, device layout, and further output growth still need to match the plan.

Before accepting a “this model fits” claim, record the model configuration, cache dtype, retained-token budget, and runtime placement beside the answer. Those four details make the estimate reproducible and show which input must change when the workload grows.