Weight Quantization vs KV Cache Quantization: What Actually Saves GPU Memory?

Weight quantization reduces the storage needed for a fixed model's parameters. KV cache quantization reduces the state retained for active requests, so its savings grow with cached tokens and concurrency. On the Qwen2.5-7B-Instruct example below, INT4 weights save far more storage than INT4 KV at one 32,768-token sequence. At higher cache occupancy, that order reverses. Neither result, by itself, tells you which option will run faster.

The comparison has two different crossover points: when the BF16 cache becomes as large as the BF16 weights, and when compressing the cache saves more bytes than compressing the weights. Keeping those questions separate makes the calculation useful.

For the groundwork, see KV Cache Explained. The arithmetic builds on How to Calculate KV Cache Memory: MHA, GQA, and MQA.

Weight and KV quantization reduce different parts of the memory budget; the calculations below are not measured GPU allocation.

Weight Quantization: A Fixed Storage Reduction

Qwen's official Qwen2.5-7B-Instruct BF16 tensor index reports 15,231,233,024 bytes, or about 15.23 GB. Its AWQ checkpoint index reports 5,570,747,392 bytes, or about 5.57 GB. The difference is approximately 9.66 GB of stored tensor data.

For those two fixed checkpoints, that difference does not grow with the number of requests. It is a storage comparison, however, not a promise that loading the model will reduce GPU allocation by exactly the same amount. Runtime conversion, placement, and working buffers still matter.

An INT4 checkpoint is not necessarily one-quarter the size of its BF16 counterpart. Quantization scales and zero points need storage, and tensors outside the quantized layers can remain at higher precision. A checkpoint's actual tensor inventory is a better starting point than assuming every parameter occupies four bits.

AWQ means Activation-aware Weight Quantization. Its scaling method uses activation statistics to identify important channels and reduce their quantization error. It does not require preserving a special one-percent subset in FP16 within the quantized weight matrices. That does not mean every tensor in an AWQ checkpoint, including embeddings and the output head, must be quantized.

KV Cache Quantization: Savings That Grow with Retained Tokens

The official model configuration has 28 layers, four KV heads, and a head dimension of 128. With a BF16 cache, the logical payload is:

2 × 28 × 4 × 128 × 2 = 57,344 bytes per retained token
57,344 × 32,768 = 1,879,048,192 bytes
                  = 1.75 GiB, or about 1.88 GB

Holding the tensor dimensions fixed, eight-bit values use half the raw data bytes of BF16; packed four-bit values use one-quarter. At 32,768 retained tokens, that gives approximately 0.94 GB for INT8 and 0.47 GB for INT4, before extra storage. INT8 and FP8 have the same bit width but different numeric representations; their quality and kernel support are not interchangeable.

Actual quantized caches may also store scales, metadata, or a higher-precision residual region. Transformers' cache documentation describes implementation-specific settings, including residual length. The quarter-size INT4 figure below is an ideal payload calculation, not a guaranteed allocation ratio.

Four Conditions on One Real Model

All values use decimal GB: one GB is one billion bytes. The comparison holds the architecture and retained-token count constant, combining official checkpoint tensor storage with idealized KV payload.

This is not measured GPU runtime memory. It excludes activations, temporary dequantization buffers, framework workspace, allocator fragmentation, and cache-management overhead. It also does not establish that a particular serving engine supports all four combinations with the same kernels or quality.

Calculated storage: Qwen2.5-7B-Instruct, one sequence retaining 32,768 tokens

Condition	Weights (GB)	Ideal KV (GB)	Sum (GB)
Neither: BF16 weights + BF16 KV	15.23	1.88	17.11
Weights only: AWQ INT4 + BF16 KV	5.57	1.88	7.45
KV only: BF16 weights + INT4 KV	15.23	0.47	15.70
Both: AWQ INT4 + INT4 KV	5.57	0.47	6.04

AI NodeLab calculation using the linked Qwen tensor indexes and architecture. Totals are calculated from unrounded bytes. At this occupancy, the weight-storage reduction is about 9.66 GB, versus about 1.41 GB of ideal KV savings. That identifies the larger capacity lever under these assumptions; it does not establish the best latency or quality tradeoff.

Two Crossover Points, Two Different Questions

Assume one model replica resides on a single GPU, with independent full-attention sequences and no shared prefixes or offloading. Count tokens actually retained at the same time, not queued requests or the sum of advertised context limits. For a distributed deployment, redo the accounting using the tensors resident on each device.

When does the BF16 cache equal the BF16 weights?

15,231,233,024 ÷ 57,344 ≈ 265,612 retained tokens

That is roughly 266K tokens across resident sequences. Eight sequences of 32,768 tokens total 262,144 tokens, just below this size-equality point. This comparison concerns the sizes before quantization.

When does INT4 KV save more than INT4 weights?

Weight-storage saving:
15,231,233,024 − 5,570,747,392 = 9,660,485,632 bytes

Ideal BF16-to-INT4 KV saving per token:
57,344 × 0.75 = 43,008 bytes

Savings crossover:
9,660,485,632 ÷ 43,008 ≈ 224,621 retained tokens

At approximately 225K concurrently retained tokens, ideal INT4 KV savings match the AWQ weight-storage saving. Beyond that, the ideal KV reduction is larger. Real cache metadata and residual precision can shift the crossing, as can a different weight checkpoint.

These are model-specific accounting thresholds, not instructions to delay KV optimization until 225K tokens. A smaller KV saving can still resolve an out-of-memory problem earlier. Conversely, a larger theoretical saving may not justify unacceptable quality loss or slower responses.

Calculated thresholds for the stated single-replica assumptions: official Qwen tensor storage plus ideal KV payload. Cache metadata, residual precision, and runtime allocations are excluded. Open the image to enlarge it.

Lower Memory Use Does Not Guarantee Lower Latency

Transformers' documentation warns that quantized caching can increase latency for short contexts when memory is already sufficient. The conversion work has a cost. Measure it against the benefit of moving or retaining fewer bytes.

An April 2026 vLLM analysis illustrates this on H100 with FlashAttention-3, Llama-3.1-8B, and concurrency one. Its fitted inter-token-latency model puts FP8's crossover near 7,000 input tokens. The FP8 intercept is 6.58 ms versus 6.44 ms for BF16: a fitted decode-latency difference, not a one-time request fee.

The FP8 slope is 54% of the BF16 slope. That means the context-dependent part grows more slowly, not that the whole response takes 54% as long. These results belong to the report's model, hardware, backend, and software versions; they do not predict INT4 KV performance on Qwen.

The report also finds slower prefill in its H100 test with a head dimension of 256, where two-level accumulation protects accuracy but adds register pressure. Decode can improve while prefill regresses. Our prefill versus decode guide explains why those phases need separate timing.

Another example comes from LMDeploy's K4V2 benchmark. On H200 with Qwen3-30B-A3B-Base, ShareGPT inputs, concurrency 64, and 5,000 requests, its four-bit-key/two-bit-value setup reduces throughput by 7.3% and increases mean end-to-end latency by 7.8%, despite reducing cache storage. It is one implementation's tradeoff, not a verdict on every low-bit cache.

Check the Hardware, Backend, and Quality Together

Storing FP8 values and executing native FP8 arithmetic are different capabilities. NVIDIA's Ada tuning guide confirms FP8 Tensor Core support, but the model, attention backend, and serving-engine version still determine whether a particular accelerated path is available. An eight-bit cache setting alone does not establish how the computation runs.

NVIDIA's TensorRT-LLM guidance favors weight-only approaches for small batches and considers weight-and-activation quantization for larger batches, while explicitly noting model dependence. Weight-and-activation quantization is not the same setting as weight-and-KV quantization. Treat its batch-size examples as starting points for testing, not universal switching rules.

For a deployment test, compare all four supported weight/KV combinations against the same baseline. Keep the model, prompts, output limits, concurrency, and engine version fixed. Check task quality as well as memory, time to first token, later token latency, and throughput. Testing weights and KV separately does not establish the quality of using both together.

A Korean Research Proposal for Where Dequantization Happens

SK hynix researchers published StreamDQ on July 9, 2026. The proposal places dequantization blocks in the HBM base die, converting compressed weights into higher-precision values during memory reads. It addresses weight dequantization, not the INT4 KV-cache calculation above.

The authors use synthesis and simulation, with performance modeling calibrated against A100 measurements. This is not a benchmark of a fabricated StreamDQ chip. Their evaluation uses public HBM3 parameters, while discussing more capable custom-HBM base dies as motivation. The paper does not establish a commercial HBM4 feature or a confirmed product roadmap.

For the Korean memory-industry connection, the useful distinction is between storing fewer bits and deciding where those bits are reconstructed. StreamDQ makes that design question concrete without proving commercial adoption. Our HBM4 base-die explainer provides the broader customization context. Product-level implementation and system validation would be needed before treating this proposal as a competitive advantage.

Choose the Constraint You Need to Solve

Start with the problem, not the smallest bit width:

• The model barely fits: compare supported weight checkpoints and measure their loaded footprint.
• Long or concurrent requests exhaust cache space: estimate retained-token occupancy, then test a supported lower-precision KV format.
• Memory fits but responses are slow: profile prefill, decode, and scheduling before assuming more compression will help.
• Both formats are reduced: validate the combined configuration against representative tasks, including long-context cases.

The 225K figure answers which idealized reduction saves more bytes in this example. Your deployment decision also needs an acceptable quality level, a compatible implementation, and measured latency. Keep those conditions beside the memory number whenever you compare a new quantization claim.