If AI Gets More Efficient, Does Memory Demand Fall?

AI efficiency can reduce the memory needed for a fixed workload. It does not, by itself, tell us whether the industry will buy less memory. A smaller KV cache, more memory per rack and rising memory shipments can coexist because they measure different things. For Korean suppliers, the useful question is which memory products a deployment needs after those savings, and how many such deployments customers actually build.

The distinction becomes clearer when we separate three quantities: memory occupied by a workload, memory installed in a system, and new memory purchased over time. The evidence below supports that distinction. It does not establish that efficiency has already increased or reduced fleet-wide demand.

Two accounting views, not a causal comparison. The left panel illustrates KV-cache-only compression; model weights are generally shared by requests using the same model replica, not allocated anew per request. The right panel is schematic and includes Rubin's maximum CPU-memory capacity. Neither panel establishes fleet-wide demand.

A stock question keeps getting answered with a flow number

Installed memory is a stock. Tokens processed per month are a flow. A service can process more tokens by using its existing hardware more effectively, buying more hardware, or changing the work each token requires. Token growth alone cannot tell us which happened.

The reverse shortcut is also unreliable. Multiplying bytes of cache per retained token by tokens processed per month does not give the capacity a service must install. Capacity depends on what must coexist in memory, including shared model weights, retained cache, temporary tensors and serving overhead.

For HBM embodied in newly shipped accelerators, a useful accounting identity is:

HBM shipped inside accelerators = sum across products of (packages shipped × HBM capacity per package).

This is not a complete measure of memory-market demand: purchasing schedules, inventories and replacement cycles also matter. Efficiency could affect either factor. Customers might need fewer accelerators for the same service, or future designs might use a different memory capacity. They could also use the savings to serve more work.

Keep memory terms separate, too. KV-cache compression does not automatically compress model weights, and weight quantization does not automatically shrink the cache. The practical difference is covered in weight quantization vs KV cache quantization.

The compression results are real, but their scope matters

DeepSeek-V2's paper reports a 93.3% KV-cache reduction compared with DeepSeek 67B. That is a comparison between particular models and architectures, not a saving that can be applied to any deployment. The paper also reports training and throughput improvements for the complete model; those should not all be attributed to its attention mechanism alone.

SAW-INT4 is a narrower example. Its authors report near-lossless accuracy in their evaluations and no measurable end-to-end overhead relative to plain INT4. The comparison is with plain INT4, not a guarantee of identical performance to a 16-bit cache on every model.

The underlying precision arithmetic is straightforward: storing a value in 4 bits rather than 16 cuts its raw payload by 75%. Scales, metadata and allocation overhead sit outside that calculation. Accuracy and actual serving speed still need testing.

Technique	What it changes	What does not follow automatically
Weight quantization	Bytes used to represent model weights	The same reduction in KV cache or total GPU memory
KV-cache quantization or compression	Storage of retained attention state	The same reduction in resident weights
Sparse MoE routing	Which experts perform computation for a token	All inactive experts disappearing from stored model weights

Sparse activation is especially easy to misread. An expert not selected for one token may still be stored on a GPU or elsewhere in the system. Lower active parameter counts are not, by themselves, lower resident-memory requirements.

When longer context cancels a cache saving

Here is a controlled example, not a market forecast. Keep the model architecture fixed, use full attention, and count distinct retained tokens without prefix sharing or offloading. Moving a cache from 16-bit to 4-bit reduces its raw payload to one quarter. Retaining four times as many tokens then brings that payload back to the starting size.

Relative raw KV payload = relative retained-token count × relative bits per value.

That relationship is useful only when its other inputs stay fixed. A new attention architecture, shared prefixes or a different placement policy changes the comparison. Use the KV-cache calculation guide to work through the tensor dimensions before applying it.

Real agent workloads make those conditions important. The September 8, 2026 vLLM AgentX benchmark post reports median input and output lengths of 142K and 444 tokens, respectively, with prefix-cache hit rates above 96%. These are observations from its agentic-coding traces, not universal medians for AI services.

In particular, 142K input tokens do not mean a separate 142K-token cache allocation for every request on a GPU. Reuse, cache lifetime and placement matter. The benchmark shows why context and caching deserve attention; it does not establish an increase in absolute memory use relative to an earlier deployment.

What NVIDIA's rack numbers show—and what they do not

NVIDIA publishes memory specifications for both GB300 NVL72 and Vera Rubin NVL72. Each has 72 GPUs and 36 CPUs, making a rack-level comparison useful. They are still different systems, not a controlled experiment in the effect of software efficiency.

Published rack specification	GB300 NVL72	Vera Rubin NVL72
GPU memory	20 TB HBM3e	20.7 TB HBM4
CPU memory	17 TB LPDDR5X	Up to 54 TB LPDDR5X
CPU-memory capacity / GPU-memory capacity	0.85:1	Up to approximately 2.61:1

Sources: NVIDIA's GB300 NVL72 and Vera Rubin NVL72 product tables, checked September 18, 2026. AI NodeLab calculations: 17 ÷ 20 = 0.85; 54 ÷ 20.7 ≈ 2.61. Each ratio uses that rack's own GPU-memory figure.

The published CPU-memory capacity ceiling is substantially higher for Rubin, while the listed HBM capacities are broadly similar. GB300's 17 TB is a directly published CPU-memory row, not an estimate obtained by subtracting two other rows.

The limits are just as important: these are rounded specifications, and Rubin's 54 TB is a maximum, not a measured average across customer installations. A maximum specification does not establish whether that configuration has shipped. Nor does the comparison prove that efficiency caused memory to move from HBM to CPU memory.

What it does show is that evaluating an AI rack by its HBM capacity alone misses a substantial part of its memory configuration.

A smaller active model is not necessarily a smaller stored model

The DeepSeek-V4-Pro preview model card illustrates the distinction. It lists 1.6 trillion total parameters and 49 billion activated parameters. Expert weights use FP4, while most other parameters use FP8. At a one-million-token context, the authors report 10% of DeepSeek-V3.2's KV-cache requirement.

For scale, a hypothetical all-4-bit representation of 1.6 trillion parameters would contain 0.8 TB of weight payload: 1.6 trillion × 4 ÷ 8 bytes, using decimal TB. That is an illustrative calculation, not the mixed-precision checkpoint size or its measured GPU footprint. Higher-precision tensors, metadata and runtime allocations must be accounted for separately.

The smaller cache and small active fraction therefore do not establish the model's total deployment footprint. Whether all experts remain in GPU memory is another deployment decision. The total-versus-active parameter guide explains why those quantities must not be used interchangeably.

Substitution and reuse can reduce particular purchases

There are also reasons not to assume every new AI workload requires more HBM.

In its September 9, 2025 Rubin CPX announcement, NVIDIA described a massive-context inference accelerator with 128 GB of GDDR7. That is a dated design announcement, not confirmation of the configuration customers are receiving today. It nevertheless illustrates that accelerator-memory choices can differ by workload. It does not mean HBM disappears from the surrounding system.

Meta's Vistara paper provides a different case: reusing DDR4 from decommissioned servers through CXL. The paper reports a 25% server-count reduction for its evaluated disaggregated ML inference workload, involving recommendation-model embedding storage. That result should not be generalized to all LLM-serving GPUs.

Reuse also breaks the simple link between capacity deployed and new memory purchased. Some capacity can come from existing modules. The paper is evidence of that mechanism, not a measurement of industry-wide DRAM purchasing. The roles of these tiers are explained in CXL vs HBM vs DDR.

Why the distinction matters for Korean suppliers

HBM is not the only relevant product. On April 20, 2026, SK hynix announced mass production of 192 GB SOCAMM2 built on 1c LPDDR5X for the Vera Rubin platform. Samsung's second-quarter 2026 results also identify HBM4, DDR5 and SOCAMM2 among its product priorities.

One source detail is worth checking for English-language readers. The Korean SK hynix announcement describes a shift from training toward inference. Its English version reverses that direction. This article follows the Korean original on that point; the two versions are inconsistent as checked on September 18, 2026.

These announcements establish product relevance, not guaranteed incremental revenue. If customers deploy larger CPU-memory configurations, suppliers of suitable low-power DRAM could have opportunities alongside HBM. If reuse, smaller models or improved utilization reduce new purchases, that opportunity could be smaller than the capacity specification suggests.

For a Korean-memory outlook, the useful evidence is therefore product adoption, shipped capacity and realized pricing—not a leap from one benchmark improvement to a supplier's earnings or share price.

What to check before the next efficiency headline

Ask four questions before turning a model result into a memory-demand forecast:

1. Which memory term actually shrank? Separate weights, KV cache, temporary tensors and allocator overhead. A percentage for one term is not a percentage for the whole system.
2. What stayed fixed? Compare workload quality, model architecture, retained tokens, concurrency and latency targets. Without those conditions, a before-and-after memory ratio can mislead.
3. Did installed capacity or new purchases change? Offloading, sharing and reused modules can alter the relationship between the two. A maximum rack specification is not a shipment-weighted average.
4. How much additional work followed the saving? If comparable services use fewer newly purchased memory bytes while meeting the same targets, that supports a reduction case. If more deployments and larger configurations outweigh the saving, aggregate purchases can rise. Shipment data must also be read alongside supply constraints and inventories.

Efficiency is a real saving at the level where it is measured. The market question is what customers do with that saving. Following the path from a smaller tensor to a deployed system—and then to a purchase—is more useful than assuming either that memory demand must fall or that it can never fall.