Multi-GPU LLM Memory: Why Total VRAM Can Mislead

I would not size a multi-GPU deployment by adding up its VRAM and stopping there. For Korea's HBM industry, I see useful serving capacity as a stronger test of progress than installed capacity alone. More memory helps only when the software can place weights and live state where that memory is available.

The easily missed detail is KV-cache replication. Dividing every tensor by the GPU count can predict a comfortable fit even when the actual layout exceeds one device's capacity. A per-GPU ledger exposes that mistake before “more aggregate memory” becomes a misleading sizing answer.

First decide what the GPUs share

Tensor parallelism divides supported tensor operations across devices. Pipeline parallelism assigns model stages, often groups of layers, to different devices. Independent replicas instead retain separate model copies to serve different requests. These arrangements spend memory differently.

vLLM's scaling guide describes tensor and pipeline strategies for fitting a model across devices. Its configuration guidance is an implementation starting point, not a guarantee that every allocation divides evenly.

For a mixture-of-experts model there is another placement decision: where the expert weights live and which other weights are replicated. Our total-versus-active parameter article explains why selectively executing experts does not eliminate the need to store their weights in an all-resident deployment.

Before doing arithmetic, write down whether you are sizing one distributed model or several copies. Four independent replicas do not give one request automatic access to four times a single device's model capacity.

The ledger belongs to each device

For each GPU, record resident weights, physically allocated KV state, runtime and communication buffers, and any additional reserve. Use the actual representation after loading. A checkpoint's disk size is useful evidence, but conversions and layout can change what remains resident.

Take a deliberately hypothetical device group: four GPUs, each with a usable 48 GiB allocation budget. Assume one dense model with 120 GiB of resident weights, perfectly divided into 30 GiB per GPU. Reserve a fixed 4 GiB per device for all non-KV runtime allocations in this simplified example.

These are teaching inputs, not specifications for a commercial GPU. “GiB” uses binary bytes; do not substitute a product's decimal GB number without conversion. Actual weights can be unevenly distributed, and runtime buffers can change with the workload.

With ideal cache sharding, 24 GiB of logical KV state would add 6 GiB per device. The tempting total is therefore 30 + 6 + 4 = 40 GiB. That answer depends on the cache really being divided four ways.

Two KV heads change the result

For this example, assume conventional head-sharded attention with two KV heads, tensor parallelism of four, and no decode context parallelism. Each KV head is stored on two GPUs.

vLLM's context-parallel explanation describes this limit: once tensor parallelism exceeds the available KV-head count, head-based distribution can replicate cache. The exact behavior remains architecture- and engine-dependent.

Illustrative head replication with tensor parallelism of four, two KV heads and no decode context parallelism. Actual placement depends on the model and engine.

Here, 24 GiB of unique KV data becomes 48 GiB across the device group: two physical copies of each head's state. Each GPU holds 12 GiB rather than the naively predicted 6 GiB.

Logical KV for the group	Naive KV per GPU	KV per GPU with stated replication	Total per GPU: weights + KV + runtime
24 GiB	6 GiB	12 GiB	30 + 12 + 4 = 46 GiB
32 GiB	8 GiB	16 GiB	30 + 16 + 4 = 50 GiB

The first row fits the assumed 48 GiB budget, with 2 GiB remaining per device. The second does not. Naive division would have predicted 42 GiB for the second row and missed the failure.

Even the logical group total looks harmless: 120 GiB of weights, 32 GiB of logical cache and 16 GiB of runtime allocations sum to 168 GiB, below the group's 192 GiB. But the physical cache is 64 GiB, making the actual assumed total 200 GiB. Logical size and allocated size answer different questions.

The KV cache calculation guide supplies inputs for logical state. The next step is applying the deployment's placement and replication rules—not blindly dividing that result by the GPU count.

Can context parallelism recover the space?

Decode context parallelism can partition cache along the token dimension rather than relying only on the number of KV heads. That can address the replication issue described above, subject to model and engine support.

It also changes execution and communication. Reducing duplicated storage is a capacity result; improving response speed requires measurement. NVIDIA's transfer-library explanation describes several distributed inference data paths, including exchanges associated with expert-parallel work. Distributed capacity comes with a data-movement design.

There is no universal rule that the largest parallel group is best. A smaller group might fit and communicate less. A larger group might accommodate a longer context or more requests. A different split might balance weight-heavy and cache-heavy stages. Evaluate the configuration that serves the intended workload, not merely the one that launches successfully.

A practical diagnosis when the model fits but requests fail

First check the failure stage. A failure during loading suggests a different allocation problem from one appearing only after longer prompts or more concurrent requests. Record per-device memory at both points.

Next inspect the actual KV-head count and parallel configuration. Compare cache allocation across devices with the logical payload estimate. Look for replication, layer imbalance and runtime workspaces before assuming a memory leak.

Finally test the largest supported request mix, not only a short prompt on an empty server. Leave room for the runtime behavior observed in that test; the fixed 4 GiB above is not a generally safe allowance.

Korea's HBM developments, including SK hynix's documented HBM4 work, improve the memory hardware available to system designers. Software placement still determines how effectively a deployment uses that hardware. A change reducing duplicated KV may increase useful serving capacity without changing the installed memory.

For me, the next convincing capacity claim should show a device-level ledger beside its aggregate number. Ask which tensors are unique, which are copied, and which request pushed the busiest device closest to its limit. Those answers turn a VRAM total into a deployment plan.