KV Cache Offloading: When CPU Memory and SSDs Beat Recomputation

If repeated AI conversations can retrieve their saved context faster than the GPU can rebuild it, storage and cache-management software become more valuable together. That gives Korean memory suppliers a role beyond adding capacity. The condition matters: KV cache offloading helps response latency only when useful reuse outweighs the cost of finding and restoring the data.

For a returning request, the practical question is simple: should the server load a matching prefix from CPU memory or an SSD, or compute it again? Comparing those two times is more informative than comparing the storage capacity printed on two product sheets.

Conceptual alternatives for a returning request with no reusable GPU-resident copy. Actual transfer paths depend on the serving implementation. Based on vLLM and LMCache documentation.

Start with the conversation that comes back

Imagine an assistant working through a long technical manual. It answers one question, waits while the user reads, then receives another question about the same material. The shared opening of the prompt may already have usable key-value (KV) states: intermediate attention data created during the earlier computation.

Keeping those states on the GPU permits direct reuse, but they may be evicted as other requests need the space. An external cache can retain them in another tier. The LMCache quickstart illustrates this lifecycle: an initial request stores state, and a subsequent request retrieves its shared prefix from CPU memory.

This article focuses on restoring reusable prefixes between requests. Moving active layers' cache during every decode step is a different execution strategy, with a different transfer budget. Likewise, storing KV tensors is different from saving the conversation as text and feeding it through the model again.

The vLLM offloading guide describes completed blocks being retained outside GPU memory and brought back on a hit. In its documented multi-tier design, secondary storage reaches the GPU through a CPU staging tier. Other implementations can have different paths. Trace the software's actual route before assigning a bandwidth number to it.

Our SOCAMM2 article explains one CPU-memory format. Extra host memory can provide room for retained state; the offloading policy determines whether that room saves work.

A cache hit must match the computation

“Both requests mention the same document” is too loose a definition of reuse. In ordinary exact-prefix caching, the earlier tokens are part of the identity of the state.

Consider two requests using the same model and a compatible cache representation:

• Request A: unchanged system instructions → the manual → “Explain section one.”
• Request B: unchanged system instructions → the same manual → “Explain section two.”

The unchanged opening is a candidate for reuse, subject to the engine's block boundaries. The new question still needs processing. If Request B instead changes the system instructions at the beginning, the later manual is no longer behind the same prefix. Its identical wording alone does not establish a hit.

The vLLM design document makes this concrete: block identity incorporates both the block's tokens and its preceding prefix, with additional identifiers for features such as adapters and cache isolation. This example concerns exact-prefix reuse, not every proposed method for reusing arbitrary text segments.

Put seconds on both choices

Use a deliberately hypothetical case. A matching prefix needs 4 GiB of transferred KV data. Recomputing that same prefix would take 0.80 seconds in the comparison workload. Finding, arranging and completing the restore adds 0.04 seconds beyond the payload transfer.

Assume no usable copy already exists on the GPU. Hold the model, cache representation and prefix fixed. For this simple calculation, the transfer and extra overhead do not overlap with useful work.

Restore time = payload size ÷ delivered bandwidth + extra overhead

“Delivered bandwidth” means useful payload bytes reaching the destination per second along the actual path. An SSD's advertised read speed is not automatically that rate. CPU staging, contention and the transfer implementation can change the result.

Hypothetical delivered rate	Calculated restore time	Against 0.80 s recomputation
4 GiB/s	4 ÷ 4 + 0.04 = 1.04 s	Restore is 0.24 s slower
8 GiB/s	4 ÷ 8 + 0.04 = 0.54 s	Restore is 0.26 s faster
16 GiB/s	4 ÷ 16 + 0.04 = 0.29 s	Restore is 0.51 s faster

These are AI NodeLab calculations, not CPU-memory or SSD benchmark results. GiB uses binary bytes; the rates use the same unit. Each row describes the same payload under a different assumed delivery rate, not a named product.

AI NodeLab hypothetical calculation: the same 4 GiB payload, 0.04 s additional overhead and no overlap. Delivery rates are assumptions, not CPU-memory or SSD specifications.

For these assumptions, the crossover is:

Delivered bandwidth > 4 GiB ÷ (0.80 − 0.04) s ≈ 5.26 GiB/s

The general form is B > S ÷ (R − L): payload S, avoided recomputation time R and additional restore overhead L. It applies only when R exceeds L. If the overhead alone takes as long as recomputation, raising transfer bandwidth cannot win this simplified comparison.

The threshold moves when the server gets busy

The middle row saves 0.26 seconds of prefix work. If contention reduces its delivered rate from 8 to 4 GiB/s while the other inputs stay fixed, restoring becomes slower than recomputing. More cache capacity has not changed that arithmetic.

Actual engines can prefetch, pipeline transfers or overlap I/O with computation. In those cases, measure the delay exposed on the request's critical path rather than adding times that already overlap. If a measured restore duration already includes lookup and staging, do not add those costs a second time.

This tradeoff also appears in the September 2026 py-kvcache research preprint. Its authors report that recomputation can be faster for short prefixes or fast GPUs, and that transfer granularity and scheduling matter alongside device bandwidth. That supports measuring the complete path; it does not supply a universal crossover for another deployment.

A faster prefix restore is also not an equal percentage improvement in the complete answer. Queueing, the unmatched prompt tail and later token generation still contribute. vLLM's prefix-caching explanation distinguishes saved prompt work from generating new tokens. Our prefill-versus-decode guide explains the corresponding latency boundaries.

Why SK hynix's SALT-KV is worth following

In its September 17, 2026 AI Infra Summit report, SK hynix described SALT-KV—Semantic-Aware Lifecycle Tiering for KV Cache—and an exhibition demonstration using a server equipped with its enterprise SSD.

The company says the approach separates context into segments, evaluates reuse value and storage cost, and selects among HBM, DRAM and SSD tiers. The relevant development is a policy for deciding what deserves storage and where it belongs. The public report establishes a demonstration; it does not establish a named production deployment or a general performance advantage.

There is a broader software connection. NVIDIA's Dynamo discussion describes retaining and sharing KV blocks across memory and storage tiers so another worker can reuse them. Together, these examples make a conditional Korean opportunity plausible: storage hardware and cache-management capability can complement each other when deployments repeatedly reuse the data.

The evidence that would strengthen that view is a matched-workload result showing useful cache hits, faster restoration and acceptable tail latency under load. Little reuse, expensive writes or slower restores would weaken it. Capacity shipments and service benefits should be evaluated separately.

Test the policy, not just the drive

Start with a GPU-only prefix-cache baseline, then test an external tier with the same model, prompt trace, output lengths and offered load. Include returning conversations and changed prefixes; a benchmark that repeats one identical prompt tests only a narrow success case.

Record four things together:

• Useful reuse: how many eligible prefix tokens were actually reused, separated by tier—not just how many requests found any cached block.
• Time saved: prefix recomputation avoided versus exposed restore time, plus first-token and later-token latency.
• Work created: first-fill writes, cache lookups, misses, eviction and traffic competing with other requests.
• Behavior under load: sustained delivery rate and high-percentile latency, including cold-cache starts rather than only warm runs.

A tier may still be useful for capacity even when it fails a particular latency test. Report the benefit actually observed. The result worth publishing is not “we stored more KV,” but “this workload avoided this much computation, paid this much retrieval cost, and met its service target.”