Disaggregated Inference: When Splitting Prefill and Decode Helps

I would judge prefill-decode separation by the number of answers that meet the application's latency limits. Separating prompt processing could protect ongoing answers, but I would want to see that gain survive KV transfer and the queues on both sides. More specialized GPU roles are useful only if they improve the service within its resource budget.

Disaggregated inference runs the phases on separate workers. It adds a handoff that a colocated request does not need in the same way. Judge that trade with an equal-resource test and count requests that meet both the first-token and streaming targets.

A fast start can hide a slow handoff

Prefill processes input and creates KV state; subsequent decoding uses that state to continue generation. Our prefill-versus-decode guide separates the phases and their timing boundaries. This article asks a narrower deployment question: when is assigning them to different workers worthwhile?

The vLLM disaggregated-prefill documentation describes independent tuning of time to first token, or TTFT, and inter-token latency, or ITL. It also explains the goal of controlling interruptions to decoding. The feature remains labeled experimental, and the page explicitly warns against expecting a throughput improvement from that documented feature.

That warning should not be turned into a law about every serving architecture. Equally, a favorable result from a different system cannot establish a benefit in this one. Identify the implementation and the outcome being measured.

A particularly revealing failure is an early first token followed by an awkward pause. The vLLM–TileRT integration provides a concrete first-token boundary: its prefill worker emits the first token before the decode engine takes over. Depending on the implementation, prefill can deliver the first token before the decode worker finishes receiving the state needed to continue. TTFT alone can then look healthy while the reader waits. Include the gap into continuing generation in the client-side trace.

Conceptual prefill-to-decode handoff. First-token delivery and transfer overlap vary by implementation; the complete user-visible latency path must be measured.

Where the extra work goes

A split deployment needs routing, KV transfer and separate scheduling decisions. NVIDIA's NIXL discussion describes the transfer library's role in moving data among workers and memory tiers. Efficient transport is an enabling component; its existence does not remove the transfer from the latency budget.

Pool balance matters as much as the link. Too much prefill capacity can deliver work faster than decode can consume it. Too little can leave decode resources waiting. This is a queueing consequence of the split, not proof that either phase always has one fixed hardware bottleneck.

Distinguish this handoff from reloading a previously saved prefix. Here the immediate question is moving newly computed state from the worker that processed the prompt to the worker continuing the answer. Persistent offloading and reuse can be combined with disaggregation, but they need separate accounting.

Same raw throughput, different useful service

Use an invented comparison with four GPUs in each deployment, the same request trace and the same output-quality requirement. Both systems complete 100 requests during a 60-second measurement window.

For this example, a successful request must have TTFT of at most 2 seconds and no client-observed gap between successive output tokens above 100 ms. These are chosen teaching targets, not recommended universal limits. Count each request once, and only when it meets both conditions.

Hypothetical outcome	Colocated deployment	Split deployment
GPUs allocated	4	4
Completed requests in 60 s	100	100
Requests meeting both latency targets	60	90
Raw completions per second	1.67	1.67
Latency-qualified requests per second	1.00	1.50

The last row is a simple request-level goodput measure: successful completions divided by elapsed time. The split system has no higher raw completion rate in this example, but it delivers more responses within the chosen limits. No inference benchmark was run to produce these numbers.

The reverse result is possible. If handoff delays cause more requests to miss their target, separation can lower goodput despite smoother GPU utilization. The table is a worksheet for reporting either outcome, not a prediction favoring disaggregation.

Real streaming interfaces may group several tokens into one response chunk. Record whether the timing tool measures tokens or chunks; do not present chunk spacing as a directly observed per-token trace. Keep the definition unchanged between systems.

Run three configurations, not just two

A useful test includes a colocated baseline, a tuned chunked-prefill configuration, and the split configuration. Chunking offers another way to reduce long prompt-processing interruptions without assigning the phases to completely separate worker pools.

Keep total GPUs, hardware, model representation, prompt/output lengths and offered load comparable. Specify the prefill/decode allocation in the split run. If it uses extra equipment, report that as a different resource budget rather than hiding it inside a software speedup.

Warm-cache runs answer a different question from cold starts. Use the same cache state policy across configurations, disclose the measurement window, and include timeouts and failures. A comparison that counts only completed, successful requests can erase the hardest part of the workload.

Record first-token timing, continuing-token gaps, transfer delay, queue depth and the number meeting the complete service target. Then vary the prompt-length mix or arrival rate. The point is to find the region in which the design holds up, rather than announce a winner from one trace.

What changes when images enter the prompt?

Text-only prefill/decode separation is not the whole multimodal pipeline. NVIDIA's September 2026 EPD discussion treats image encoding as another separable stage and explains that results depend on media load and output characteristics.

Keep that as a scope boundary. A multimodal result includes work absent from a text-only test; transferring visual representations is not interchangeable with transferring a language model's KV state. A claimed gain needs a description of which stages were moved.

For Korea's memory industry, the relevant issue is the placement of memory demand across these stages. SK hynix's FMS 2026 portfolio spans HBM, DRAM and storage, but a software diagram alone cannot establish how much of each a deployment buys or which supplier wins.

Before I would draw that conclusion, I would ask for the hardware allocation and memory footprint of each worker pool at the target goodput. Separating stages may improve utilization, increase duplication, or shift where capacity is needed. The evidence should show which happened.

The deployment worth keeping is the one that delivers more acceptable answers within the stated resource budget. Measure that across the handoff, not just on either side of it.