Prefill vs Decode: What Actually Determines LLM Response Speed

Ask an LLM a question and you often see a pause followed by a stream of text. Prefill processes the prompt and builds the KV cache; decode produces subsequent tokens. But the pause is not prefill alone: queueing and other request-handling work also affect when the first token arrives.

The useful starting rule is that prefill often stresses compute while single-stream decode often stresses memory bandwidth. The actual bottleneck depends on the workload and serving system. To diagnose a slow answer, separate first-token delay from later token delivery, then check which resource or scheduling constraint explains the slowdown.

Prefill often stresses compute, while single-stream decode often stresses memory bandwidth. These are workload-dependent tendencies, not fixed rules.

Two Phases, One Request

In a conventional dense autoregressive decoder, prefill can process multiple known prompt positions together. An engine may process the prompt in one prefill operation or split it into chunks. During ordinary decode, each sequence advances by one token per iteration. Linear layers use model weights, while attention accesses that sequence's retained KV state. This description does not cover every optimization or architecture.

How that cache grows and what it costs in bytes is covered in KV Cache Explained and how to calculate KV cache memory. This article picks up on the timing side.

Time to first token (TTFT) measures the interval from a defined request start to the first output token. At the client, it can include request transport, queueing, preprocessing, and prefill. A server-side prefill timer measures only part of that interval.

Inter-token latency (ITL) describes the spacing of later output tokens or streaming responses. Time per output token (TPOT) is often reported as an average after the first token. Check the tool's definition rather than assuming the names are interchangeable: a response chunk can contain multiple tokens. NVIDIA's GenAI-Perf documentation specifies its response-based measurement and token normalization.

A slow TTFT and slow later token delivery can have different causes. Compare measurements with the same request boundaries and workload, and set latency targets for the application rather than borrowing an unrelated benchmark's limits.

From Request Arrival to Streamed Tokens

A simplified request timeline separates waiting from model execution:

1. Arrival and preparation: the service receives and prepares the request.
2. Queueing: the request waits for execution capacity.
3. Prefill: the model processes the prompt, building any required KV state.
4. First output token: the chosen measurement boundary ends TTFT.
5. Decode and completion: later tokens are generated and delivered until the response ends.

A simplified server-side request timeline. Queueing contributes to time to first token; later token delivery depends on decoding and scheduling.

Work can overlap, and cached prefixes can reduce the prefill work required. In ordinary cached decoding, processing the previous output token adds KV state used to predict the next token. The timeline is an accounting aid, not a claim that every server executes five isolated blocks. Hugging Face's cache explanation describes how stored states are reused.

A Worked Timing Example

Assume a hypothetical response contains 101 output tokens, including the first. TTFT is 0.8 seconds, and the remaining 100 token intervals average 30 milliseconds. Using the same client-side measurement boundary throughout:

Total response time = TTFT + (output tokens - 1) × average later-token interval
                    = 0.8 s + 100 × 0.030 s
                    = 3.8 s

Reducing TTFT to 0.3 seconds would cut the total to 3.3 seconds. Keeping TTFT at 0.8 seconds but reducing the later-token average to 15 milliseconds would cut it to 2.3 seconds. These are calculated examples, not benchmark results. They show why a faster start and a faster complete answer are different optimization goals.

The Compute-Bound / Memory-Bound Line Isn't Fixed

The default case

Prefill's linear layers can reuse weights across many prompt positions, giving more arithmetic work per byte moved than a single-token operation. With enough work available, compute throughput can become the limiting resource. Short inputs, inefficient kernels, or other overheads can prevent that outcome; prompt length alone does not prove compute saturation.

At low batch size, dense decode's linear layers reuse each weight across relatively few token positions. Reading weights and retained KV data can therefore dominate execution. As context grows, attention accesses more retained state in a full-attention model. Do not infer constant step time, or a fixed arithmetic intensity for the entire model, from the fact that each sequence advances one token at a time.

What changes at higher batch size

Batching lets a linear operation apply the same weights to more token positions. A roofline analysis in How To Scale Your Model compares compute capability with memory bandwidth to estimate where those operations might become compute-limited. Its crossover depends on hardware, precision, matrix shape, and the assumptions used to count data movement.

This is a prediction for a specified operation, not a universal request-count threshold. More batching can improve weight reuse without making every kernel in a decode iteration compute-bound.

The attention operation that compares a new query with retained keys and combines values is different from the learned projection layers. Independent sequences have distinct KV histories, so adding sequences also adds cache data to access. Linear layers and KV attention can therefore face different limits within the same decode iteration.

Two exceptions worth knowing

A very short prompt may leave too little work to use the accelerator efficiently. At long contexts, the attention portion becomes increasingly important, and its behavior depends on the attention implementation. Calling either case compute-bound or memory-bound requires examining the operations actually running.

The second caveat is theory versus measurement. In Mind the Memory Gap, a 2025 study, researchers found DRAM bandwidth remained a major bottleneck in the large-batch inference configurations they examined, particularly for smaller models. A throughput plateau did not automatically mean the GPU had exhausted its compute capability.

That finding supports profiling the actual deployment. It is not a same-hardware test of every crossover estimate in another source, nor proof that large-batch inference can never become compute-bound.

The phase label remains useful, but it is not a diagnosis. Batch composition, retained context, model architecture, and the serving implementation determine which work dominates. Production concurrency is a reason to measure again, not an automatic reversal of the single-stream pattern.

Why Scheduling Makes This Messier

Disaggregating Prefill and Decode

When prefill and decode share execution resources, a long prefill can delay token delivery for requests already decoding. The effect depends on scheduling and workload; it does not mean every request must experience the same latency spike.

vLLM's experimental disaggregated-prefill documentation describes placing the phases in separate instances to tune TTFT and ITL and control tail latency. It also warns that the documented feature does not improve throughput. Keep that warning attached to this implementation rather than treating it as a law for all disaggregated serving systems. KV transfer and resource allocation must be included in any comparison.

Chunked Prefill: A Lighter Fix

Chunked prefill splits prompt processing into smaller pieces that can be scheduled alongside ongoing decode work. It can reduce the length of interruptions, but the chunk size and scheduling policy trade off prompt completion, token-delivery latency, and throughput.

In an original TNG Technology Consulting report, chunked prefill increased total token throughput by about 50% in a standard vLLM deployment with evenly sized requests. That is their reported observation, not a general multiplier or a measurement performed by AI NodeLab. Benchmark the effect with your own request-length distribution.

Concurrency: Throughput Is Not Individual Speed

Combining requests into a batch can improve weight reuse and aggregate throughput. It does not make additional requests free: each brings computation and cache demand, and an individual user may wait longer even while the server produces more total tokens per second.

Measure both throughput and latency as offered load increases. Queueing, memory traffic, compute, and scheduling can become limiting at different points. A single-user test does not establish the latency that a busy server will deliver.

KV Cache Cost Doesn't Wait for the Crossover

For the same full-attention model and cache precision, one request's logical KV payload grows with that request's retained token count. The total across independent resident requests is the sum of their payloads. Batch size affects that aggregate, not the size of an unchanged individual request.

During decode, attention reads retained KV state and the newly processed position adds state. Many long-context requests can therefore increase memory traffic even if linear-layer weight reuse improves. Physical allocation and traffic also depend on sharing, layout, and implementation; capacity arithmetic alone does not predict a token's latency.

When Your LLM Feels Slow: A Diagnostic Table

Use symptoms to choose a measurement, not to declare a bottleneck before observing it.

Symptom	Possible causes	What to check first
Slow to first token	Long prompt, queueing, preparation or cold-start work	Separate queue time and prefill time; compare client-side and server-side timing
Slow token delivery when running alone	Weight or KV memory traffic; inefficient execution or delivery overhead	Hold model and precision fixed, vary retained context, and compare kernel timing with client-visible gaps
Fine alone, slow under concurrent load	Queueing, prefill/decode contention, or compute and memory limits	Track request load, active batch, queue depth, and tail latency; test one scheduling change at a time
Throughput plateaus with larger batches	Memory bandwidth, compute, or runtime constraints	Profile compute and memory activity rather than assuming a theoretical crossover explains the plateau

The next time a deployment feels slow, record the model and precision, prompt and output lengths, offered load, and timing definitions beside the result. Then change one variable at a time. A faster first token, smoother streaming, and higher server throughput are separate outcomes; the useful optimization is the one that improves the outcome your application actually needs.