The Life of a Request: What Happens When You Hit Enter
Tokenization, prefill, decode, stream: the full path from your prompt to the first word back.
When a user sends a prompt to a large language model API (application programming interface), the server does not simply feed words into the model. The request passes through a pipeline of distinct stages: tokenization splits the text into tokens, the prefill phase processes the entire prompt in one forward pass to build a key-value cache, and then the decode phase generates tokens one at a time, each step reusing that cache. The first token’s latency is dominated by prefill, while every subsequent token arrives much faster because only the newest token’s attention must be computed. Streaming is possible because the server can send each token as it is produced, long before the full response is complete.
The surprising part: during prefill, the model processes all prompt tokens in parallel, making it compute-bound and saturating GPU (graphics processing unit) utilization. Once decode begins, the workload becomes memory-bound. The same GPU that was roaring through matrix multiplies suddenly spends most of its time waiting on memory bandwidth. This shift explains why throughput drops sharply when you move from batch prefill to autoregressive generation.
How does the server receive and route the request?
A typical LLM serving stack exposes an HTTP or gRPC endpoint that accepts a JSON (JavaScript Object Notation) payload containing the prompt, sampling parameters, and a streaming flag. When the request arrives, the API server validates the input, assigns a unique request ID, and places the request into a scheduling queue. The scheduler decides when to allocate GPU resources to this request, possibly batching it with others to improve throughput.
In systems like vLLM, the scheduler maintains a waiting queue and a running batch. The running batch holds requests actively being processed on the GPU. The scheduler uses a policy (for example, first-come-first-served or priority-based) to move requests from the waiting queue into the running batch, respecting constraints like maximum sequence length and available GPU memory for the key-value cache.
Once the request is admitted to the running batch, the server invokes the model’s forward pass. The request’s entire lifecycle from this point is managed by the inference engine, which orchestrates tokenization, prefill, and decode steps.
What happens during tokenization?
The raw prompt string must be converted into a sequence of integer token IDs that the model understands. The tokenizer applies the same vocabulary and tokenization algorithm used during training. For most modern LLMs, this is a byte-pair encoding (BPE) tokenizer like the one used by GPT models or SentencePiece for LLaMA.
Tokenization splits the text into subword units. For example, the prompt “Write a haiku about coding” might become tokens like [“Write”, “ a”, “ ha”, “iku”, “ about”, “ coding”]. The tokenizer also prepends a beginning-of-sequence token and may add special tokens depending on the chat template. The output is a list of token IDs, plus an attention mask that indicates which tokens are real versus padding.
In high-throughput serving, tokenization happens on the CPU (central processing unit), often in the API server process. The resulting token IDs are sent to the inference engine alongside the request metadata. The inference engine never sees the raw text. HuggingFace tokenizers
How does the prefill phase work?
Prefill is the first forward pass of the model on the entire prompt token sequence. The goal is to compute the key and value tensors for every prompt token and store them in a key-value (KV) cache. This cache will be reused during decode to avoid recomputing attention over the prompt tokens for every new generated token.
During prefill, the model processes all prompt tokens in parallel. The input is a tensor of shape [batch_size, prompt_length]. The attention mechanism computes query, key, and value projections for all positions simultaneously. Because the full sequence length is known, the model can use highly optimized matrix multiplications that fully utilize GPU compute units. Prefill is typically compute-bound, meaning the GPU’s arithmetic units are the bottleneck.
The KV cache stores the key and value tensors for each transformer layer and each token position. In vLLM’s PagedAttention, this cache is managed in blocks of virtual memory, allowing non-contiguous storage and efficient memory sharing across requests. After prefill, the KV cache contains entries for every prompt token. The model then samples the first new token from the logits produced at the final prompt position. vLLM PagedAttention paper
How does the model generate the first token?
After prefill, the model has produced a logits vector for the last prompt token. The server applies the requested sampling parameters: temperature, top-k, top-p, and possibly repetition penalty. The logits are transformed into a probability distribution, and a token is sampled. This is the first generated token.
The sampling step itself is cheap compared to the forward pass. It runs on the GPU but uses negligible compute. The chosen token ID is appended to the sequence, and the server can immediately send this token to the client if streaming is enabled.
From now on, every subsequent token is generated by the decode phase. The model no longer processes the full sequence from scratch. Instead, it runs a forward pass with only the newest token as input, using the KV cache to attend to all previous tokens. This makes decode memory-bound because the model must read the large KV cache from GPU memory for every new token. The arithmetic intensity is low, so GPU utilization drops dramatically.
Why is streaming possible before the full response?
Streaming works because the server does not wait for the entire response to be generated before sending data. As soon as a token is sampled, the server can push it to the client over the open HTTP connection, typically using server-sent events or chunked transfer encoding.
The inference engine runs a loop: prefill once, then repeatedly decode and sample, sending each token as it is produced. The client receives tokens one by one and can display them incrementally. This loop continues until a stop condition is met: the model generates an end-of-sequence token, the maximum token limit is reached, or a stop string is detected.
The key insight is that the server’s internal state (the KV cache) persists across decode steps. Each new token extends the cache, and the model’s forward pass operates on the growing sequence without recomputing earlier attention. Streaming is a natural consequence of this autoregressive design, not an afterthought.
Quick Reference
| Property | Typical Value |
|---|---|
| Tokenizer algorithm | BPE (GPT-2/3/4), SentencePiece (LLaMA) |
| Prefill compute characteristic | Compute-bound |
| Decode compute characteristic | Memory-bound |
| KV cache storage | Per-layer key/value tensors, stored in GPU memory |
| Streaming protocol | SSE or chunked transfer encoding over HTTP |
| First token latency | Dominated by prefill time |
| Per-token latency after first | Dominated by memory bandwidth |
- Processes all prompt tokens in parallel
- Compute bound, saturates GPU utilization
- Populates the entire KV cache
- Latency scales with prompt length
- Processes only the latest token
- Memory bound, limited by cache reads
- Updates the KV cache incrementally
- Per token latency is roughly constant
Test yourself
A user reports that the first token of a response takes 2 seconds, but subsequent tokens arrive at 50 ms each. The prompt is 4,000 tokens long. You suspect the prefill phase is the bottleneck. How would you verify this, and what tuning knobs could reduce first-token latency without changing the model?
Answer: First, measure GPU utilization during prefill using a profiler like Nsight Systems or PyTorch Profiler. If the GPU is at high compute utilization and the prefill kernel dominates the timeline, the bottleneck is confirmed. To reduce latency, you can increase the prefill batch size if the serving system supports continuous batching, allowing the GPU to process more tokens in parallel. You can also enable FP8 or INT8 quantization for the KV cache to reduce memory traffic, or use flash attention to lower the memory footprint and improve prefill throughput. If the serving system supports prefix caching (e.g., vLLM’s automatic prefix caching), reusing KV cache entries from previous requests with the same prompt prefix can skip prefill entirely for shared parts. Finally, adjusting the GPU’s power limit or clock speeds can squeeze out more prefill performance if thermal headroom exists.
Frequently Asked Questions
Q: Why is the first token so much slower than the rest? The first token requires a full forward pass over the entire prompt (prefill), which is compute-intensive. Subsequent tokens only process one new token and reuse the cached key-value states, making them much faster.
Q: Can I skip prefill if I reuse a prompt prefix? Yes. Some serving systems implement prefix caching, where the KV cache for a common prompt prefix is stored and reused across requests. This turns prefill into a near-instant operation for the cached portion.
Q: How does batching affect the life of a single request? A request may wait in a queue until the scheduler groups it with others. During prefill, all requests in the batch are processed together, sharing GPU compute. During decode, requests may be batched dynamically as they generate tokens at different rates.
Q: What happens if the KV cache runs out of memory? The scheduler will either preempt (evict) some requests, swapping their KV cache to CPU memory, or it will refuse to admit new requests until memory is freed. This can cause latency spikes or out-of-memory errors.
Q: Does streaming change the model’s output? No. Streaming only changes delivery timing. The model generates tokens identically whether streaming is on or off. The server simply sends tokens as they are produced instead of buffering the full response.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com. Next in this series: we’ll open up the KV cache and see exactly how PagedAttention manages memory block by block.
Sources
- vLLM GitHub repository
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
- HuggingFace Tokenizers library