The Life of a Request: What Happens When You Hit Enter

When a user sends a prompt to a large language model API (application programming interface), the server does not simply feed words into the model. The request passes through a pipeline of distinct stages: tokenization splits the text into tokens, the prefill phase processes the entire prompt in one forward pass to build a key-value cache, and then the decode phase generates tokens one at a time, each step reusing that cache. The first token's latency is dominated by prefill. Every later token arrives much sooner, because the model only has to push the newest token through its layers and can reuse the cache for everything before it. Streaming is possible because the server can send each token as it is produced, long before the full response is complete.

During prefill the model processes all prompt tokens in parallel, so the work is compute-bound and keeps the GPU (graphics processing unit) busy with math. Once decode begins, the workload becomes memory-bound. The same GPU that was roaring through matrix multiplies now spends most of its time waiting for data to arrive from memory. This shift explains why a server handles prompt tokens far faster than it generates new ones.

How does the server receive and route the request?

A typical LLM serving stack exposes an HTTP or gRPC endpoint that accepts a JSON (JavaScript Object Notation) payload containing the prompt, sampling parameters, and a streaming flag. When the request arrives, the API server validates the input, assigns a unique request ID, and places the request into a scheduling queue. The scheduler decides when to allocate GPU resources to this request, possibly batching it with others to improve throughput.

In systems like vLLM, the scheduler maintains a waiting queue and a running batch. The running batch holds requests actively being processed on the GPU. The scheduler uses a policy (first come first served by default, or priority based) to move requests from the waiting queue into the running batch, respecting constraints like maximum sequence length and available GPU memory for the key-value cache.

Once the request is admitted to the running batch, the server invokes the model's forward pass. The request's entire lifecycle from this point is managed by the inference engine, which orchestrates tokenization, prefill, and decode steps.

What happens during tokenization?

The raw prompt string must be converted into a sequence of integer token IDs that the model understands. The tokenizer applies the same vocabulary and tokenization algorithm used during training. For most modern LLMs this is a byte-pair encoding (BPE) tokenizer. GPT models and Llama 3 and later use byte-level BPE (Meta's Llama 3 announcement describes a vocabulary of 128K tokens), while older models such as Llama 2 ran BPE through Google's SentencePiece library.

Tokenization splits the text into subword units. For example, the prompt "Write a haiku about coding" might become tokens like ["Write", " a", " ha", "iku", " about", " coding"]. The tokenizer also prepends a beginning-of-sequence token and may add special tokens depending on the chat template. The output is a list of token IDs, plus an attention mask that indicates which tokens are real versus padding.

In high-throughput serving, tokenization happens on the CPU (central processing unit), often in the API server process, using fast libraries such as Hugging Face Tokenizers. The resulting token IDs are sent to the inference engine alongside the request metadata. The inference engine never sees the raw text.

How does the prefill phase work?

Prefill is the first forward pass (one trip of the input through every layer of the network) on the entire prompt. The goal is to compute the key and value tensors for every prompt token and store them in a key-value (KV) cache. This cache will be reused during decode to avoid recomputing attention over the prompt tokens for every new generated token.

During prefill, the model processes all prompt tokens in parallel. The input is a tensor of shape [batch_size, prompt_length]. The attention mechanism computes query, key, and value projections for all positions simultaneously. Because the full sequence length is known, the model can use highly optimized matrix multiplications that fully utilize GPU compute units. Prefill is typically compute-bound, meaning the GPU's arithmetic units are the bottleneck.

The KV cache stores the key and value tensors for each transformer layer and each token position. Part 3, Batching and PagedAttention: How vLLM Serves Hundreds at Once, shows how engines like vLLM store this cache in fixed-size blocks. After prefill, the KV cache contains entries for every prompt token. The model then samples the first new token from the logits (one raw score for every token in the vocabulary) produced at the final prompt position.

How does the model generate the first token?

After prefill, the model has produced a logits vector for the last prompt token. The server applies the requested sampling parameters: temperature (how much to sharpen or flatten the scores; you met it in Careful or Adventurous), top-k (keep only the k highest scoring tokens), top-p (keep the smallest set of tokens whose probabilities add up to p), and possibly a repetition penalty. The logits are turned into a probability distribution, and a token is sampled. This is the first generated token.

The sampling step itself is cheap compared to the forward pass. It runs on the GPU but uses negligible compute. The chosen token ID is appended to the sequence, and the server can immediately send this token to the client if streaming is enabled.

From now on, every subsequent token is generated by the decode phase. The model no longer processes the full sequence from scratch. Instead, it runs a forward pass with only the newest token as input, using the KV cache to attend to all previous tokens. Decode is limited by memory bandwidth: to produce just one token per sequence, each step must read every weight of the model, plus that sequence's KV cache, from GPU memory. Very little math happens per byte read (engineers call this low arithmetic intensity), so the arithmetic units mostly wait. This is also why servers batch many users together during decode: one read of the weights then serves every sequence in the batch. As a conversation grows, reading the KV cache takes a growing share of each step, which is the subject of Part 2.

Why is streaming possible before the full response?

Streaming works because the server does not wait for the entire response to be generated before sending data. As soon as a token is sampled, the server can push it to the client over the open HTTP connection, typically using server-sent events or chunked transfer encoding.

The inference engine runs a loop: prefill once, then repeatedly decode and sample, sending each token as it is produced. The client receives tokens one by one and can display them incrementally. This loop continues until a stop condition is met: the model generates an end-of-sequence token, the maximum token limit is reached, or a stop string is detected.

The server's internal state (the KV cache) persists across decode steps. Each new token extends the cache, and the model's forward pass operates on the growing sequence without recomputing earlier attention. Streaming is a natural consequence of this autoregressive design, not an afterthought.

Quick Reference

Frequently Asked Questions

Q: Why is the first token so much slower than the rest? The first token requires a full forward pass over the entire prompt (prefill), which is compute-intensive, and it may also wait in the queue first. Subsequent tokens only process one new token and reuse the cached key-value states, so each one arrives much sooner.

Q: Can I skip prefill if I reuse a prompt prefix? Yes, for the shared part. Some serving systems implement prefix caching, where the KV cache for a common prompt prefix is stored and reused across requests. vLLM's Automatic Prefix Caching lets a new query skip the computation of the part it shares with an earlier one. Only the new tokens still need prefill.

Q: How does batching affect the life of a single request? A request may wait in a queue until the scheduler admits it. Modern engines then mix requests at every step: new prompts are prefilled while running requests decode, and a finished request leaves the batch at once. Part 3 covers this continuous batching.

Q: What happens if the KV cache runs out of memory? The scheduler preempts some requests: it pauses them and frees their KV cache, or it stops admitting new requests until memory is freed. Current vLLM recomputes a paused request's cache when it resumes (vLLM's tuning guide says the V1 default is to recompute rather than swap); some engines, and older vLLM versions, could copy it to CPU memory instead. Either way, preemption shows up as latency spikes.

Q: Does streaming change the model's output? No. Streaming only changes delivery timing. The model generates tokens identically whether streaming is on or off. The server simply sends tokens as they are produced instead of buffering the full response.

Test yourself

A user reports that the first token of a response takes 2 seconds, but subsequent tokens arrive at 50 ms each. The prompt is 4,000 tokens long. You suspect the prefill phase is the bottleneck. How would you verify this, and what tuning knobs could reduce first-token latency without changing the model?

Answer: First split the 2 seconds into its parts. Time to first token is queue wait plus tokenization plus prefill, and vLLM, for example, reports vllm:request_queue_time_seconds and vllm:request_prefill_time_seconds separately; a profiler such as Nsight Systems or the PyTorch Profiler shows the same on a timeline. If most of it is queueing, the request is stuck behind others: give interactive requests a higher priority (vLLM has a priority scheduling policy) or add capacity. If prefill dominates, the best knob is prefix caching: when the 4,000 tokens start with a shared system prompt, its cache is reused and only the new part is computed. A larger max_num_batched_tokens in vLLM also prefills more of the prompt per step, which helps the first token at some cost to other users' token speed. After that come a shorter prompt, tensor parallelism across more GPUs, and FP8 weights and activations on GPUs such as the H100. Quantizing only the KV cache helps little, because prefill is limited by math, not memory reads.

Next: The KV Cache: Why Long Chats Get Slow and Expensive. It shows what that cache holds, why it grows with every token, and the tricks that keep it small.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• vLLM GitHub repository
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al. 2023)
• Optimization and Tuning (vLLM documentation)
• Automatic Prefix Caching (vLLM documentation)
• FP8 W8A8 (vLLM documentation)
• Introducing Meta Llama 3 (Meta, April 2024)
• Hugging Face Tokenizers library