Attention, Gently: How Models Decide What Matters
The mechanism behind the magic, explained with zero math.
In the first two parts of this series, you saw that a large language model’s whole job is to guess the next word, and you learned that it chops your sentences into tiny pieces called tokens. The question still hanging in the air is: how does the model figure out which pieces of the whole conversation are relevant when it’s about to pick the next token? That decision, the mechanism that decides what to pay attention to, is what this article explains, in simple terms, with no math.
It’s not magic. It’s a system of asking questions and listening to answers that runs tens of thousands of micro-decisions every second. When you chat with a bot, this system is the reason it can remember that you mentioned “a vegan lasagna” three paragraphs ago and still pull that detail into the current reply. But here’s the part that surprises most engineers: the colorful attention maps you see in demos, those bright lines connecting words, are not actually what the model uses to decide what matters. Those maps are a by-product, not the driver. The real machinery is a set of learned projections that route information, and they work in ways that visualizations often mislead us about. Let’s take it apart.
How does attention decide which tokens matter?
Attention decides which tokens matter by generating a query that describes what the current position is looking for, and comparing it to keys that describe what every other token offers. The model then uses the comparisons to grab a weighted blend of the actual information, the values, from all the tokens it can see. The result is a new representation for the current token that has “listened” to the rest of the sequence in a very deliberate, learnable way.
Think of a crowded room where everyone is a token. Each person carries three things. They have a question they want answered right now (the query). They also wear a name tag that states what kind of information they can provide (the key). And they hold a piece of paper with the actual facts they know (the value). When a person needs to decide what to say next, say, the word “lasagna”, they walk around the room mentally. For every other person, they check how well that person’s name tag matches their current question. If the tag says “I know about ingredients” and the question is “What else goes with spinach?”, the match is strong. The listener then takes the facts from that person, weighted by how good the match was, and blends them into a collective whisper. That blended whisper is attention’s output: a summary of the most relevant information the entire room could give.
In a real transformer, these questions, tags, and facts are all vectors. A small learned projection takes the same input token and creates three different versions of it, one for each role. The dot product between a query and a key gives a raw similarity score. Passing those scores through a softmax turns them into a set of positive weights that sum to one. Finally, the weights act as mixing coefficients on the value vectors. This whole operation, scaled dot-product attention, is the core engine behind every modern language model. The original paper that introduced it is Attention Is All You Need, and the scaling factor, dividing by the square root of the key dimension, is a practical fix that stops the softmax from becoming overly confident before the model has learned anything.
But a single set of questions, name tags, and answers can only capture one type of relationship at a time. A token might need to focus on grammar, topic consistency, and the immediate next word all at once. That’s why attention almost never works alone.
Why does attention use multiple heads?
Multi-head attention lets the model pay attention to several different kinds of relationships at the same time. Instead of running one attention calculation per layer, the model runs many in parallel, each with its own learned projections for query, key, and value, and then combines their outputs. This gives the model the ability to look at the conversation from different angles simultaneously.
Imagine the same room full of people, but this time there are several small groups of specialists. One group is tuned to listen for grammar patterns, another tracks the topic of the discussion, a third spots contradictions. When a person speaks, they listen to all the groups at once and then mix the advice they receive into a single refined statement. That mixing is exactly what the final linear projection in a multi-head attention block does. Each head operates in a lower-dimensional subspace, and the model can dedicate different subspaces to different relational patterns. In practice, researchers have found individual heads that specialize in copying recent words, in resolving pronouns like “it” to earlier nouns, or in detecting quotation boundaries. That’s why editing just a handful of heads can sometimes fix a specific failure without retraining the whole model. The multi-head design is not an optional add-on; even in small models, a single head is too rigid to handle the variety of connections real language demands.
Now, not all relationships are allowed. During generation, the model must never peek at words it hasn’t written yet. That restriction is enforced by masking.
How does attention handle what the model can and cannot see?
Masking stops attention from looking at tokens that should be invisible for the current task. In autoregressive generation, a causal mask blocks every position from attending to future positions. Without this, the model would cheat by reading the answer before it’s generated, and training would collapse.
Implementing the mask is straightforward. Before the softmax, any score that corresponds to a forbidden connection gets a huge negative value pushed into it, effectively minus infinity, so the softmax probability for that connection becomes zero. The resulting attention pattern is an upper-triangular matrix of zeros and non-zero weights, which means each token can only mix information from itself and tokens that appeared earlier in the sequence. Padding masks work the same way. If a batch of inputs has different lengths, the shorter ones get empty filler tokens, and the mask ensures the model never accidentally blends in those empty slots.
When the chatbot talks to you, its decoder layers use causal self-attention over the conversation history built so far. For encoder-decoder architectures, there’s also cross-attention, where the decoder uses its own queries but grabs keys and values from the encoded representation of the user’s prompt. That cross-attention has no causal restriction on the encoder side, so the decoder can freely look back at the entire input at every generation step. But even with perfect masking, attention still has a blind spot: it cannot, by itself, tell whether “the cat sat” and “the sat cat” are the same.
How does attention know the order of words?
Attention knows the order of words because positional information is explicitly injected into the token representations before they reach any attention layer. Without this injection, a bag-of-words would all look the same to the model, and it would confuse “dog bites man” with “man bites dog.”
The original transformer added a fixed sinusoidal pattern to each token embedding, with different frequencies encoding absolute position. Later models switched to learned position embeddings, which the model can tune during training. More recent designs like rotary position embeddings (RoPE) encode relative position directly into the attention score calculation, allowing the model to better handle sequences longer than any it saw during training. This is important because it means the same question-and-answer pattern can work whether you ask it in a short prompt or in the middle of a long conversation. The exact choice of scheme affects how far a model can extrapolate and how it handles distant drafts, but all the schemes solve the same core problem: telling attention who came first so that meaning stays intact.
Positional encoding makes attention position-aware, but it also contributes to the mechanism’s biggest weakness. When the model maps every position to every other position, the cost grows fast.
Why does attention get slow with long conversations?
Standard attention compares every token with every other token, so the number of pairwise comparisons is proportional to the square of the context length. For a 1,000-token prompt, the model must compute one million compatibility scores. For 2,000 tokens, it’s four million. This quadratic growth is the reason very long chats feel sluggish, why models have a context window limit, and why engineers spend so much effort optimizing attention kernels.
The slowness is not just about arithmetic. The real bottleneck is that attention must materialise large intermediate matrices and move them between memory and compute units on a GPU (graphics processing unit). The FlashAttention family of algorithms rearranges the computation to use the GPU’s SRAM more cleverly, tiling and recomputing some values in the backward pass instead of storing them. This can cut memory use and speed up long-sequence training by an order of magnitude. For applications that need even longer contexts, researchers have explored sparse attention patterns that only let tokens attend to a local window plus a few global tokens, or linearised approximations that avoid the full N×N matrix altogether. These are all attempts to get most of the benefit of full attention without the brutal scaling law.
When you actually run a request through a chatbot, all these pieces work together in a pipeline. Let’s walk through what happens during a real generation step.
What does attention actually look like when a chatbot responds?
When you type “Write a short poem about a sleepy cat” and hit enter, the model first converts your text into tokens: “Write”, “ a”, “ short”, “ poem”, “ about”, “ a”, “ sleepy”, “ cat”. Each token gets its embedding plus a positional encoding. As the model begins to produce the poem line by line, attention inside the decoder layers acts like a conductor.
At the moment the model is about to generate the token “dreams”, its attention heads are busy weighing the surrounding context. Some heads will put high weight on the token “poem” to reinforce the task format. Others will attend strongly to “sleepy” and “cat” to keep the subject consistent. Still others will look at the immediate preceding token to decide the probable next syllable. The weights shift token by token, layer by layer, but overall the system routes information so that the generated stream stays on theme. This is why a good chatbot can handle a follow-up question about “fluffy paws” without you repeating “cat” every time. The attention patterns have effectively copied the relevant concept from earlier in the conversation and baked it into the hidden state of the current token.
Now, those bright lines between words that visualisation tools often draw? They are approximations. They show you which tokens received high attention weights, but those weights are not a direct map to what the model actually used. Research has shown that you can change attention patterns dramatically without changing the model output, meaning attention weights are just one piece, a proxy, for the decisions the network makes. So treat those visualizations as hints, not as explanations. The real answer to “how the model decided” is spread across every layer and every head, and it’s still an open research question to trace it fully.
Before we wrap the concepts into a compact reference, let’s address some of the questions engineers inevitably ask once they start peeking under the hood.
| Concept | What it does |
|---|---|
| Query (Q) projection | Encodes what the current token is looking for |
| Key (K) projection | Encodes what a candidate token offers |
| Value (V) projection | Encodes the actual information to be mixed |
| Scaled dot-product | Computes similarity between Q and K, scaled to control saturation |
| Softmax weighting | Converts similarity scores into a probability distribution |
| Multi-head attention | Runs several attention instances in parallel, then merges them |
| Causal mask | Forbids future tokens from being attended to during generation |
| Cross-attention | Lets decoder tokens attend to encoder outputs in sequence-to-sequence tasks |
| Positional encoding | Adds sequence order information so attention can distinguish word order |
| FlashAttention | Algorithm that speeds up attention by keeping data in fast SRAM and reducing memory traffic |
Frequently Asked Questions
Q: Is attention computed for every token at every layer? Yes. In a standard transformer, every token at every layer runs the full multi-head attention block (unless a sparse pattern skips it). This means the raw computational footprint is substantial, but it also allows the model to refine its “focus” at each layer, mixing different levels of abstraction.
Q: Can I use the attention weights to debug why the model gave a wrong answer? Not reliably. Studies like Attention is not Explanation and The elephant in the interpretability room show that attention weights correlate poorly with feature importance, and alternative attention patterns can often yield the same prediction. Use them as a clue, not as a final verdict. Mechanistic interpretability, which looks at the internal circuits, is more direct but harder.
Q: What’s the difference between self-attention and cross-attention in a chatbot? In a decoder-only chatbot (the most common LLM), there is no explicit cross-attention; all attention is self-attention over the concatenated conversation history. In classic encoder-decoder models, self-attention operates within the encoder and within the decoder, and cross-attention connects the decoder to the encoder’s output. For a modern chatbot, the “cross” effect is achieved by having the prompt and conversation all in the same sequence, with causal masking separating the assistant’s turns from earlier input.
Q: Why does my long prompt sometimes cause the model to forget the beginning? Standard attention becomes expensive for sequences much longer than the model’s training context window. Even with optimizations, the architecture may struggle to retain crisp information from thousands of tokens back. Models with positional schemes like RoPE or ALiBi can extrapolate a bit, but the effective range still degrades beyond a point, and key information can get diluted.
Q: Why do models need four sets of projections (Q, K, V, and the output projection) just for attention? Separating Q, K, and V lets each token wear different hats for different roles. A token might advertise itself as a good source of factual information (high key score in that dimension) while asking for syntactic guidance (a different query pattern). The output projection then mixes the heads’ results back into a unified space that the next layer can use. This separation increases the model’s flexibility and makes training more stable.
Test yourself
You’re building a small assistant that answers questions about a user’s past conversations. The system stores the history as a plain text block and feeds it to the model each time. A user says: “Remember that restaurant I mentioned last week?” The model responds with the correct name. But when the conversation grows past about 8,000 tokens, it starts missing earlier references. How would you explain, using only the attention mechanism, why this happens, and what would you try first?
Answer: The model’s forgetfulness is a direct consequence of the quadratic cost and limited context window of standard attention. As the history grows, the softmax in each attention block must spread a fixed probability mass over an increasing number of tokens. Information from many interactions gets compressed into a finite-size representation, and distant tokens’ contribution can fall below the noise floor. Even if the model’s positional encoding permits very long sequences, the attention weights for tokens 8,000 steps back may become so small that the model effectively ignores them. The first practical step is to use a retrieval-augmented approach: instead of feeding the raw history, extract only the relevant sections with a lightweight search and inject them into the prompt. This shortens the effective sequence length and brings the critical tokens back into the model’s attention spotlight. If you must keep everything, moving to a model fine-tuned with a larger context window and using FlashAttention-2 to manage memory might help, but retrieval is almost always the simpler start.
If you want this kind of breakdown every week, how real systems actually work under the hood without the fluff, subscribe to Internals Decoded at internalsdecoded.com. Next in the series: we stop looking at forward passes and dive into the training loop, where attention and billions of predictions collide to shape the model you chat with.
Sources
- Attention Is All You Need
- Self-attention Does Not Need O(n²) Memory (FlashAttention)
- Transformer Architecture (official Tensor2Tensor documentation)
- An Analysis of Multi-Head Attention
- Attention is Not Explanation
- RoPE: Rotary Position Embeddings
- ALiBi: Train Short, Test Long
- The elephant in the interpretability room