Attention, Gently: How Models Decide What Matters

In the first two parts of this series, you saw that a large language model's job is to guess the next token, and that it chops your text into those small pieces first. The question still hanging in the air is this: when the model is about to pick the next token, how does it work out which of all the earlier pieces matter? The mechanism that decides is called attention, and this piece explains it in plain words. If you came here from The Groundwork, you met attention briefly in Words as Places in Meaning, as the way a model decides which surrounding words to mix in. Here is how it actually works.

It is not magic. It is a system of asking questions and listening to answers, repeated a huge number of times for every token. When you chat with a bot, it is the reason the model can pick up that you mentioned "a vegan lasagna" three paragraphs ago and pull that detail into the current reply.

One thing surprises many people. The colorful attention maps you see in demos, with bright lines connecting words, do show real numbers the model uses. But those numbers are only one small part of the whole computation, so the maps are a poor explanation of why the model answered the way it did. We will come back to that at the end. First, let's take attention apart.

How does attention decide which tokens matter?

Attention gives each token a question that describes what it is looking for right now, and compares that question with a label on every other token that describes what it can offer. The model then takes a blend of the actual information from all the tokens it can see, mixing in more from the tokens whose labels matched the question best. The result is an updated version of the token that has "listened" to the rest of the text.

Think of a crowded room where everyone is a token. Each person carries three things. They have a question they want answered right now (the query). They wear a name tag that says what kind of information they can provide (the key). And they hold a sheet of paper with what they actually know (the value). When a person needs to update their understanding, they walk around the room. For every other person, they check how well that person's name tag matches their own question. If the tag says "I know about ingredients" and the question is "What else goes with spinach?", the match is strong. The listener then takes the facts from each person, more from the strong matches and less from the weak ones, and blends them together. That blend is attention's output: a summary of the most relevant information the whole room could give.

In a real transformer, these questions, name tags and fact sheets are lists of numbers. You met lists like these in The Groundwork as places on a map of meaning, and Embeddings: Meaning as Numbers, the next part, goes deeper. The model makes all three from the same token, using three sets of knobs learned in training. Then it scores how well each name tag matches the question, turns the scores into percentages that add up to 100%, and blends the fact sheets in those proportions. This design comes from the paper that introduced the transformer, Attention Is All You Need.

A single set of questions, name tags and fact sheets can only capture one kind of relationship at a time. A token might need to follow grammar, keep to the topic and track the word just before it, all at once. That is why attention almost never works alone.

Why does attention use multiple heads?

Multiple heads let the model attend to several kinds of relationships at the same time. Instead of one round of attention per layer, the model runs many side by side, each with its own knobs for questions, name tags and fact sheets, and then combines what they found. Each of these is called a head.

Imagine the same room, but now there are several small groups of specialists. One group listens for grammar, another tracks the topic, a third keeps an eye on what was just said. Each person listens to all the groups at once and mixes their advice into one updated understanding. Researchers who study real models have found heads with jobs like these. In BERT, a well known model from 2018, some heads pick out the object of a verb or link a pronoun to the noun it refers to (What Does BERT Look At? An Analysis of BERT's Attention). In translation models, a few specialized heads do most of the work: in one study, 38 of the 48 heads examined could be switched off with almost no loss in quality, and the specialized ones were the last to go (Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned).

Not every connection is allowed, though. When the model writes, it must never peek at words it has not written yet.

How does attention handle what the model can and cannot see?

Each token may look only at itself and the tokens before it, never at the ones after. This rule is called the causal mask. It matters most during training, when the model practices on complete texts. Without the mask, the model could simply read the next word instead of predicting it, and it would learn nothing useful. In practice, the scores for any later token are pushed so low before they become percentages that their share comes out as zero.

When a chatbot answers you, every layer uses this kind of attention over the whole conversation so far: your messages, its earlier replies and the reply it is writing. The original transformer was built for translation and also had a variant called cross-attention, which lets the half that writes the translation look at the half that read the original sentence. Chatbots built the way most of today's LLMs are do not need it.

Even with the mask in place, attention has a blind spot of its own. By itself, it cannot tell "the cat sat" from "the sat cat".

How does attention know the order of words?

Attention knows the order of words because the model adds information about position. Without it, the tokens would be a bag of words with no order, and "dog bites man" would look the same as "man bites dog".

The original transformer added a fixed wave pattern to each token's numbers, a different one for each position. Later models learned their position signals during training instead. Many of today's models use rotary position embeddings, or RoPE, which build the distance between two tokens right into the attention score (RoFormer: Enhanced Transformer with Rotary Position Embedding). That way, the same pattern of questions and answers can work whether it appears near the start of a prompt or deep in a long conversation. Stretching a model to texts longer than the ones it was trained on still takes extra techniques and more training, which Context Windows: The Model's Working Memory covers. Whatever the scheme, all of them solve the same problem: telling attention which tokens came first, and how far apart they are, so that meaning stays intact.

Position makes attention aware of order, but it does not fix attention's biggest weakness. Comparing every token with every other token gets expensive fast.

Why does attention get slow with long prompts?

Standard attention compares every token with every token that came before it, so the number of comparisons grows with the square of the length. Double the length of a prompt and the number of pairs roughly quadruples. A 1,000 token prompt already involves on the order of a million pairs in every layer. This growth is one reason very long prompts are slow and costly, and why engineers work so hard on faster attention.

Engineers have found clever ways to do exactly the same calculation with far less shuffling of data around the chip. The best known, FlashAttention, trained models up to about three times faster in its authors' tests and uses much less memory (FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness). Other designs let each token look at only some of the others, trading a little quality for speed.

When you send a request to a chatbot, all of these pieces work together. Here is what that looks like.

What does attention actually look like when a chatbot responds?

When you type "Write a short poem about a sleepy cat" and press enter, the model first turns your text into tokens: "Write", " a", " short", " poem", " about", " a", " sleepy", " cat". Each token gets its list of numbers, and the model keeps track of each token's position. As the model starts writing the poem, attention inside every layer acts like a conductor.

Picture the moment just before the model writes the word "dreams". Across its many heads, attention is weighing the earlier tokens. Some heads might lean on "poem" to keep the format. Others might lean on "sleepy" and "cat" to stay on the subject. Others might look at the word just before, to keep the sentence flowing. The weights shift with every token and every layer, but together they keep the poem on theme. It is also why a good chatbot can handle a follow up question about "fluffy paws" without you repeating "cat": attention pulls the relevant information from earlier in the conversation into the numbers of the token being written now.

So what about those bright lines between words in visualization tools? They show real weights: how much one token mixed in from each other token, in one head of one layer. But a model has many heads and many layers, and much of the work happens outside attention, so the weights alone rarely explain an answer. Research has found that quite different attention patterns can lead to the same prediction (Attention is not Explanation). Treat the maps as hints, not explanations. Fully tracing how a model reached an answer is still an open research question.

Before we wrap up, here is a compact reference, then some questions people ask once they start looking under the hood.

Frequently Asked Questions

Q: Is attention computed for every token at every layer? In a standard transformer, yes. Every token runs through attention in every layer. That takes a lot of computation, but it lets the model refine what each token "knows" layer by layer, from simple patterns near the bottom to richer meaning higher up.

Q: Can I use the attention weights to find out why the model gave a wrong answer? Not reliably. Research has found that attention weights often disagree with other measures of which words mattered most, and that very different patterns can give the same prediction (Attention is not Explanation). Other researchers argue that methods built to score each word's influence on the output answer that question better (The elephant in the interpretability room). Use attention as a clue, not a verdict.

Q: Why does a long prompt sometimes seem to make the model forget the beginning? Mostly because attention spreads thin. Every token has a hundred percent of attention to share out, so the longer the text, the smaller the share any one early detail can get. Models also tend to use the start and the end of a long prompt better than the middle, and many chat apps quietly trim or summarize old messages. Context Windows: The Model's Working Memory covers all three and what to do about them.

Q: Why does each token need three different versions of itself? Because one token plays different roles at the same time. It might offer itself as a good source of facts (its name tag) while looking for grammar clues (its question), and what it passes on when chosen (its fact sheet) is different again. Keeping the three separate gives the model the freedom to learn each role well.

Test yourself

You are building a small assistant that answers questions about a user's past conversations. It stores the whole history as one block of text and hands it to the model with every request. A user asks, "Remember that restaurant I mentioned last week?", and the model names it correctly. But once the history grows to tens of thousands of tokens, it starts missing earlier details. Using what you know about attention, explain why, and say what you would try first.

Answer: As the history grows, each token's attention is shared across far more tokens. Every token has the same hundred percent to divide, so a detail mentioned once, long ago, gets a smaller and smaller share and becomes easier to miss. Models also tend to make less use of the middle of a long text than of its start and end, and a restaurant named last week sits right in that middle. The window is not the problem: the text still fits, but the detail is easy to overlook. The first thing to try is retrieval: instead of sending the raw history, search it for the passages that match the user's question and put just those next to the question. That makes the text short again and puts the key tokens where attention can find them. Summarizing old conversations into short notes helps too.

Next: Embeddings: Meaning as Numbers. It shows how each token's list of numbers is learned, and how the same idea lets search find meaning instead of matching words.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Attention Is All You Need (Vaswani et al. 2017)
• What Does BERT Look At? An Analysis of BERT's Attention (Clark et al. 2019)
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (Voita et al. 2019)
• RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al. 2021)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al. 2022)
• Attention is not Explanation (Jain and Wallace 2019)
• The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? (Bastings and Filippova 2020)