Words as Places in Meaning

A quick note on how we teach here. You won't find equations or numbers in this series, and that is on purpose. The goal is understanding, not arithmetic, so every idea is explained in plain words and simple pictures. When you want the math later, the rest of Internals Decoded is ready for you.

Welcome back. You have pulled language apart into tiny pieces, and you know a model learns from examples. Now we do something almost magical. We turn those dry little pieces into a rich sense of meaning.

The idea is simple. Every piece of language gets its own spot on a huge, invisible map. Pieces that work in similar ways end up close together. This map is how the model begins to grasp what words are about.

A seating chart for words

Imagine you are organizing a big celebration dinner. You have a long guest list and a room full of tables. Your job is to decide where everyone sits. You do not just scatter name cards randomly. You think about who gets along. You put cousins who love sports together. You seat the dessert enthusiasts near each other. Suddenly the seating chart itself carries meaning. Even a stranger looking at the chart could guess a lot. If two names sit side by side, they probably share an interest. If a group clusters in one corner, they might be from the same family or workplace. The arrangement tells a story.

This is exactly how a language model starts to understand words. It builds a seating chart for every token it knows. A token that often appears in the same kind of sentence as another token gets a place nearby. Tokens that never appear together end up far apart. The chart is not made by a person. It is learned, all by itself, during training.

Turning tokens into neighbors

You remember tokens. They are the little word pieces we met in the last piece. A model sees a sentence not as whole words but as a stream of tokens. At first, each token is just a label with no meaning attached. The model’s job during training is to figure out where each token belongs on the great meaning map.

It does this by watching example after example. Every time two tokens show up in similar surroundings, the model nudges their spots a little closer. If the tokens “happy” and “joyful” both appear next to “she felt” again and again, their positions drift together. “Cat” and “dog” might settle near each other too. They are not the same, but they share a neighborhood because both are pets, both can sit on a lap, both appear with words like “feed” or “stroke.” Over countless examples, the map organizes itself. Tokens that mean something alike become neighbors.

This is not done by the model understanding feelings or imagining furry animals. It is pure pattern. The model only sees which tokens travel together. But the result is a map that reflects a surprising amount of human meaning. Direction on the map can stand for something like “how large” or “how positive.” Tokens on one side might relate to warmth and kindness. Tokens far away might relate to cold or sharpness. The model never learns these labels, but it learns the layout, and that layout turns out to be useful.

A map with many directions

Earlier in our journey we looked at patterns in just two directions. We could draw them on a flat page. That was perfect for seeing how numbers could capture an idea. But language needs a much richer space. A flat map is too simple to capture all the subtle ways words relate. So the model gives every token not just a left and a right coordinate but a long list of tiny nudges that reach out in many directions at once. The list might stretch in so many dimensions you could never picture them all. That is fine. We do not need to draw it. We only need to know that similar tokens have similar lists, and that moving through this space lets the model do its work.

Think of it like a city. A street map has two dimensions. But the real life of a city has more. One neighborhood might be known for music, another for food, another for quiet parks. You could not capture all that on a flat grid. You would need many different scales that overlap. The meaning map works the same way. Every token’s address is a rich bundle of hints across many invisible dimensions. Those hints let the model sense not just which tokens are neighbors but exactly how they differ.

Naming the map

A long time ago, in this series, we talked about turning meaning into places. That was the seed of the idea. Now we have grown it into something solid. In the world of AI, these learned positions have a name. They are called embeddings. When they belong to words or tokens, people say “word embeddings.” Next time you hear that phrase, you will know exactly what it means. A word embedding is a little piece of language placed onto a vast meaning map so that neighbors share meaning. It sounds fancy, but it is really just a spot on a seating chart, a spot the model found all by itself.

Where this is heading

Now every token has a home on the meaning map. When the model sees a fresh sequence of tokens, it looks at their positions and guesses where the next spot should be. It can pick a token that sits right in the middle of the most likely neighborhood. That is the safe, obvious choice. But it could also wander a little and pick something a bit farther away, something less expected. How bold should the model be? Should it stick to the safest guess every time, or should it take a risk and surprise us? That single choice, between caution and creativity, is a knob we can turn. And that is exactly what we will explore in the next piece.