Turning Meaning Into Places
Machines handle meaning by turning things into places, so similar things sit close together. Meet embeddings.
A quick note on how we teach here. You won't find equations or numbers in this series, and that is on purpose. The goal is understanding, not arithmetic, so every idea is explained in plain words and simple pictures. When you want the math later, the rest of Internals Decoded is ready for you.
Welcome back. Last time, we saw that a pattern is something that repeats in a way you can notice and expect, enough to sort new things or guess what comes next. Once a machine can spot a pattern, it can tell the difference between a cat and a cup, or between a happy sentence and an angry one. Today we ask a new question. How can a machine handle something as slippery as meaning when all it has to work with is on and off?
A machine handles meaning by turning each thing it knows into a place on a map. Similar things end up close to each other. Unalike things end up far apart. Closeness on that map becomes the machine’s way of saying, “These two belong together.”
A supermarket for meaning
Think of the last time you walked into a large supermarket. You probably did not stop and read a manual. You just knew without thinking that milk would be somewhere near the cheese and the yogurt. Baking flour would be near the sugar and the chocolate chips. Nobody would put the toothpaste next to the oranges. The whole store is arranged so that things that go together live near each other. The arrangement itself tells you something about what those things are.
A library does the same. Travel guides sit with travel guides. Cookbooks share a shelf with other cookbooks. If you pulled a book at random and looked at its neighbors, you could guess its topic before even reading the title. That is because the shelf position carries meaning. The meaning is “this book is about food,” not because a librarian whispered that to the shelf, but because all the other food books are right there.
A machine builds exactly this kind of map, but inside its own world of switches. It takes every word, every picture, every sound it can learn about and gives each one a little spot on a giant imaginary map. The map is not flat like a supermarket floor. It is more like a huge, open field that stretches in many directions at once. You cannot draw it on a piece of paper because it has too many dimensions, but you can still picture the idea. Each thing sits somewhere.
How words find their place
Picture that huge empty field. At first, the machine scatters every word across it at random. “King” might land right next to “banana.” But by the end, the whole royal court is clustered in one corner: “king,” “queen,” “prince,” “princess.” On the far edge of the field, you find “banana,” “apple,” and “grape” huddled together. In another area, you see “run,” “jump,” and “walk” all crowded close.
Why did the machine put them there? Not because it opened a dictionary. Not because someone told it that a king is a ruler. The machine has never understood anything the way you and I do. It only watched many, many sentences. It noticed that the words found around “king” were very often the same words found around “queen”: “throne,” “crown,” “royal,” “reigned.” The same kind of other words surrounded them both. “Banana,” on the other hand, showed up with “smoothie” and “yellow” and “peel,” almost never with “throne” or “crown.” Over time, the machine nudged its little pins around the field until words that keep the same company, meaning they turn up surrounded by the same kinds of words, ended up as neighbors. Closeness on this map became a stand in for similarity of meaning. The map turned into a map of meaning. How a machine does this nudging all by itself is a story for a later piece. For now, picture the pins slowly drifting into place.
The machine never “knows” what a king is. But its map holds “king” near “queen” and far from “banana,” and that turns out to be very useful, as you will see in a moment.
The names people give those places
Now you have a name for a spot on this map. People call that position an embedding. When you hear someone say that a word has been turned into an embedding, they just mean that the machine gave it a place on the map of meaning. The idea of a position itself, the little arrow that points from the center of the field to that exact spot, is called a vector. Inside the machine, each place is written down as a list of numbers, one for each direction of the field, a bit like a very long street address. That list is the vector. Both words, embedding and vector, are used constantly in AI. But now you know they simply mean: a place on the map.
Almost anything can get an embedding. A whole sentence can have an embedding. A photograph of a cat can have an embedding. In each case, the machine looks at the thing, decides where on the map it belongs, and plants it there. Things that are alike end up as neighbors.
This simple sense of near and far is already at work around you. A search box can use a map like this to find “low cost airfare” when you typed “cheap flights,” even though the words do not match, because the two sit close together on the map. Apps that suggest something similar to what you just read or watched can work the same way: they look for things that sit close to what you chose. OpenAI's guide to embeddings lists search and suggestions among their main uses.
Where this is heading
The machine can now capture meaning without ever understanding a single word. It does this by giving each thing a home on a map, a home chosen purely by watching what turns up in which surroundings. Combine that idea with the patterns we talked about earlier, and the machine gains a startling new ability. It can look at a half finished sentence and guess which word is likely to come next. This simple trick, guessing what comes next, is the engine inside the chatbots and writing assistants you hear about today.
Test yourself
On a map of meaning, which pair would sit closer together: “cat” and “kitten,” or “cat” and “teapot”? What would the machine have noticed to put them there?
Answer: “Cat” and “kitten” would sit much closer. The machine never learned what a cat is. It noticed that the words found around “cat” and the words found around “kitten” are very often the same: purr, fur, whiskers, feed, lap. Because the two keep the same company, their places drifted close together. “Teapot” turns up among quite different words, like pour, boil, cup and lid, so it settled far away. Notice that the two words never had to appear side by side. What mattered was that they turn up in similar surroundings.
Next: Guessing What Comes Next. It shows how a machine reads a sentence so far, scores every word it knows, and picks a likely next one, with the map of meaning helping along the way.
