Embeddings: Meaning as Numbers
How machines measure similarity, and why it powers search, recommendations, and memory.
When you ask a chatbot for a pasta recipe, it doesn’t understand “pasta” the way you do. It sees a list of numbers. Those numbers are not random. They are arranged so that “pasta” sits close to “spaghetti” and far from “car.” This is an embedding. It turns meaning into coordinates that a machine can measure, compare, and search. Every language model starts by turning each token into one of these lists, and every time a search box finds a relevant passage or an app recommends a product, embeddings are at work behind the scenes.
If you read The Groundwork, you met this idea as a map of meaning in Turning Meaning Into Places and Words as Places in Meaning. Here we look at how the map is actually built and used.
The machine never learns what “pasta” means. It only learns which words appear in the same kinds of sentences. That statistical shadow turns out to be so rich that it captures everything from synonyms to analogies. The rest of this article unpacks how that happens, step by step, using the same chatbot interactions you already know from this series.
What exactly is an embedding?
An embedding is a list of numbers that represents something discrete, a word, a token, a user ID, a product. Each number is a coordinate in a high-dimensional space. If you pick the right coordinates, similar things end up near each other. That is the entire idea.
Think of a map. A city’s latitude and longitude don’t tell you its name or history. But if you know that Paris is at (48.9, 2.3) and Lyon is at (45.8, 4.8), you can measure the distance and see they are both in France. Embeddings work the same way, except they use hundreds of dimensions instead of two. Every dimension captures some latent feature of the input, learned from data. The model never labels those features. It just arranges points so that words that behave similarly in text end up with similar coordinates.
In a chatbot, every token from Tokens: The Currency of AI gets its own embedding vector. The model then uses those vectors as the starting point for everything else, including the attention you met in Attention, Gently: How Models Decide What Matters. The quality of the embeddings directly determines how well the model can tell that “I need a quick dinner idea” and “fast evening meal suggestions” mean roughly the same thing.
How does a chatbot turn words into numbers?
You already know that the chatbot breaks your message into tokens. For the query “give me a pasta recipe,” GPT-4o’s tokenizer produces five token IDs: [91615, 668, 261, 33847, 14420], one each for “give”, “ me”, “ a”, “ pasta” and “ recipe”. Those integers are just labels. They carry no meaning on their own.
The model has a large table called an embedding matrix. It has one row for every token in the vocabulary. Each row is a vector of, say, 768 numbers. When the model sees token ID 33847, it looks up row 33847 and pulls out that vector. This lookup is the embedding layer. It turns a sequence of token IDs into a sequence of dense vectors that the rest of the network can process.
This lookup is fast, and training can adjust it like any other knob. During training, the model adjusts the numbers in those rows so that the vectors become useful for the task at hand. If the model is trained to predict the next word, then words that lead to similar next-word predictions will gradually drift toward each other in the vector space. The embedding layer itself has no built-in notion of meaning. It is just a giant spreadsheet of numbers that gets updated by backpropagation.
The same mechanism applies to any discrete input. When a recommendation system sees your user ID, it fetches a vector that represents your preferences. Looking up a vector for each symbol is the first step inside these models. A whole document’s embedding, the kind a search engine stores, is then produced by running an embedding model over the document.
Why do similar words end up with similar vectors?
The key insight is the distributional hypothesis. Words that appear in similar contexts tend to have similar meanings. If you see the word “pasta” surrounded by “sauce,” “boil,” “dinner,” and “recipe,” and you see “spaghetti” in the exact same kinds of sentences, then a model that learns from those contexts will place “pasta” and “spaghetti” close together. It doesn’t know what either word means. It only knows they are interchangeable in many sentences.
Early embedding methods made this explicit. They would scan a huge corpus and count how often every pair of words appeared near each other. That produced a giant co-occurrence matrix. Then they would squash that matrix down to a small number of dimensions using a technique like singular value decomposition. The result was a compact vector for each word that preserved the most important co-occurrence patterns. This is like taking a huge spreadsheet of word relationships and compressing it into a few columns that capture the gist.
Word2vec, a method from 2013, learned the vectors on the fly while training a small neural network instead (Efficient Estimation of Word Representations in Vector Space). It did not count everything first. It looked at a small window of words, tried to predict a word from its neighbors (or the neighbors from the word), and adjusted the vectors to get better at that prediction. Over millions of examples, the vectors settled into a geometry where words that predict the same contexts cluster together. Today’s language models learn their embedding table the same way they learn everything else: as part of predicting the next token.
This is why embeddings can solve some analogies. The classic example: start from “king”, subtract “man” and add “woman”, and the result lands closest to “queen”, as the word2vec paper reports. It works best for older word embeddings like these, and not this neatly for every pair. The vectors don’t store royal titles. They store the directions that separate gender and royalty, learned from thousands of sentences where these words appear in parallel roles.
How does the model learn these vectors?
The most famous example is Word2Vec’s skip-gram model with negative sampling. The idea is simple. Take a sentence like “I need a quick pasta recipe.” Slide a window over it. For each center word, like “pasta,” pick a nearby context word, like “recipe.” The model’s job is to decide whether this pair is real or fake.
The model computes a dot product between the embedding of “pasta” and the embedding of “recipe.” A large positive dot product means the model thinks they belong together. A negative or small dot product means they don’t. The model is also shown fake pairs, like “pasta” and “elephant,” sampled at random. It must learn to give high scores to real pairs and low scores to fake ones. When people compare embeddings, they usually report cosine similarity: the dot product of the two lists after both are scaled to the same length, which always lands between minus one and one. The chart below shows the idea with invented numbers.
Over time, a word and the words that tend to surround it get large dot products, and random pairs get small ones. (Strictly, word2vec keeps two tables, one for the word in the center and one for its neighbors.) Because the model sees millions of real and fake pairs, it ends up encoding subtle relationships. “Pasta” and “spaghetti” both turn up next to “recipe,” “boil” and “sauce,” so both get pushed toward the same region of the space, even if they rarely appear in the same sentence.
Levy and Goldberg showed in 2014 that, in an idealized analysis, this training ends up approximating a table of pointwise mutual information (PMI) scores between words and their neighbors, shifted by a constant (Neural Word Embedding as Implicit Matrix Factorization). PMI measures how much more often two words appear together than you would expect by chance. So the vectors are a compressed summary of which words keep each other company, and that is why the numbers capture so much of meaning.
In practice, the model never builds the full PMI matrix. It learns the vectors directly from the data stream, which scales to billions of words. The embedding dimension, typically a few hundred for word2vec style vectors and several thousand inside today’s large language models, is a setting chosen before training that controls how much information can be packed into each vector. More dimensions can capture finer distinctions, but they also cost more memory and computation.
How are embeddings used beyond single words?
A chatbot needs to understand whole sentences, not just individual words. After the token embeddings are looked up, the transformer layers mix them together using attention. The final output is a sequence of vectors, one per token, each now carrying information about the entire sentence.
To get a single vector for the whole sentence, a common trick is to take the vector corresponding to a special token, like [CLS], or to average all token vectors. This sentence embedding can then be compared to other sentence embeddings using cosine similarity. If two sentences have a high cosine similarity, the model considers them semantically close.
This is how a search or help center system knows that “how do I make pasta” and “pasta cooking instructions” are asking the same thing. The query is turned into a sentence embedding, and the system compares it to the embeddings of stored answers or documents. The one with the highest similarity is returned.
Modern sentence embedding models are often trained with contrastive learning. They pass the same sentence through the model twice with a little random noise inside it, or use known pairs of sentences that mean the same thing, and train the model to give each pair nearly identical embeddings (SimCSE: Simple Contrastive Learning of Sentence Embeddings). At the same time, they push embeddings of unrelated sentences apart. This makes the embedding space more uniform and reliable for similarity comparisons.
This same idea powers recommendation systems. A user’s interaction history can be averaged into a user embedding. Items get their own embeddings. The dot product between user and item embeddings predicts how much the user will like the item. Behind every “you might also like” is a nearest-neighbor search in embedding space.
What does this mean for search, recommendations, and memory?
When a search system, or a chatbot with document search (often called RAG, retrieval augmented generation), gets your query, it does not just match keywords. It converts your query into an embedding and finds the stored passages whose embeddings are closest. This is semantic search. A plain chatbot does not do this: it simply reads your tokens. Embedding search is an add on that fetches documents for it. It works even when the words don’t overlap at all. “Inexpensive lodging near the beach” and “budget hotel by the sea” will map to nearby points, so the system retrieves the same results.
Within one conversation, the model simply sees the earlier messages, which Context Windows: The Model's Working Memory explains. Some apps add long term memory across conversations by storing past messages as embeddings and retrieving the relevant ones when you ask something new. The model has no photographic memory. The app has a vector database that finds stored text with a meaning close to your new message.
Recommendation systems use the same principle. Every song, movie, or product gets an embedding learned from user behavior. The system finds items whose embeddings are close to yours. The math is the same as word similarity, just applied to different kinds of entities.
The takeaway is that embeddings are the universal adapter. They turn messy, discrete things, words, sentences, users, items, into a clean, continuous geometry where distance equals similarity. Every language model relies on this trick, and so do most modern search and recommendation systems.
Quick Reference
| Concept | Plain English |
|---|---|
| Embedding vector | A list of numbers that represents a word, token, or item |
| Embedding dimension | The length of that list, from a few hundred to several thousand numbers |
| Embedding matrix | A table with one row per vocabulary item, each row a vector |
| Distributional hypothesis | Words that appear in similar contexts have similar meanings |
| Co-occurrence | How often two words appear near each other in text |
| PMI (pointwise mutual information) | A measure of how much more often words co-occur than chance |
| Negative sampling | Training by contrasting real word pairs with random fake pairs |
| Cosine similarity | A measure of angle between vectors, used to compare embeddings |
| Sentence embedding | A single vector that represents a whole sentence |
| Contrastive learning | Training to pull similar pairs together and push dissimilar pairs apart |
Frequently Asked Questions
Q: Do embeddings really capture meaning, or just word co-occurrence?
They capture co-occurrence patterns that happen to align with meaning. The model never accesses definitions or real-world knowledge. But for many practical tasks, the statistical shadow of meaning is enough. That is why “pasta” and “spaghetti” end up close, even though the model has never tasted either.
Q: Why are embeddings high-dimensional? Couldn’t we use just a few numbers?
A few dimensions can’t capture the many different ways words can be similar. “King” and “queen” are similar in royalty but differ in gender. “King” and “ruler” are similar in role but differ in formality. High-dimensional spaces give the model enough room to represent many independent axes of variation simultaneously.
Q: How do I choose the right embedding dimension?
It’s a tradeoff. Larger dimensions can capture finer distinctions but take more memory and slow down every comparison. Classic word2vec vectors used a few hundred numbers; sentence embedding models today produce anything from a few hundred to a few thousand. Start with what the pretrained model provides, and only shrink it if speed or storage forces you to.
Q: Can I use embeddings for languages other than English?
Yes. The same principles apply to any language. Multilingual models train on many languages at once and learn a shared embedding space where “pasta” in English and “pâtes” in French end up close if they appear in similar translated contexts. This enables cross-lingual search without explicit translation.
Q: Are embeddings the same as the hidden states inside a transformer?
Not exactly. The token embeddings are the input to the transformer. The hidden states are the outputs after attention has mixed the token embeddings with context. Both are vectors, but hidden states depend on context: the vector for “bank” will be different in “river bank” versus “bank account.” The word “embedding” is used for both the fixed lookup vectors at the input and the vectors a model produces for a whole sentence. Only the input ones are the same in every context.
Test yourself
You are building a chatbot that helps users find recipes. A user types “I want something warm and comforting for dinner.” The chatbot retrieves a recipe for chicken soup, which is correct. But when the user types “I need a cozy meal for tonight,” the chatbot retrieves a salad recipe. You suspect the embedding model is not capturing the similarity between “warm and comforting” and “cozy.” How would you diagnose and fix this?
Answer: First, check the cosine similarity between the sentence embeddings of the two queries. If it is low, the model sees them as unrelated. Next, compare each query with the soup recipe’s embedding, and look at which stored recipes sit closest to “cozy meal.” If “cozy” pulls toward the wrong things, the model’s training text likely had few examples where “cozy” appeared in food contexts. To fix this, you can fine-tune the sentence embedding model on a small dataset of paraphrased recipe queries, using contrastive learning to pull “cozy meal” and “warm comforting dinner” together. Alternatively, you can augment the retrieval index with synonyms or add a reranker, a second model that reads the query and each candidate recipe together and scores the match, which is more accurate but slower. The root cause is that the generic embedding space doesn’t specialize in your domain’s phrasing, and fine-tuning or reranking bridges that gap.
Next: Training vs Inference: Why Building Costs Millions and Asking Costs Cents. It shows how all these knobs get set during training, and why using the finished model is so much cheaper.
If you want this kind of breakdown every week, how real AI systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Efficient Estimation of Word Representations in Vector Space (Mikolov et al. 2013), word2vec
- Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al. 2013), negative sampling
- Neural Word Embedding as Implicit Matrix Factorization (Levy and Goldberg 2014)
- GloVe: Global Vectors for Word Representation
- SimCSE: Simple Contrastive Learning of Sentence Embeddings (Gao et al. 2021)
- Attention Is All You Need (Vaswani et al. 2017), token embeddings in the transformer
- PyTorch nn.Embedding documentation
