Embeddings and Vector Search, Demystified
Cosine similarity, dimensions, and what 'semantic' actually buys you.
An embedding model turns each chunk of your handbook into a list of numbers, a vector, so that chunks with similar meaning get similar vectors. Vector search then answers a question by turning it into a vector the same way and finding the chunks whose vectors are closest. This part builds that search for the handbook assistant, measures how fast the simplest version is, and explains when you need the faster, approximate indexes such as HNSW and IVF-PQ that vector databases are built around.
The short version: for up to a few hundred thousand chunks, comparing the question with every chunk takes milliseconds to tens of milliseconds, and that exact search is often all you need. Approximate indexes earn their place at millions of vectors, high query volumes or tight memory budgets.
How do embeddings turn text into a geometric space?
You met embeddings in Embeddings: Meaning as Numbers: a model places each text at a point in a space with hundreds of dimensions, so that "work-from-home policy" and "remote work guidelines" land near each other and far from "lunch menu". Here we go deeper on how that map is used for search.
The map is shaped during training. Embedding models for search are usually trained contrastively: the model sees pairs of texts that belong together, such as a question and the passage that answers it, and is pushed to place them close while pushing unrelated texts apart. Researchers describe a good embedding space by two properties: alignment, meaning texts that belong together land close together, and uniformity, meaning the vectors spread out over the whole space instead of bunching up (Wang and Isola 2020, Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere).
Every embedding model also has an input limit. all-MiniLM-L6-v2, a popular small model from 2021, truncates anything longer than 256 word pieces without telling you (its model card). bge-small-en-v1.5 reads up to 512 tokens (its model card), and OpenAI's text-embedding-3 models accept 8,192 (OpenAI's embeddings guide). The chunk size you chose in Part 2 must fit inside your model's limit, or the end of every long chunk is silently ignored. That is why chunking.py counts tokens with the embedding model's own tokenizer.
Build it: embed the handbook and search it
This is the series' second file, embed.py. It loads the chunks from Part 2's chunking.py, embeds them with bge-small-en-v1.5, and searches them exactly: it compares the question with every chunk. We use bge-small-en-v1.5 because it is small (about 33 million parameters), reads 512 tokens, runs on a laptop's CPU and needs no account. Newer models score higher on benchmarks; see the FAQ for how to choose one.
When we ran it (sentence-transformers 6.1.0, October 2026), it printed:
The first question works even though the handbook never says "from home": the remote work chunk, h4.2, comes first. The second shows the weak spot of meaning-based search. The form number HR-27 means almost nothing to the embedding model, so the vacation chunk edges out the home office chunk that actually mentions the form. Part 5 fixes this with keyword search.
What does cosine similarity actually measure?
Cosine similarity is the cosine of the angle between two vectors. It measures whether two vectors point in the same direction and ignores their lengths. Two passages about working from home point in nearly the same direction and score close to 1. A passage about office snacks points elsewhere and scores lower.
For vectors of length 1, cosine similarity equals the dot product, which is why embed.py normalizes every vector and then simply multiplies: doc_vectors @ query_vector. Many embedding APIs return normalized vectors already; OpenAI's are normalized to length 1. Once vectors are normalized, cosine similarity, dot product and Euclidean distance all give the same ranking.
The ranking is what matters, not the raw score. Scores from different models live on different scales. For "Can I work from home?", the right chunk scored 0.70 with bge-small-en-v1.5, while unrelated chunks scored about 0.5. With granite-embedding-small-english-r2 the right chunk scored 0.88 and unrelated ones about 0.8. With all-MiniLM-L6-v2 the right chunk scored only 0.46 and unrelated ones about 0.1. A fixed cut-off of 0.6 would keep every chunk with the second model and drop the right one with the third. If you filter by score, calibrate the threshold on your own labelled questions, for your model.
When is exact search fast enough?
Exact search, also called brute force, compares the question with every stored vector. It sounds slow, but it is one matrix multiplication, which modern CPUs do very fast. Here is the measurement:
On a laptop with an AMD Ryzen 7 7730U processor and NumPy 2.5, it printed:
So searching 50,000 chunks exactly takes about 4 milliseconds, and a million takes under a tenth of a second. A single 200-page handbook is a few hundred chunks, so for it exact search is instant. Even a company's whole knowledge base of tens of thousands of chunks is comfortably within reach. The time grows in step with the number of vectors, so at tens of millions, or at hundreds of queries per second, or when the vectors no longer fit in memory, you need something faster.
Why not use a classic spatial index, like the trees databases use for maps? Because of the curse of dimensionality. For data spread randomly in many dimensions, the nearest and the farthest points end up at almost the same distance from a query (Beyer et al. 1999, When Is "Nearest Neighbor" Meaningful?). Tree indexes such as k-d trees speed up search by ruling out whole regions, and when every region is about as close as every other, little can be ruled out. Real embeddings are not random, since training groups related texts together, which is why nearest neighbor search still finds meaningful results. But the classic trees lose their advantage, and a different family of indexes took over.
How do approximate nearest neighbor algorithms cheat gracefully?
Approximate nearest neighbor (ANN) algorithms accept that they will sometimes miss a few of the true nearest vectors, and in exchange they look at only a small fraction of the data. Their quality is measured by recall@k: the fraction of the true top-k, as found by exact search, that the index returns.
Careful: this is ANN recall, how close the fast index gets to an exact search. It says nothing about whether the results are relevant to the question. A vector database can report 99 percent recall while your answers are wrong, because the exact top-k itself was wrong. Part 7 measures relevance, also confusingly called recall@k, against questions labelled by people.
The most widely used in-memory ANN index is HNSW, short for Hierarchical Navigable Small World (Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs). It builds a graph in several layers, where each vector is a node linked to nearby nodes. The top layer is sparse and gives long jumps; the bottom layer holds every vector. A search starts at the graph's fixed entry point on the top layer, repeatedly moves to whichever neighbor is closer to the query, drops a layer when it can get no closer, and gathers the best candidates on the bottom layer. It visits only a small part of the graph. A parameter usually called ef or efSearch sets how many candidates the search keeps in play: higher values raise recall and cost time.
Disk-based variants such as DiskANN store a single-layer graph on an SSD and keep compressed copies of the vectors in memory. Its authors reported serving a billion vectors on one machine with 64 GB of memory, at more than 5,000 queries a second with a mean latency under 3 milliseconds and 95 percent or better recall of the single nearest neighbor (Subramanya et al., DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node). That scale is far beyond a handbook, but it is what makes company-wide or web-scale search affordable.
Partition-and-compress methods, of which IVF-PQ (inverted file with product quantization) is the best known, take a different approach. A k-means step divides the space into cells, each with a center. At query time, the search picks the nprobe cells whose centers are nearest the query and scores only the vectors inside them. Product quantization shrinks each stored vector to a short code, often a few dozen bytes instead of the 1.5 KB of a 384-number vector, and scores candidates with these compressed codes, optionally re-scoring the best few with the full vectors. FAISS implements these indexes on CPUs and GPUs (FAISS's index overview). IVF-PQ shines when memory is tight.
This is IVF-PQ search. Every one of these indexes has knobs that trade recall against speed and memory, and the right choice depends on your data size and traffic. Part 4 covers what changes when these indexes live inside a vector database.
What does semantic actually buy us in a RAG pipeline?
Semantic search finds relevant chunks even when the words differ. A user might type "How do I submit a vacation request?" while the handbook says "Request vacation in the HR Portal". Here the words partly overlap, but embeddings also connect "time off" with "vacation" and "work from home" with "work remotely", which a plain keyword search cannot.
That connection is learned, so it is only as good as the training data. For company jargon, internal product names or codes, a general model may not know that two phrases mean the same thing. Fine-tuning the embedding model on pairs of real questions and the passages that answer them can help with this (Sentence Transformers' training overview). Combining embeddings with keyword search, as Part 5 does, is usually the cheaper first step.
Chunking from Part 2 matters here too. An overly long chunk blends several topics, and its vector points in a vague direction. A chunk that is too short may lack the words that say what it is about. Our chunks carry their heading path, "Working arrangements > Remote work", for exactly this reason.
Quick Reference: Key Configurations and Defaults
| Property | Typical Value |
|---|---|
| Common embedding dimensions | 384 (bge-small, all-MiniLM), 768, 1,024 (many open models), 1,536 and 3,072 (OpenAI text-embedding-3), up to 4,096 (Qwen3-Embedding 8B) |
| Input limit | 256 word pieces (all-MiniLM-L6-v2), 512 (bge-small-en-v1.5), 8,192 (OpenAI text-embedding-3) |
| Similarity for normalized vectors | Cosine, which then equals the dot product |
| Exact search, measured | About 4 ms for 50,000 vectors and 85 ms for 1 million (384 dimensions, laptop CPU) |
| One float32 vector, 384 dimensions | 1,536 bytes |
| HNSW search knob | ef or efSearch: higher means more recall, slower |
| IVF-PQ search knob | nprobe: how many cells to scan |
| ANN recall@k | Overlap with the exact top-k; not the same as relevance |
Test yourself
Your handbook assistant uses a 384-dimension embedding model and an HNSW index with efSearch=64 over a company knowledge base of 25,000 chunks. For the question "What is the remote work policy?", the top three chunks are about travel approval, office supplies and the remote work policy itself, in that order. The assistant's answer wrongly claims that remote work always needs manager approval, because it mixed the travel chunk's approval rule into its answer. How would you diagnose and fix this?
Answer: First, check whether the index is the problem: run an exact search for the same question. At 25,000 chunks it takes milliseconds. If exact search gives the same order, the index is fine and the issue is ranking. The right chunk was retrieved, just third, so add a re-ranker that reads the question and each chunk together and reorders them (Part 5). Then tell the model to answer only from chunks that address the question and to cite them, and check the citations (Part 6). If off-topic chunks keep scoring close to relevant ones, try a stronger embedding model or fine-tune one on real question and passage pairs. Do not add a fixed similarity cut-off such as 0.6 without testing it: scores mean different things for different models, so calibrate any threshold on your own labelled questions.
Frequently Asked Questions
Q: Should I always use cosine similarity? For text retrieval it is the usual choice. If your vectors are normalized, the dot product gives identical rankings and is cheaper to compute. Use whatever metric the model's documentation recommends.
Q: How do I choose an embedding model and its dimension?
Start from the retrieval results on the MTEB leaderboard (MTEB leaderboard), then test two or three candidates on your own questions, because leaderboard rank does not always carry over to your documents. As of October 2026, options include small open models such as bge-small-en-v1.5 and granite-embedding-small-english-r2, the open Qwen3-Embedding family, and API models such as OpenAI's text-embedding-3, Cohere Embed, Voyage and Google's Gemini embedding models. Many newer models also let you shorten their vectors while keeping most of their quality, an idea known as Matryoshka representation learning (Kusupati et al. 2022, Matryoshka Representation Learning): OpenAI's text-embedding-3 models take a dimensions parameter for this, and Google's gemini-embedding-001 and the Qwen3-Embedding models were trained for it. Fewer dimensions mean less memory and faster search, at some cost in accuracy, so measure the trade on your own data.
Q: Can I update an ANN index incrementally? HNSW lets you add vectors one at a time, though each insert does real work: it has to search the graph to find the new node's neighbors. Deletes are harder. Most implementations mark a vector as deleted and clean up later, by rebuilding segments or vacuuming, and FAISS's HNSW index does not support removing vectors at all (FAISS's index overview). IVF indexes may need their cell centers retrained if your data shifts a lot. Vector databases hide most of this for you.
Q: How do attribute filters (department, date, permissions) work with ANN? The naive way is to search first and filter afterwards, which can leave you with few or no results when the filter is strict. Better databases apply the filter during the graph search, add extra graph links for filtered fields, or switch to exact search when the filter leaves only a small set. Part 4 covers this in detail.
Q: Is exact nearest neighbor search ever practical? Often. In our measurement, exact search over a million 384-dimension vectors took under a tenth of a second on a laptop, and 50,000 took about 4 milliseconds. Use it for small and medium collections, for checking an ANN index's recall, and whenever a filter narrows the candidates to a small set.
Next: Vector Databases: What Actually Matters When Choosing. It puts these vectors in a database, adds metadata and permission filters, and compares the popular options.
If these deep dives into how real retrieval systems work under the hood resonate with how you build, subscribe to Internals Decoded at internalsdecoded.com. Every article unpacks the mechanics that make modern RAG, search, and ML (machine learning) infra tick.
Sources
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (Wang and Isola 2020)
- all-MiniLM-L6-v2 model card (Hugging Face)
- bge-small-en-v1.5 model card (Hugging Face)
- Vector embeddings guide (OpenAI, read October 2026)
- When Is "Nearest Neighbor" Meaningful? (Beyer et al., ICDT 1999)
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (Malkov and Yashunin)
- DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (Subramanya et al., NeurIPS 2019)
- Faiss indexes (FAISS wiki)
- Training overview (Sentence Transformers)
- MTEB leaderboard (Hugging Face)
- Matryoshka Representation Learning (Kusupati et al. 2022)
