Embeddings and Vector Search, Demystified
Cosine similarity, dimensions, and what 'semantic' actually buys you.
Embedding models convert your company handbook’s text chunks into high-dimensional vectors, and approximate nearest neighbor (ANN) indexes like HNSW or IVF-PQ retrieve the most semantically similar chunks in milliseconds. Cosine similarity gives this search its discriminative power by measuring angular closeness, not just keyword overlap. Together they replace rigid symbolic matching with geometric proximity.
The curse of dimensionality tells us that in high dimensions, all points look equally far apart. Yet embedding spaces, trained with contrastive objectives, actually make semantically related chunks physically close. ANN algorithms then cheat gracefully, trading perfect recall for sub-millisecond latencies while keeping over 95% of the true top-k neighbors. That mismatched intuition is the central magic, and the central engineering challenge, of vector search.
How do embeddings turn text into a geometric space?
Think of a city map where every landmark is placed according to its function, not its street address. A library, a bookstore, and a university would sit near each other even if their physical locations differ wildly. Embedding models build exactly this kind of map: they assign each text chunk a coordinate in a high-dimensional space so that “work-from-home policy” and “remote work guidelines” land near each other, and far from “lunch menu.”
A modern text embedding model is just a deterministic function, usually a transformer encoder, that ingests tokenized text and outputs a fixed-length vector. When we call model.encode(["All employees may work remotely on Fridays."]) with a model like all-MiniLM-L6-v2, we get back a 384-dimensional array of floats source. That array is the chunk’s “address” in the semantic map. Every chunk from the handbook gets its own address, and we store them all in a giant coordinate database. When a user asks “What is the remote work policy?”, we run the same encoder on the query, compute its address, and then find the database addresses that are closest.
The geometry of this map is not random. It is shaped during training by a contrastive learning objective. The model sees millions of pairs of sentences that are either semantically similar (paraphrases, answers to the same question) or unrelated. It is penalized when similar sentences end up far apart, and when unrelated ones end up close. Over time, the space becomes uniform (vectors spread evenly across the unit sphere) and aligned (angular distance accurately reflects meaning). That is why a small model can rival a large one in retrieval accuracy: the map’s structure, not its raw size, does the heavy lifting.
Our running assistant splits the handbook into chunks that fit inside the model’s maximum sequence length (usually 512 tokens) source. If we skip chunking, a document about “expense policies” that also briefly mentions remote work might produce an embedding that blends both topics, muddying the map. Good chunk boundaries, something we explored in Part 2, keep each coordinate meaningful. After we embed every chunk, we load them into an index that can answer “find the k nearest addresses to this query vector” fast. The next section unpacks what “nearest” really means under the hood.
What does cosine similarity actually measure?
Cosine similarity is the cosine of the angle between two vectors. In the embedding map, it measures how much two items point in the same conceptual direction, ignoring their individual lengths. Two handbook passages that both circle around “work-from-home eligibility” will have a small angle between them, yielding a cosine similarity close to 1. A passage about office snacks points somewhere else entirely, giving a similarity near 0 (or negative). That angle alone often outperforms Euclidean distance on semantic tasks because the scale of the vector can vary with text length or model quirks, while the direction captures the topic more robustly.
Mathematically, for unit-length vectors, cosine similarity equals the dot product. Many embedding models output normalized vectors by default, so maximizing inner product is equivalent to minimizing angular distance. For maximum inner product search (MIPS) in recommendation systems, algorithms like ScaNN explicitly adapt their pruning to that equivalence source. In our handbook assistant, we can treat cosine similarity and dot product as interchangeable as long as we normalize embeddings; then the only difference is a constant scaling that does not affect ranking.
Why does this matter? Because the handful of chunks that the ANN index returns will be ranked by this angle. If the model has placed all remote-work-themed chunks in a tight cluster, the top-3 results will all be highly relevant. But if noise or poor training makes the cluster diffuse, the closest chunks may still be about travel or food, just slightly nearer by accident. The next section explores why high dimensionality itself can make clusters diffuse, even if the model is decent.
Why does the curse of dimensionality break exact search?
In low dimensions, if you are in San Francisco, Los Angeles is far, and the nearest neighbor notion is sharp. But in a 384-dimensional space, something bizarre happens: almost all points look roughly equally far from a given query source. The contrast between the nearest and the farthest neighbor collapses; the ratio approaches 1. This is distance concentration, the core of the curse. For a brute-force scan over all chunks, this is merely a CPU (central processing unit) cost problem. But for spatial indexes like k-d trees or ball trees, which work by pruning large regions, the curse is fatal. In high dimensions, bounding volumes overlap so much that the pruner cannot rule out any branch, and performance degrades to scanning nearly everything.
Our handbook assistant has tens of thousands of chunks. A brute-force scan over 50,000 chunks of dimension 384 takes a few hundred milliseconds on a modern CPU, already too slow for a chat interaction. If the handbook grows to a million chunks, it becomes seconds. And naïve spatial indexes would be even slower. We cannot rely on exact nearest neighbor at scale; we need an algorithm that deliberately sacrifices perfect correctness to stay fast. That is where approximate nearest neighbor (ANN) enters, which we address next.
How do approximate nearest neighbor algorithms cheat gracefully?
ANN algorithms accept that they will miss a few of the true top-k neighbors, and in exchange deliver orders-of-magnitude speedup. They are measured by recall@k: the fraction of the true top-k that actually appears in the returned set source. A typical production system targets 95-99% recall at values of k between 5 and 50.
The most widely used class of in-memory ANN indexes is the graph-based family, led by HNSW (Hierarchical Navigable Small World). HNSW builds a sparse, multi-layer proximity graph where each chunk is a node and edges point to nearby nodes source. At query time, it performs a greedy walk: start at a random entry point on the top, sparsest layer, move toward nodes that are closer to the query, drop down to denser layers, and finally collect the closest nodes at the bottom layer. The walk visits only a tiny fraction of the graph. The parameter efSearch (exploration factor) controls how many candidates are examined; larger efSearch raises recall but also latency. For our handbook assistant, efSearch=64 might give 97% recall at under a millisecond per query on 100k vectors.
Disk-backed variants like DiskANN rearrange the graph as a single dense layer optimized for SSD access, with a compression catalog (product quantization) kept in RAM (random-access memory) source. This design lets us serve billion-chunk indexes from relatively cheap commodity hardware, which is overkill for a single handbook but indispensable when the system scales to enterprise-wide document stores.
Partition-and-quantize methods, represented by IVF-PQ (inverted file with product quantization), take a different approach. A coarse quantizer (k-means) partitions the space into, say, 1024 cells. The query is first assigned to the nearest nprobe cells, and only the vectors in those cells are scored using compressed (PQ) codes. Faiss implements this efficiently in both CPU and GPU (graphics processing unit) source. For the handbook, we might store embeddings in a Faiss IVF-PQ index with nprobe=16 and achieve sub-millisecond latency even on server-grade CPU, with recall around 95%. This method shines when memory is tight, because the PQ compression shrinks each vector to a few dozen bytes instead of 1.5 KB.
All these algorithms expose knobs that trace a recall-vs-latency curve; the engineer’s job is to pick the index that dominates that curve for the specific workload. Once we have a fast vector retrieval layer in place, we can finally answer the central question: what does “semantic” actually buy?
What does semantic actually buy us in a RAG pipeline?
Semantic search gives us the ability to retrieve relevant chunks even when the exact words are absent. In our handbook assistant, a user might type “How do I submit a vacation request?” while the policy document says “Paid time off must be requested via HR Portal at least two weeks in advance.” A keyword engine would miss this, but an embedding of the query lands close to the embedding of that sentence because the model has learned that “vacation request” and “paid time off request” point the same way.
That directional alignment is fragile. It relies on the embedding model having been exposed to similar paraphrases during training. Fine-tuning the embedding model on company-specific language (with a contrastive objective and a set of paired questions and handbook answers) can dramatically boost recall for jargon or internal names source. Without that, the semantic map may cluster “vacation” near “holiday” but still place “vacation request” far from “PTO submission” if the training data was too general.
The interplay with chunking from Part 2 becomes acute. An overly long chunk may average out multiple topics, diluting the semantic signal and making the chunk’s vector point to a “generic” direction instead of a precise one. Conversely, a chunk that is too short may lack enough context to position itself reliably, drifting toward an unrelated cluster. The semantic promise only holds when chunk boundaries respect topic coherence, and when the embedding model captures the right distinctions. With those pieces in place, the ANN index reliably returns the most concept-aligned chunks, giving the LLM (large language model) in the RAG (retrieval-augmented generation) pipeline the right raw material to answer accurately. In the next episode, we will examine how the retrieved chunks are actually fed to the model: prompt assembly strategies and the subtle ways that ordering and truncation affect the final answer.
Quick Reference: Key Configurations and Defaults
| Property | Typical Value |
|---|---|
| Common embedding dimension | 384 (all-MiniLM), 768 (BERT base), 1024 (large models) |
| Max sequence length | 512 tokens (≈ 350-400 words) for many models |
| Default similarity metric (normalized) | Cosine similarity |
| HNSW efSearch range | 16-128, higher = more recall, slower |
| IVF-PQ nprobe range | 1-64, scans that many cells |
| PQ compressed vector size | ~8-64 bytes per vector |
| Typical recall target | 0.95-0.99 for top-k |
Test yourself
Your handbook assistant uses a 384-dim embedding model and an HNSW index with efSearch=64. The index holds 25,000 chunks. For the query “what is the remote work policy?” the top-3 returned chunks are about travel approval, office supplies, and the remote work policy itself. The assistant’s answer incorrectly claims remote work is only permitted with manager approval because it fused the travel and remote chunks. How would you diagnose and fix this?
Answer: The recall of the true relevant chunk is high but its rank is pushed down by two off-topic chunks that somehow end up closer to the query. This suggests the embedding model is not adequately separating the “remote work” concept from other corporate topics. First, inspect the embedding of the query and the offending chunks with PCA to see if they cluster along noisy dimensions. Consider fine-tuning the embedding model on a supervised dataset of (query, relevant passage) pairs built from past HR tickets. Increase chunk overlap slightly so that the remote work chunk captures more context. Finally, if precision matters more than recall, lower the k returned or apply a hard cosine similarity threshold filter after retrieval, discarding chunks below 0.6 similarity before passing them to the LLM.
Frequently Asked Questions
Q: Should I always use cosine similarity? Cosine similarity is robust when vector magnitudes vary, and it aligns naturally with contrastively trained embeddings. If you normalize all vectors, dot product gives identical results. Use Euclidean distance when magnitude carries useful information, but in text retrieval cosine is the safer default.
Q: How do I choose the embedding dimension? Larger dimensions can store more nuanced differences, but they increase storage, latency, and can worsen distance concentration if the model is not correspondingly expressive. Start with a compact, well-trained model like all-MiniLM-L6-v2 (384 dims) and only upgrade if retrieval quality is insufficient after trying finer chunking and domain fine-tuning.
Q: Can I update an ANN index incrementally? Graph-based indexes like HNSW support single-vector insertion and deletion with little overhead. IVF-PQ indexes may need occasional coarse quantizer retraining if the data distribution shifts significantly. For a living handbook that changes weekly, HNSW is usually a better fit.
Q: How do attribute filters (e.g., department, date) work with ANN? Run the vector search first to get a candidate set, then filter by attributes, but this can break recall if the filtered candidates are few. Better approaches use pre-filtering within the index (Faiss enables combining IVF with metadata filtering) or use product quantization that encodes attributes alongside the vector. Always benchmark recall with your specific filter mix.
Q: Is exact nearest neighbor search ever practical? Exact search is viable for up to a few thousand high-dimensional vectors. Beyond that, the latency of a full scan becomes unacceptable for interactive applications. Use exact search for unit tests, debugging recall, or verifying top-k correctness of your ANN index against a gold standard.
If these deep dives into how real retrieval systems work under the hood resonate with how you build, subscribe to Internals Decoded at internalsdecoded.com. Every article unpacks the mechanics that make modern RAG, search, and ML (machine learning) infra tick.
Sources
- Sentence Transformers Quickstart
- Training Sentence Embedding Models
- On the Surprising Behavior of Distance Metrics in High Dimensional Space (Aggarwal et al.)
- HNSW paper: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
- Faiss building blocks: approximate nearest neighbor search
- DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node
- ScaNN: Efficient Vector Similarity Search
- Faiss indexes documentation
- sbert.net · Pretrained Models