Vector Databases: What Actually Matters When Choosing

Vector databases look remarkably similar on the outside. Most wrap the same open-source ANN libraries and offer search, insert, delete over HTTP. The parts that actually change your RAG (retrieval-augmented generation) system’s behavior are the internal mechanics of metadata filtering, scaling, index freshness, and consistency. This article picks apart those mechanics across the popular options, using the handbook assistant we’ve been building as a running example.

Most vector databases will give you 99% recall on a pure nearest-neighbor search. The moment you add a simple WHERE department = "engineering" filter, that number can collapse to 50% or worse. The way the database applies that filter before, during, or after the ANN search is the first thing that separates production-ready systems from toys. That’s where we’ll start.

How does a vector database store and search vectors?

A vector database stores embeddings in a specialized index, not a B-tree. For collections larger than a few thousand vectors, it uses approximate nearest neighbor algorithms to avoid scanning every vector. The exact variant does scan every vector. The approximate one trades a few percent of recall for a hundred-fold speedup. Both live inside the query engine as a data structure the planner can choose at runtime.

You can think of exact search as checking every person in a city to find the three who live closest to you. An HNSW (Hierarchical Navigable Small World) graph works more like a friend network. You ask a few well-connected people, they point you to someone closer, and within a dozen hops you’re at the right doorstep. That’s the “small world” property: the graph is big, but the number of steps between any two nodes is tiny. HNSW paper

Under the hood, HNSW builds a multi-layer proximity graph. Every vector is a node. Edges connect nodes that are close under the chosen distance metric. Upper layers are sparse and provide long-range shortcuts. The bottom layer is dense and guarantees local accuracy. A search starts at the top and greedily walks toward the query, dropping to the next layer whenever it gets stuck. Parameters like M (max connections per node) and efSearch (search breadth) control the tradeoff between recall, latency, and memory. FAISS HNSW

For the handbook assistant’s 100 000 vectors, an HNSW index with M=16 and efSearch=200 fits easily in memory and returns the top 10 docs in under 5 milliseconds. Larger datasets force you to consider compression. Product quantization (PQ) splits each vector into subvectors, learns a codebook of centroids per subspace, and replaces the original floats with short integer codes. FAISS PQ That shrinks memory by 90% or more but adds quantization error. Indexes like IVF+PQ combine a coarse Voronoi partition (IVF) with PQ to keep memory low while still probing only a few cells per query. FAISS IVF

The index choice is rarely the hard part. The hard part is what happens when you add a filter.

How do metadata filters interact with ANN search?

Filters can be applied before or after the ANN search. Post-filtering runs the ANN search first, then throws out results that don’t match the metadata predicate. This is simple but dangerous. If your filter excludes 99% of the corpus, the ANN search might return zero matching candidates from its original top-k list. The database then has to expand the candidate set (increase efSearch or re-run the search) to recover recall, which adds latency and can still miss relevant documents. Qdrant filtering

Pre-filtering narrows the set of vectors the ANN index sees. The system uses an inverted index on the metadata fields to identify the IDs that satisfy the predicate, then restricts the ANN search to only those points. This guarantees every candidate matches the filter, but it only works if the ANN index can be efficiently constrained to a subset. HNSW graphs are not naturally partitionable; you can only pre-filter by building separate HNSW indexes per partition, which multiplies memory and complicates updates.

In our handbook assistant, you might want to search only documents updated in the last 30 days. If the assistant contains 10 000 such documents, post-filtering with k=10 might return only 2 results. The database would need to fetch a larger candidate set (maybe 200) and then filter, which isn’t too painful if the latency budget allows. But if the filter selects only 50 documents, even a large candidate set might miss them, and recall drops sharply. A system that supports partition-aware ANN indexes (like Milvus with partition key isolation) can handle this cleanly. Without it, you’re forced to accept a recall-latency compromise.

Many databases also offer “filtered search” that uses a hybrid approach: the filter is applied during the graph traversal, pruning edges that lead to non-matching nodes. This keeps the search inside the allowed set but can break the graph’s connectivity if the filter is too sparse. Vespa nearest neighbor search The system that actually gets this right under your exact filter pattern is the one that will let you sleep at night.

What’s the real cost of scaling?

Scaling a vector database is not just about sharding vectors. You must replicate indexes, keep them consistent, and maintain recall under node failures. A sharding strategy that splits the vector space arbitrarily can cause queries to touch every shard, multiplying latency. A replication model that uses asynchronous followers can return stale results on a read after a write. These are not edge cases; they are the reason your RAG system might silently return irrelevant documents.

Most production vector databases shard by a primary key, often a document ID, and use a hash or range partitioning. The query planner then sends the ANN search to all shards because the query vector could be close to points in any shard. Each shard performs its local search and returns its top-k. The coordinator merges the results and picks the global top-k. This works well for collections up to a few million vectors across a handful of shards. Beyond that, the fan-out cost becomes noticeable, and you need more sophisticated strategies like key-based sharding that aligns metadata partitions with vector shards, so a filtered query only hits a subset of shards. Qdrant distributed deployment

Replication is where things get tricky. Metadata updates (like changing a collection schema) are often coordinated by a consensus protocol such as Raft. Vector data itself is usually replicated by copying the raw bytes or the index files from a leader to followers. Some systems replicate the index directly, so a follower can serve ANN queries immediately. Others replicate only the vector data and require followers to rebuild the index locally, which can cause a window where the follower returns incomplete results. Milvus architecture For the handbook assistant, if a new policy document is added and the index rebuild takes 30 seconds, a user querying immediately after the write might not see it. The system’s consistency model determines whether that’s acceptable. Pick a database that lets you choose between eventual and strong read consistency for the vectors themselves, not just the metadata.

How do ingestion and index updates affect performance?

Ingestion throughput and index freshness are a tradeoff. HNSW inserts are expensive because adding a new node requires navigating the graph, finding neighbors, and updating adjacency lists. A steady stream of writes can degrade search latency if the index is updated synchronously. FAISS HNSW performance For the handbook assistant, you might add a few dozen documents per day. That’s low enough to insert directly into an HNSW index without trouble. If you were ingesting millions of documents per day, you’d need a different approach.

High-write systems often stage new vectors in a separate, smaller index (a “buffer” or “fresh” index) and periodically merge it into the main ANN index in the background. This keeps write latency low but introduces a delay before new vectors are fully searchable. The merge process itself is a heavy operation that can impact query performance, so databases schedule it during low-traffic periods or use incremental compaction. Qdrant indexing Some systems allow you to configure the refresh interval, giving you control over the tradeoff between freshness and resource usage.

The index build itself is a batch operation. For HNSW, building from scratch (with efConstruction typically around 200-500) is faster than inserting one by one, but it still requires a full pass over the data. IVF+PQ indexes need a training phase to learn the codebooks and centroids, which can take minutes to hours on large datasets. FAISS training If you need to update the index frequently, you’ll want a database that supports incremental index building or that can rebuild a new index in the background and swap it in atomically. Otherwise, you’ll be stuck waiting for occasional full rebuilds that block writes.

Which index type should I choose for my workload?

Start with the numbers. For the handbook assistant, 100 000 vectors of 1536 dimensions, read-heavy, writes few per day. An HNSW index with default parameters fits in memory (about 1.2 GB for the raw vectors plus graph overhead) and gives 99% recall at sub-5ms p99. That’s the default choice for most sub-1M vector workloads.

If the collection grows to 10 million vectors, the raw memory for floats alone is ~60 GB, and the HNSW graph adds another 30-50%. You can cut that by 10x using PQ. An IVF+PQ index with 10 000 clusters and 64-byte codes would need roughly 6 GB for the compressed vectors plus the codebooks. Recall drops to around 95-98%, depending on the training data and the number of probes. That’s often acceptable for a RAG system where the LM re-ranks the top results anyway. FAISS index selection

If the dataset is tens of billions of vectors, you need disk-backed indexes like DiskANN or ScaNN’s hybrid approach. These store the compressed vectors and a graph structure on SSD, using RAM (random-access memory) only for a cache. Latency increases to tens of milliseconds, but the cost per query drops dramatically. This is a scale the handbook assistant will never reach, but it’s the reality for large-scale recommendation systems.

The table below summarizes the most common starting points. The exact numbers are rules of thumb, not guarantees. Your dimension, data distribution, and latency tolerance will shift them.

Vector count	Recommended index	Memory per vector	Typical recall	Notes
< 1M	HNSW (flat)	4fd + overhead	99%	Fastest, simplest
1M-100M	IVF+PQ	~0.3fd (compressed)	95-98%	Training required
> 100M	DiskANN or similar	Minimal (disk-backed)	90-95%	Higher latency

(f = float size, d = dimension)

Frequently Asked Questions

Q: When should I use exact search instead of ANN? Use exact search when the filtered candidate set is small (e.g., fewer than 10 000 vectors) or when you need to verify the recall of your ANN index. Some systems let you annotate a query to use a flat index for exact results, which is useful for debugging or auditing. Vespa exact search

Q: How do I avoid recall collapse when using filters? Prefer a database that supports pre-filtering or partition-aware ANN indexes. If you must use post-filtering, increase the ANN candidate set size (e.g., fetch 5x or 10x more candidates than your final k) and measure recall under your actual filter patterns. Test with the filters that select the smallest subset of documents, because those are the most likely to break.

Q: Can I run a vector database on a single machine? Yes, for many workloads. A single node with HNSW and enough RAM can handle millions of vectors and thousands of queries per second. Start with a single node and only scale out when you hit memory or throughput limits. Distributed setups add complexity, not magic.

Q: What’s the real difference between Qdrant, Weaviate, and Vespa? Qdrant focuses on vector search with strong filtering and a simple API (application programming interface). Weaviate adds a built-in object store and GraphQL interface, making it more self-contained. Vespa is a full-featured serving engine that combines vector search, text ranking, and structured filtering in one query language, but demands more operational knowledge. The right one depends on how much of your stack you want the database to own.

Q: How important is PQ for production? PQ is essential when your dataset no longer fits in the memory you can afford. It’s also a tuning burden: training codebooks, choosing the number of subvectors, and deciding whether to re-rank with exact distances all require careful testing. If you can fit everything in memory without PQ, skip it. If you can’t, PQ is how you stay in the game.

Test yourself

Your handbook assistant serves 100 000 documents, embeddings from text-embedding-3-small (1536 dimensions). You add a filter for “last updated in the last 7 days” and suddenly queries return only 1 or 2 results, even though you know at least 20 documents match. You’re using an HNSW index with default parameters and efSearch=100. What went wrong?

Answer: The filter likely selected a tiny subset (say, 50 documents) out of the 100k. The HNSW search with efSearch=100 visited a few hundred nodes, but most of those were not in the filtered set, so the candidate pool after filtering was nearly empty. To fix this, you can increase efSearch to 500 or 1000 to force the ANN search to explore a larger subgraph, increasing the chance of finding the filtered documents. A better long-term fix is to use a database that supports partition-based indexes (e.g., a separate HNSW per time bucket) or a filter-aware graph traversal, so the search stays inside the allowed set from the start. Without that, you’re stuck trading recall for latency.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• HNSW paper
• FAISS HNSW
• FAISS PQ
• FAISS IVF
• FAISS index selection guidelines
• Qdrant filtering
• Qdrant distributed deployment
• Qdrant indexing
• Vespa nearest neighbor search
• Vespa exact search
• Milvus architecture