Vector Databases: What Actually Matters When Choosing

Vector databases look much alike from the outside. Many use the same handful of search algorithms, mostly HNSW, and they all offer insert, delete and search over an API. The parts that actually change your RAG (retrieval-augmented generation) system's behavior are how they handle metadata filters, memory, updates and keyword search. This part looks at those, compares eight popular options side by side, and adds the series' third file: the handbook in a real vector database, with a permission filter.

Filters come first, because they are where systems differ most. Add a strict filter such as "only documents updated in the last 7 days" to a search done the naive way, searching first and filtering afterwards, and the right results can vanish entirely, because few of the nearest vectors happen to pass the filter. How a database applies that filter, before, during or after the vector search, is the first thing to check.

What changes when vectors live in a database?

Part 3 covered how exact search, HNSW and IVF-PQ find nearest neighbors. A vector database wraps those indexes with everything else a real system needs: storing each chunk's text and metadata next to its vector, filtering on that metadata, keeping the index up to date as documents change, persisting to disk, splitting large collections across machines, and often keyword search as well.

Remember Part 3's measurement: exact search over 50,000 vectors took about 4 milliseconds on a laptop. A handbook, or even a company's whole knowledge base of tens of thousands of chunks, does not need an approximate index for speed. You choose a vector database for filtering, updates, operations and scale, more than for raw search speed.

Build it: the handbook in a vector database

This is the series' third file, store.py. It puts the handbook chunks from Part 2 into Chroma, an open-source vector database that can run inside your Python program, and adds one chunk that only managers may read. Each chunk carries an audience field in its metadata, and every search filters on the groups the current user belongs to.

# pip install chromadb
import chromadb

from chunking import load_chunks
from embed import model

chunks = load_chunks()
# One extra chunk that only managers may see, to show a permission filter.
chunks["m9.1"] = ("For managers > Remote work abroad. Before you approve a request to work "
                  "from another country, check the employee's remaining days in the 20 day yearly limit.")
audience = {chunk_id: "managers" if chunk_id.startswith("m") else "all" for chunk_id in chunks}

client = chromadb.EphemeralClient()  # in memory; chromadb.PersistentClient(path="db") saves to disk
collection = client.create_collection(
    name="handbook",
    configuration={"hnsw": {"space": "cosine"}},  # the default distance is squared L2
)
collection.add(
    ids=list(chunks),
    documents=list(chunks.values()),
    embeddings=model.encode_document(list(chunks.values()), normalize_embeddings=True),
    metadatas=[{"audience": audience[c], "chapter": c[1]} for c in chunks],
)

def search(query, user_groups, k=3):
    result = collection.query(
        query_embeddings=model.encode_query([query], normalize_embeddings=True),
        n_results=k,
        where={"audience": {"$in": user_groups}},  # the database drops what this user may not read
    )
    return [(cid, round(1 - dist, 2)) for cid, dist in zip(result["ids"][0], result["distances"][0])]

if __name__ == "__main__":
    question = "Can I work from another country?"
    print("employee:", search(question, ["all"]))
    print("manager: ", search(question, ["all", "managers"]))

When we ran it (chromadb 1.5.9, October 2026), it printed:

employee: [('h4.2', 0.69), ('h5.2', 0.59), ('h3.2', 0.56)]
manager:  [('m9.1', 0.78), ('h4.2', 0.69), ('h5.2', 0.59)]

The manager sees the managers-only chunk first; the employee never gets it, so the model answering the employee cannot leak it. This is the simplest form of access control for RAG: store who may read each chunk, and filter in the database on every query, rather than trusting the model to keep a secret it has already been shown. Chroma returns cosine distance, so the script turns it back into similarity with 1 - distance. Its single-node index is HNSW with defaults of 100 for both ef_construction and ef_search and 16 links per node (Chroma's collection configuration docs).

How do metadata filters interact with ANN search?

There are three basic strategies, and the handbook's "updated in the last 7 days" filter shows the difference.

Post-filtering runs the vector search first, then throws away results that fail the filter. It is simple, but if only 1 percent of the collection matches, the top 10 nearest vectors may contain none of the matching documents. The database then has to fetch many more candidates, or the results come back short.

Pre-filtering finds the matching IDs first, usually with an ordinary index on the metadata, then searches only those. Every result passes the filter. The difficulty is restricting an HNSW graph to a subset: if the search simply skips non-matching nodes, the walk can get stuck in a region it cannot leave. When the filter leaves only a small set, the easy answer is exact search over that set, which is fast at that size.

Filtering during the search keeps the graph walk inside the allowed set. Databases do this in different ways. Qdrant builds extra graph links from the values in indexed metadata fields, so a filtered search stays connected, and its query planner switches to the metadata index with exact scoring when a filter is very strict (Qdrant's indexing docs). Weaviate passes an allow list of matching IDs into the HNSW search, uses a filter strategy called ACORN by default since version 1.34, and switches to brute force search when a filter becomes too restrictive (Weaviate's filtering docs).

For hard boundaries, such as separate customers who must never see each other's data, a separate collection or partition per tenant is a third option. It costs more memory and management, but the boundary is structural rather than a filter you might forget to apply.

The options side by side

Here is how eight popular choices handle the things that matter, checked against each project's documentation in October 2026 (pgvector 0.8.7, Chroma 1.5.9, Qdrant 1.19, Weaviate 1.40, Milvus 3.0, Pinecone API 2026-07, Elasticsearch 9.5, OpenSearch 3.9). Pricing changes often, so the table names each managed service's pricing model, not its prices.

Sources for the table: pgvector's README, Chroma's docs and its note that the Search API is available in Chroma Cloud only, Qdrant's indexing docs, Weaviate's filtering docs, Milvus's documentation, Pinecone's full-text search guide, Elasticsearch's filtered kNN docs and OpenSearch's efficient filtering docs.

How to choose, roughly in this order:

• If you already run Postgres, pgvector keeps vectors next to the rest of your data, with transactions, backups and permissions you already have. For many RAG systems that is enough.
• For a prototype on a laptop, an embedded database such as Chroma, Milvus Lite or Qdrant's local mode needs no server at all.
• If you rely on strict filters or many tenants, test the dedicated vector databases (Qdrant, Weaviate, Milvus) with your real filters, because that is where their designs differ.
• If you already run Elasticsearch or OpenSearch for keyword search, their vector support gives you hybrid search in one system.
• If you want no servers to run at all, a managed service such as Pinecone or the cloud versions of the others trades control for convenience, priced by usage or by capacity.

How much memory does an index need?

Start with the numbers for the vectors themselves. A vector of 1,536 numbers, the default size of OpenAI's text-embedding-3-small, takes 6,144 bytes as 32-bit floats. So 100,000 chunks take about 0.6 GB, and 10 million take about 61 GB.

An HNSW graph adds its links on top. FAISS's guidelines give the memory per vector as d * 4 + M * 2 * 4 bytes, where d is the dimension and M is the number of links per node (FAISS's index guidelines). With M = 16, the links add 128 bytes per vector, about 2 percent at 1,536 dimensions. At this dimension the vectors, not the graph, are what fill memory.

Compression is how large collections fit. Product quantization replaces each vector with a short code; FAISS notes that code sizes are typically between 4 and 64 bytes (FAISS's index overview). With 64-byte codes, 10 million vectors need about 0.64 GB for the codes instead of 61 GB, roughly 96 times smaller, plus IDs and the codebooks. The price is lower recall, which you win back by re-scoring the best candidates with their full vectors.

From a few hundred million vectors up, disk-based indexes such as DiskANN keep compressed vectors in memory and the full vectors and graph on an SSD. Its authors served a billion vectors from one machine with 64 GB of memory at under 3 milliseconds mean latency (Subramanya et al., DiskANN, NeurIPS 2019).

These are rules of thumb, not guarantees. Your dimension, data and latency budget will move them.

How do ingestion and index updates affect performance?

Adding a vector to an HNSW graph does real work: the database searches the graph to find the new node's neighbors and updates their links. For a handbook that changes a few times a week, that is nothing. For millions of new documents a day, it matters.

Deletes are harder than inserts. Most systems mark a vector as deleted, skip it in results, and clean up later by rebuilding parts of the index. Qdrant marks deleted points and a background optimizer later removes them and reorganizes the data (Qdrant's optimizer docs). pgvector relies on Postgres's usual vacuuming, which its README notes can take a while for HNSW indexes. A collection with heavy churn needs that cleanup to keep up.

High-write systems often put new vectors into a small, fresh segment first and merge it into the main index in the background. This keeps writes fast but can introduce a short delay before new vectors are searchable at full quality. If a user adds a policy and searches for it a second later, check what your database promises about that window.

Building an index from scratch in one batch is usually faster than inserting vectors one by one. The ef_construction setting controls how carefully the graph is built: higher values give a better graph and take longer. Defaults range widely, from 40 in FAISS to 200 in hnswlib, with pgvector at 64, Chroma and Qdrant at 100 and Weaviate at 128. IVF-PQ indexes also need a training step to learn their cell centers and codebooks before you can add data.

What's the real cost of scaling?

A single machine with enough memory can hold millions of vectors, so start with one and scale out only when you hit memory or throughput limits. When you do, two ideas matter.

Shards are pieces of one collection stored on different machines. A vector search usually has to ask every shard, because the nearest vectors could live anywhere, and then merge their results; more shards mean more of that fan-out. Choosing shard keys that match your filters, such as one tenant per shard, lets a filtered query touch fewer machines.

Replicas are copies of a shard, for safety and for serving more queries. Copies take time to catch up after a write, so a read right after a write may not see the newest data, depending on the database's consistency settings. Qdrant, for example, uses the Raft consensus protocol to keep the cluster's layout and collection structure consistent across nodes (Qdrant's distributed deployment docs), while the vectors themselves are copied between replicas. For a handbook assistant, a few seconds of delay before a new policy is searchable is usually fine; for some systems it is not.

Frequently Asked Questions

Q: When should I use exact search instead of ANN? When the collection, or the part of it a filter leaves, is small: up to tens of thousands of vectors exact search takes milliseconds, as Part 3 measured. Exact search is also how you check an ANN index's recall. Many databases switch to it automatically when a filter is very selective.

Q: How do I avoid losing results when using filters? Prefer a database that filters during the search or switches to exact search for selective filters. If you must post-filter, fetch many more candidates than you need and measure what you get back. Test with the filters that select the smallest share of your data, because those break first.

Q: Can I run a vector database on a single machine? Yes, for many workloads. One million 384-dimension vectors are about 1.5 GB as 32-bit floats, so millions fit in the memory of one server. Distributed setups add complexity, not magic.

Q: Do I need a dedicated vector database at all? Not always. If your data already lives in Postgres, pgvector may be all you need. If you already run Elasticsearch or OpenSearch, their vector search adds hybrid search in one place. A dedicated vector database earns its place when filters, scale or update rates push past what your existing store handles well.

Q: How important is PQ for production? PQ matters when your vectors no longer fit in the memory you can afford. It is also a tuning burden: training codebooks, choosing code sizes and deciding whether to re-score with full vectors all need testing. If everything fits in memory without it, skip it.

Test yourself

Your handbook assistant serves 100,000 documents with embeddings from text-embedding-3-small (1,536 dimensions) in an HNSW index with efSearch=100. You add a filter for "last updated in the last 7 days", and queries suddenly return only 1 or 2 results, although you know at least 20 recent documents are relevant. What went wrong, and how do you fix it?

Answer: The filter is very selective: only a small fraction of the 100,000 documents were updated in the last 7 days. Your database is effectively post-filtering: the HNSW search gathered its usual hundred or so candidates, almost all of them older documents, and the filter removed nearly everything. The fix that fits this case is to filter first and search the matching vectors exactly. A few hundred or even a few thousand recent vectors can be scanned in well under a millisecond to a few milliseconds, with perfect recall. Better still, use a database whose planner does this automatically for selective filters, or that filters during the graph search. Raising efSearch helps only with moderately selective filters; with a filter this strict the search still spends most of its effort on documents that cannot pass.

Next: Hybrid Search and Re-Ranking: The Cheapest Quality Win. It adds keyword search to the vector search, fuses the two, and re-ranks the results.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Collection configuration (Chroma, read October 2026)
• Search API overview (Chroma, read October 2026)
• pgvector README (read October 2026)
• Indexing (Qdrant, read October 2026)
• Optimizer (Qdrant)
• Distributed deployment (Qdrant)
• Filtering (Weaviate, read October 2026)
• Milvus documentation (read October 2026)
• Full-text search (Pinecone, read October 2026)
• Filter approximate kNN results (Elasticsearch)
• Efficient k-NN filtering (OpenSearch)
• Guidelines to choose an index (FAISS wiki)
• Faiss indexes (FAISS wiki)
• DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (Subramanya et al., NeurIPS 2019)