RAG vs Knowledge Graphs: Two Kinds of AI Memory Lookup
Semantic search answers fuzzy questions. Graph traversal answers precise ones. Real systems need both.
RAG uses vector embeddings to answer fuzzy semantic queries across unstructured text. Knowledge graph traversal answers precise multi-hop questions about how specific entities connect. Mailmind, our AI inbox assistant, combines them: graph filters for precision, then RAG scores by meaning within that narrowed set. The two lookups answer different questions.
Search is fast. Graph traversal is fast. But they break on each other’s questions. The smarter system isn’t the one with the better index. It’s the one that knows when to use which memory.
After the last episode’s production-grade prompt engineering, you can make Mailmind behave consistently. But an agent that always thinks the same way still answers based on what it remembers. What does Mailmind actually know about your inbox? And when a question goes beyond the last few messages in the chat window, how does it find the right memory?
When does semantic search beat a database query?
RAG retrieves passages by meaning, not by exact word match. That makes it the right choice when you can describe what you need but can’t name it.
Think of RAG as a search engine that indexes ideas, not keywords. You type “emails where the customer was unhappy about a missed deadline,” and it surfaces messages that never mention “unhappy” or “missed deadline” but convey frustration through phrasing like “This took longer than expected.” That facility comes from how RAG stores and searches text.
RAG works by breaking every email into overlapping chunks of a few hundred tokens. Each chunk gets turned into an embedding, a list of numbers that captures its semantic shape. Those embeddings live inside a vector database, which can find the nearest neighbors to any vector you give it. When Mailmind gets a question, it computes an embedding for the question and asks the vector DB for chunks whose vectors are closest by cosine similarity. The top results get passed to the LLM (large language model) as context. The original RAG architecture was described by Lewis et al..
That retrieval path is perfect for open-ended discovery. It fails hard on structural questions, though. The next section explains why.
What kind of question does a knowledge graph answer?
A knowledge graph answers questions about how specific real-world entities connect. You ask it for the relationship between two concepts, and it walks a network of facts to give you a precise answer.
A knowledge graph is a map of your inbox’s people, projects, threads, and orders. You can trace a path from one node to another and count the edges. Mailmind builds this map as a set of nodes (Person, Thread, Order) connected by relationships: SENT, WORKS_ON, ORDERED, REPLIED_TO.
When a user asks “which people from the Falcon project have I not replied to this week?” the graph does the work without guessing. It starts at the Falcon project node, follows WORKS_ON edges to Person nodes, then follows SENT edges to the emails they sent this week. Finally it filters out any person who already has a REPLIED_TO edge to those emails. The answer is exact. No similarity search necessary. No “maybe” in the result.
This is a graph traversal, often expressed in a query language like Cypher. The database doesn’t scan rows. It walks edges. That gives it a fundamentally different cost model and failure mode than a vector search.
Graph queries are unbeatable for precision, but ask them to find “complaint-like threads” and they return zero results. That failure points directly at the limits of structured data and why Mailmind must combine both.
Why can’t either memory handle all questions alone?
RAG stumbles when the answer depends on a specific, real-world relationship. Ask “has Sam replied to the Stratos thread?” and RAG might retrieve a chunk where Sam mentions Stratos in another context. The LLM then sees “Sam” and “Stratos” together and may fabricate a reply. Traversal does not guess; it checks for a REPLIED_TO edge and returns a binary answer.
The graph stumbles on anything shaped like a description. “Find emails where the sender was upset about shipping” fails because “upset” isn’t a node or a relationship. No edge links Emotion to Email. The graph has no way to compute tone from raw text. It returns nothing, even when the right emails exist.
RAG and the graph cover two different axes of memory. One axis represents structure: who did what to whom. The other axis represents meaning: what kind of message this is. Combining them gives Mailmind the power to ask questions that mix both axes.
How does Mailmind combine both kinds of lookup?
Mailmind doesn’t decide between RAG and graph. It routes the query to the graph to get a precise candidate list, then uses RAG to rank by semantic relevance within that list. The graph handles “who, where, when”; RAG handles “what kind of.”
Consider the question “Show me Falcon project threads this month that discuss budget.” The flow looks like this:
- The system parses the question and recognizes two constraints: structural (Falcon project, this month) and semantic (discussing budget).
- The graph traverses to all threads connected to the Falcon project and filters to those with activity this month. That yields, say, 14 threads.
- Mailmind collects the chunks from those threads, embeds them, and embeds the query’s semantic part (“discussing budget”).
- Cosine similarity scores rank the chunks. The top-scoring threads land in the LLM’s context.
The graph prunes the search space from “everything in the inbox” to exactly the relevant entities. RAG then contributes the nuance that a keyword filter would miss: budget might be called “spend,” “allocation,” or “runway.”
There is one crack in this elegant stack. None of it works if a person exists as two separate nodes. Entity resolution, the problem of knowing that “Sam K.” and “Samuel Kim” are the same person, is what makes the bridge solid. That’s for the next episode.
Quick Reference: RAG vs Knowledge Graph
| Property | Vector (RAG) Memory | Graph Memory |
|---|---|---|
| Data shape | Unstructured text (chunks) | Structured nodes and edges |
| Query type | Semantic: “find things like this” | Structural: “how are A and B connected?” |
| Best for | Open-ended discovery, tone, topics | Multi-hop paths, exact counts, filtering |
| Mailmind example | “Show me threads where the sender was upset about a delay” | “Who from Falcon haven’t I replied to?” |
| Failure mode | Hallucinates connections when precise structure is needed | Returns nothing when the question involves inherent meaning |
| Underlying store | Vector database (e.g., Pinecone, pgvector) | Graph database (e.g., Neo4j, DuckDB’s graph extension) |
- Unstructured text chunks
- Embedding similarity search
- Fuzzy semantic queries
- Weak on exact relationships
- Structured nodes and edges
- Graph traversal (Cypher)
- Precise multi-hop queries
- Weak on semantic nuance
Frequently Asked Questions
Q: If I already have a vector database, why add a graph?
A vector DB can’t reliably answer “which Falcon contacts did I email in April?” because it treats each chunk independently; it has no concept of a person or a thread. A graph stores relationships natively and answers such questions in a single traversal with 100% precision. Without it, you’d need to embed every chunk and filter by guessed metadata, which breaks when metadata is inconsistent.
Q: Can a knowledge graph store full email text, avoiding RAG entirely?
You could store text as a property on nodes, but graph traversal can’t determine semantic similarity across bodies of text. If the user asks “find any thread where the tone matches this complaint,” the graph has no way to compare tone; it would return nothing or all nodes. RAG is needed to score by meaning.
Q: How do you keep a knowledge graph in sync with live email traffic in Mailmind?
Every incoming email triggers a small pipeline: extract entities with an NER (named entity recognition) model, resolve identities, upsert nodes and relationships into the graph. Updates and deletions are trickier. Mailmind uses a micro-batch approach that replays events from a write-ahead log. The graph is not a static dump, it’s a continually updated reflection of the inbox.
Q: Doesn’t building a knowledge graph require a rigid schema?
Modern property graph databases let you mix schemaless flexibility with lightweight constraints. Mailmind’s graph defines a handful of core node labels (Person, Thread, Order, Meeting) and relationship types, but you can add new ones without downtime. Uniqueness constraints on identity fields, like email address, keep the graph useful without locking you in.
Q: When does RAG fail on a question that seems simple?
When the answer depends on a specific relationship between two entities, RAG can retrieve chunks that mention both but fail to establish the connection. For example, “Did Alice reply to the strategy doc thread?” might pull an email where Alice mentions the strategy doc in a different context, causing the LLM to hallucinate a reply. A graph traversal answers definitively by checking for a REPLIED_TO edge.
Test yourself
User asks Mailmind: “Find threads from this week that have a similar frustrated tone to this complaint email about a delayed refund.” You run a pure knowledge graph query and get back nothing useful. Why does the graph return nothing, and what do you do to actually answer the question?
Answer: The knowledge graph knows about timestamps, participants, and refund order IDs. But “frustrated tone” is not a stored property, it’s a semantic quality inferred from the text. A graph traversal can’t compute similarity between text passages; it can only follow defined edges. To answer the question, Mailmind combines both memories. First, the graph narrows the candidate set to threads from this week that mention a refund order (by tracing ORDERED edges). That yields a manageable list of candidate chunks. Then Mailmind embeds each thread’s content and the complaint email’s content, computes cosine similarity, and returns the top matches. The graph provides precision filtering; RAG provides the similarity scoring that turns “similar tone” into a searchable concept. Without the graph filtering, RAG would have to search the entire inbox, risking noise and slower response.
Every week, we decode one more subsystem that turns a naive language model into a reliable AI assistant. If that’s your idea of fun, subscribe to Internals Decoded at internalsdecoded.com.
Next up: we’ll connect these two worlds permanently by tackling entity resolution, the problem of knowing when “Sam K.” and “Samuel Kim” are the same person in your inbox.