Hallucination Detection: Trust, but Verify
Confident, plausible, wrong. The detection layers that stop invented facts from reaching users.
Last time we gave Mailmind two kinds of memory, semantic search and graph traversal. Now it can find facts. But what happens when it invents them?
Mailmind catches hallucinations with a layered detection system. Grounding verification checks every claim against source emails. Confidence scoring flags uncertain extractions. A cross-verification agent double-checks high-stakes outputs. Consistency checks run extraction twice and compare. Structured citations force every fact to carry a verbatim quote. Deterministic checks catch invented dates and dollar amounts with regex. Together, these layers stop plausible fictions before they become wrong replies.
Even with perfect retrieval, an LLM (large language model) will confidently state a flight departs at 3:15 PM when no email contains that time. The model isn't lying. It's pattern-matching, producing the most likely next token regardless of truth. The real challenge is catching those inventions before a user books a taxi for the wrong time.
What is a hallucination in an inbox assistant?
A hallucination is a confident, plausible statement that has no source in the user's data. The model is not lying. It is completing a pattern, and the most likely next token is not always the true one. Think of an intern too embarrassed to say "I don't know," so they invent an answer with a straight face.
In Mailmind, hallucinations come in four flavors. Factual hallucinations state something not in any email, like a refund amount that never appeared. Entity hallucinations confuse people or companies, attributing a message to the wrong sender. Citation hallucinations point at a message that says something else entirely. Temporal hallucinations get dates wrong, a missed-flight class of bug. Each type can erode trust in the assistant, and each demands a different detection strategy.
How does grounding verification catch invented facts?
Grounding verification takes every claim the agent makes and checks it against the original source text. If no supporting sentence exists, the claim gets flagged. It is the simplest and most direct defense.
Mailmind runs this as a post-extraction step. After the agent drafts a reply that says "Your refund of $540 will arrive by Friday," a verification prompt asks: "Is the claim '$540 refund' supported by any email in the thread?" The model scans the conversation. If it cannot find the number, the claim is marked ungrounded. This catches the most common hallucination pattern: a number or fact that sounds right but was never written. Grounding verification costs one extra LLM call per claim, so Mailmind uses it selectively on high-impact extractions like financial amounts and meeting times.
Why confidence scoring alone isn't enough?
Confidence scoring asks the model to rate its own certainty for each extracted fact. Low-confidence facts route to human review before they propagate. It is cheap and fast, so it works as a first-pass triage.
The problem is that models can be confidently wrong. A hallucinated flight time might get a score of 0.95 because the pattern looks exactly like a real itinerary. Confidence scoring is a signal, not a verdict. Mailmind uses it to prioritize which claims get the more expensive verification steps. A claim with low confidence goes straight to a "needs review" queue. High-confidence claims still pass through other checks. Confidence scoring alone would let too many fictions through. It is the triage nurse, not the surgeon.
How does a cross-verification agent work?
A cross-verification agent is a second, independent LLM call that checks the primary agent's output against the source. It asks the same question but with a different prompt and no access to the first agent's reasoning. This breaks the echo chamber of a single model's biases.
Mailmind deploys this for scheduling actions. When the primary agent extracts a meeting time from a thread, a verification agent receives only the raw emails and the proposed time. It must confirm or reject the extraction. If the two disagree, the action is blocked and the discrepancy goes to the user. This doubles the LLM cost for high-stakes operations, but a wrong meeting time is far more expensive than an extra API (application programming interface) call. The verification agent is the skeptical colleague who double-checks your work before you hit send.
Why consistency checking reveals hallucinations that single-pass verification misses?
Consistency checking runs the same extraction twice with slightly varied prompts. Real facts tend to appear in both runs. Hallucinations, being random artifacts of the sampling process, often differ. If the two runs disagree on a fact, it is likely invented.
Mailmind uses this for batch processing of order confirmations and refunds. The system extracts the refund amount from a thread using two different prompt phrasings. If both runs return $540, the fact is considered stable. If one says $540 and the other says $520, both are flagged. This method requires no ground truth labels and works even when no human has reviewed the thread. It is an unsupervised smoke detector for factual drift.
How structured output with source citations enforces truthfulness?
Mailmind requires every extracted fact to include a verbatim quote from the source email and a pointer to the message ID. The output schema rejects any claim that lacks a citation. This is not a suggestion. It is a hard constraint enforced by the parser.
When the agent drafts a reply that includes a meeting time, the structured output must look like: {"time": "3:15 PM", "quote": "Let's meet at 3:15 PM", "message_id": "msg_abc123"}. If the quote does not appear in the referenced message, the claim is discarded. This shifts the burden of proof onto the agent. It cannot invent a time and quietly slip it into the reply. The citation is the receipt, and the schema is the auditor that checks every receipt.
When do deterministic checks beat LLM judgment?
For structured facts like dates, dollar amounts, and tracking numbers, deterministic checks are faster, cheaper, and more reliable than any LLM. A regex can confirm whether "$540" appears verbatim in the email thread. If it does not, no amount of LLM reasoning can make it true.
Mailmind runs deterministic checks before any LLM verification. When the agent claims a refund of $540, the system greps the raw email text for that exact string. If it is absent, the claim is rejected immediately. No LLM call needed. This catches the most embarrassing class of hallucination: a number that was never in the inbox. Save the LLM for semantic claims like "the sender is angry." For anything that can be matched with a pattern, use a pattern. Deterministic checks are the bouncer at the door. They don't negotiate.
How does Mailmind layer these defenses for high-stakes actions?
The full pipeline for a critical action, scheduling a meeting, confirming a refund, follows a check-your-work loop. Extraction happens first. Then verification runs in layers. Only verified facts persist. Discrepancies go to the human.
The layers build on each other. Prompt-level grounding instructions reduce hallucinations at generation time. Schema-level citations prevent unsupported claims from ever leaving the agent. Confidence thresholds route uncertain facts to review. Verification agents catch what the primary agent missed. Spot-check audits sample a fraction of processed threads to measure the system's real hallucination rate. Every caught hallucination becomes an episodic memory that improves future prompts. The defense is not a single wall. It is a series of tripwires, each catching what the previous one missed.
Quick Reference
| Technique | What it checks | Relative cost | Best for |
|---|---|---|---|
| Grounding verification | Claim vs. source text | 1 LLM call per claim | Semantic facts |
| Confidence scoring | Model's own certainty | Minimal | Triage |
| Cross-verification agent | Second LLM checks output | 1 LLM call | High-stakes actions |
| Consistency checking | Two extractions compared | 2 LLM calls | Unsupervised detection |
| Structured citations | Schema-enforced quote + pointer | Minimal | All extractions |
| Deterministic checks | Regex match in source | Near zero | Dates, amounts, IDs |
- Cost: ~1 LLM call per claim
- Latency: ~1.5 seconds
- Catches: semantic mismatches
- Cost: negligible
- Latency: under 10ms
- Catches: exact string matches
Frequently Asked Questions
Q: How do you detect a hallucination when you don't have the source text? If the source is unavailable, you cannot ground the claim. In that case, Mailmind marks the fact as unverifiable and either refuses to use it or asks the user for confirmation. No source means no certainty. The system never guesses.
Q: Does confidence scoring actually work, or is it just a number? It works as a weak signal. Models are often overconfident on hallucinations, so a high score does not guarantee truth. But a low score is a reliable indicator that something is wrong. Mailmind uses low confidence to trigger deeper verification, not to approve facts.
Q: Why not just use a second LLM to verify everything? Cost and latency. Running a verification agent on every extracted fact would double the LLM spend and slow every reply. Mailmind reserves cross-verification for high-stakes actions where the cost of an error exceeds the cost of the extra call.
Q: Can't the model just hallucinate the citation quote too? It can. That's why the citation must be an exact string match against the source email. If the model invents a quote that isn't in the thread, the deterministic check catches it. The citation is not a promise. It is a testable claim.
Q: How do you measure the real hallucination rate in production? Spot-check audits. Mailmind samples a random fraction of processed threads and has a human review the extracted facts against the original emails. That gives a true error rate. The system also tracks how often verification layers flag a fact, but the audit is the ground truth.
Test yourself
Mailmind processes 10,000 email threads in a week. Spot-check audits show that roughly 6% of threads contain at least one wrong fact, but you don't know which ones. You cannot have a human review all 10,000. How do you find the bad threads without reviewing everything?
Answer: Risk-stratify using metadata you already have. Sort threads by signals that correlate with hallucinations: low confidence scores, extraction runs that took many ReAct loop iterations, very long threads, new sender domains, and threads with multiple similar numbers near each other (e.g., two different dollar amounts). Take the top 20% riskiest threads and run deterministic checks on them first. Does the claimed date or amount appear verbatim in the source? If not, flag immediately. For the remaining high-risk threads, run a cross-verification agent. Only the threads that fail both deterministic and LLM checks go to a human for final review. This triage catches most hallucinations while keeping human review to a few hundred threads. Going forward, capture confidence scores and citations on every extraction so this risk stratification becomes automatic and improves over time.
If you want this kind of breakdown every week, how real AI systems detect and stop hallucinations before they cause real damage, subscribe to Internals Decoded at internalsdecoded.com.