Résumé
All writing

7 min read

Your RAG Found the Right Document. So Why Is the Answer Still Wrong?

LLMs · AI

I read 106 wrong answers where retrieval worked. Half came from old and new versions of the same page, a fifth from the right document arriving without the right passage.

You built retrieval, measured it, and it works: the right document comes back almost every time. The answers are still wrong often enough that nobody trusts them.

In my system the right document reached the model for 99 to 100% of questions on every setup I tried, yet about one answer in five was wrong. So I went through every wrong answer where the right document had been retrieved, 106 of them, and labelled why each failed, with AI assistance.

Why 106 wrong answers went wrong: half mixed an old version with the current one

Only 2 of the 106 were the model reasoning badly from good evidence. Most went wrong before it wrote a word: the documents disagreed with each other, or the passage holding the answer never arrived. The best fix wasn’t a better model or better search. Telling the system which document was current lifted accuracy from 85% to 93%.

The setup

This comes out of a larger project, written up in [I Tested 10 Ways to Stop an AI Chatbot From Lying. The Fix Wasn’t the AI.](https://medium.com/@sandeshjung/i-tested-10-ways-to-stop-an-ai-chatbot-from-lying-the-fix-wasnt-the-ai-2479f0ce0f65?sharedUserId=sandeshjung). In brief:

Found the document, missed the passage

Most retrieval metrics ask whether the right document came back. But a model doesn’t read documents. It reads the handful of passages you cut from them, six in my case. So for each of the 82 answerable questions I copied out the exact sentences an answer needs, then checked whether they were among the passages the model was shown.

Document recall was 99 to 100% on every setup; passage recall was 76 to 78%, and 87% with ten passages

Hybrid search, a re-ranker and every safety feature after them left passage recall stuck at about three in four. Only changing what went into the context moved it.

And whether the passage arrived decided whether the answer was right:

When every needed passage arrived, 85% of answers were right; with only some, 24%; with none, 11%

When the passage is missing, the model rarely makes something up. It answers from the nearest passage it does have, from the right document, with a citation. That is why these answers look trustworthy, and why document-level recall hides them.

Cause 1: two versions of the truth

53 of the 106 wrong answers, from 12 questions. The archived 2025 pages are near-duplicates of the current ones, so they retrieve just as well. A superseded passage reached the model on 39 of the 82 answerable questions, on every setup until I started dropping them. Nothing inside a passage says which version is in force.

The model gave both figures, blended them into a rule that no longer applies, or picked the old one and sounded certain. In the third row the current page’s passage never reached the model at all: the archived copy took its slot. A stale duplicate doesn’t just add a wrong figure; it can crowd out the right one.

Better search can’t fix this, because search is working: both versions really are relevant. Which one is current is a fact about the documents, not in them.

Cause 2: the passage that never arrived

23 of the 106 wrong answers, from 8 questions. The right document was there; the paragraph or table row holding the answer wasn’t.

The clearest case asks: what probation period applies to a nine-month fixed-term contract? The policy has a table with the answer, four months. That row never reached the model on any setup, even with ten passages. What did arrive was a sentence saying probation for short contracts “will be outlined in their contract”. Every setup but one answered with that, confidently and with a citation; the current-documents setup refused.

I expected the re-ranker to rescue passages like this. It mostly couldn’t. On the answerability-gate setup, 19 answerable questions were missing part of their answer passage. I checked where each one ranked:

11 of 19 missing passages were in the shortlist of 20 at ranks 7 to 16; 8 were not shortlisted at all

Eleven were shortlisted but ranked just below the six-passage cut. Eight never made the shortlist, so the re-ranker never saw them. Many of the hardest are table rows: “Up to 9 months → 4 months” shares almost no words with the question and means little on its own. What beat them was usually a sibling from the same document, or the near-duplicate archived copy from Cause 1.

Causes 3 to 5: the smaller ones

The first is the one cause that truly happens during generation: a summary that stops one sentence too early. In HR, legal or medical text, that sentence is often the exception that applies to the reader.

The second is a safety net doing harm. A checker whose only move is deletion will cut sentences that merge two sources or are mostly right. Flagging or rewriting would do less damage.

The third matters beyond my numbers. If an AI grader judges your answer quality, part of what you’re chasing may be the grader. Read a sample of failures before you fix anything.

What fixed what

I tested the two fixes aimed at the two biggest causes, each added on top of the answerability-gate setup.

Dropping old versions lifted accuracy from 85% to 93%; ten passages lifted passage recall to 87% but accuracy fell to 84%

Dropping superseded documents worked. A six-line record lists each superseded document and what replaced it. Passages from a superseded document are dropped before the six are chosen, so a current passage takes the slot. Of the 12 questions behind Cause 1, 8 are now answered correctly and 2 are refused rather than guessed. Across all 132 questions it got 10 right that the gate setup got wrong, and 3 the other way round. With this sample that is suggestive rather than proof, but it was the largest gain in the project, and it came from six lines of metadata.

Showing ten passages was a mixed result. The answer passage now arrived 87% of the time instead of 78%, and it answered more questions. But accuracy fell back to 84%: it fixed 8 answers and broke 8 others. The longer context invited longer answers that wandered into the extra material, such as adoption rules added to a paternity answer. Getting the passage in is necessary, not sufficient.

Diagnosing your own system

If your retriever finds the right documents and the answers are still poor, check these in order:

  1. Write down the evidence, not just the document. For 50 to 100 test questions, copy the exact sentence or table row the answer needs.
  2. Measure passage recall next to document recall. A large gap between them is your first suspect.
  3. Split your failures in two. Evidence never arrived: a retrieval and chunking problem. Evidence arrived: a generation problem. They need different fixes.
  4. Look for old versions. Check how often a superseded passage lands in the context. Mine did on almost half the questions.
  5. Check your tables. A row cut away from its column headings is nearly impossible to retrieve and easy to misread.
  6. See what fills the slots. When the right passage misses the cut, look at what beat it.
  7. Diff drafts against final answers. If anything edits answers after generation, compare its input and output on your failures.
  8. Read the failures yourself before trusting the grader. Some of them won’t be failures.

What this doesn’t show

  • It’s small. 106 wrong answers from 34 distinct questions. Many are the same question failing on several setups, so the percentages lean towards the questions that failed most often.
  • It’s one library. HR policies and statutory-pay pages are dense with numbers, conditions and yearly updates. A library without old versions would put more weight on the other causes.
  • It’s one generator. Gemini 3.5 Flash Lite wrote every answer. In the first article, Claude was more careful about conflicting sources but still repeated planted figures, so a stronger model moves these numbers without removing the causes.
  • The labels are one reader’s. They were written with AI assistance: each wrong answer got one main cause, assigned by reading it against the reference and the passages shown, and I have not had them checked independently. They are in the repository to check and argue with.

The code, the evidence passages and every labelled answer are on GitHub: github.com/sandeshjung/rag-safety-ladder. You can also browse every setup’s answers online, side by side with the passages each one was shown.

Sources

The documents

The models

This project