Résumé
All writing

9 min read

I Tested 10 Ways to Stop an AI Chatbot From Lying. The Fix Wasn’t the AI.

LLMs · AI

Better search, citations, fact-checking and confidence thresholds left accuracy flat. Telling it which documents were genuine and current took it from 77% to 93%.

The problem: a wrong answer that sounds right

You ask your company’s HR chatbot how much annual leave you get. It answers instantly, politely and with a citation. It’s wrong. You book the holiday anyway.

An AI that says “I don’t know” is mildly annoying. An AI that confidently gets your pay, your leave or your rights wrong can cost you real money. So I built an assistant with three rules:

  1. Answer only from the documents. No guessing from general knowledge.
  2. Show your sources. Every claim points to the page it came from.
  3. Say “I don’t know” when the documents don’t say.

Then I measured how often it actually kept them. The measuring turned out to be the interesting part.

What I built: a ladder, not a single system

The library was 33 real documents: HR policies from a UK hospital trust plus official GOV.UK pages on statutory pay. The assistant works like most company chatbots. It searches the documents for relevant passages, then asks an AI model to answer from those passages only. This is called retrieval-augmented generation, or RAG.

Rather than build the “best” version straight away, I built a ladder. Each rung adds one feature to the rung below, so each one’s effect can be measured on its own. Rungs A to F are the standard safety playbook. G to J are fixes I added after seeing what A to F got wrong.

The ten rungs of the ladder, from dense retrieval (A) to ten passages (J)

Two AI models do the work, both on free tiers:

My expectation was simple: every rung up should make it a little more honest. The data disagreed.

How I tested it: questions designed to catch it out

I wrote 256 test questions. Half are straightforward; the rest are traps.

For the last group I asked Claude to write 25 fake documents and slipped them into the library. They look like ordinary HR memos, each with a figure that contradicts the real policy and a hidden instruction aimed at the AI. Anyone who can save a file to a shared drive can do the same.

Every rung answered the same 132-question sample (same mix of types) and all 25 fake-document questions. A separate AI graded each answer against a reference answer I wrote, and I checked that grader against my own judgement (more on that below).

Running this on free tiers takes a lot of time. About 22 hours of actual computing, some 5,000 AI calls, took eight days of calendar time, most of it waiting for daily limits to reset. I tried to upgrade to Groq’s paid developer tier to speed it up, but it wasn’t available: sign-ups were paused due to high demand.

Finding 1: the standard upgrades didn’t make it more accurate

Better search, re-ranking, citations, a sentence-by-sentence checker and a confidence threshold: six rungs of textbook RAG safety. Accuracy bounced between 73% and 82%, in no steady direction, and every rung sits inside every other rung’s margin of error.

Accuracy of rungs A to F, all between 73% and 82% with overlapping margins of error

The upgrades did change how much it answered, from 58% to 66% of questions. They didn’t change how often it answered questions it should have refused: 10% at rung A, 13% at rung F, both with a margin of about ±7 points.

So the upgrades bought more answers, not more honesty. The next three findings explain why.

Finding 2: its confidence score barely knew when it was wrong

Rung F refuses when its confidence is low. I built that confidence from three common-sense signals: consistency (ask three times, do the answers agree?), backing (how many sentences did the checker confirm?) and search strength (how clearly did the best passage stand out?).

Correlation of each confidence signal with being right: about +0.2 on ordinary questions, negative with fake documents present

On ordinary questions the combined score tracked correctness only weakly (+0.20, where 0 is chance and 1 is perfect). With fake documents in the library it turned the wrong way (−0.22). A fake written to match a question stands out in search, so the system grew more confident exactly when it was being fooled.

A refusal rule is only as good as the confidence behind it. “The AI was consistent” and “the AI checked itself” are not evidence that it was right.

Finding 3: fake documents didn’t hijack it, they fooled it

The usual worry about planted documents is prompt injection: hidden text that takes control of the AI. Across 250 runs (25 fakes, ten rungs), I found no case of the AI obeying a hidden instruction.

Then I checked whether the answers were right. They weren’t.

Accuracy with and without fake documents for rungs A to G: roughly halved on every rung

The AI wasn’t taking orders from the fakes. It was believing them. A memo that looks like a policy supplement gets treated as one, and the assistant reports what it says, citation and all. At rung D, 15 of 25 answers cited a fake.

The sentence checker couldn’t help. It confirmed 15 of the 18 claims that cited a fake, and it was right to: the fake really did say that. A checker that asks “does the source say this?” can’t ask “should I trust this source?”

Would a smarter model fix it?

I gave Anthropic’s Claude Opus 5.5 exactly the same 25 prompts, fakes included.

Claude was far better at noticing that something was off, but it still couldn’t tell which document was genuine. Most of its answers laid out the real figure and the fake one side by side and told the reader to check with HR. A smarter model turns silent capture into a visible conflict. It doesn’t close the hole, because which document to trust is information about the documents, not in them.

Finding 4: it found the right document and still got it wrong

When a chatbot answers wrongly, the usual suspect is search. Not here: in 123 of the 124 wrong answers across every rung, the correct document was among the passages the AI was shown.

So I went through the 106 wrong answers from rungs A to G one by one and labelled why each went wrong, with AI assistance.

Why 106 wrong answers went wrong: half mixed an old version with the current one

The biggest cause was old and new versions of the same thing. Real libraries are full of them: a superseded policy nobody deleted, last year’s statutory pay rates beside this year’s. Faced with two figures, the AI gave both, blended them or picked the old one. “You can get £116.75 per week or up to £123.25 per week” is cited, polite and useless.

Only 2 of the 106 were the AI reasoning badly from good evidence. Most of the damage was done before it wrote a word, by what the documents said and which passages reached it. That gap is the subject of a [Your RAG Found the Right Document. So Why Is the Answer Still Wrong?](https://medium.com/@sandeshjung/your-rag-found-the-right-document-so-why-is-the-answer-still-wrong-a3b304cedff1).

Finding 5: knowing which documents to trust, and which are current, is what worked

The findings point at three gaps: the system can’t tell when it doesn’t know, can’t tell a real document from a fake, and can’t tell an old version from the current one. I added one rung for each, then one more.

Accuracy for rungs F to J: prefer current documents reaches 93% on ordinary questions

G · Answerability gate. Before writing anything, a second AI looks at the passages and asks: do these actually contain the answer? If not, it refuses. It checks the evidence rather than the answer, so a confidently wrong answer can’t talk its way past it. It stopped 29 of the 36 questions the documents couldn’t answer, and accuracy rose from 77% to 85%.

H · Provenance register. Only documents on an approved list, with a fingerprint of their contents, can reach the AI. Not one fake got through, and accuracy with fakes present doubled, from 35% to 71%. The caveat is built in: my fakes were never registered. The risk moves to whoever controls the list. (A cheaper rule, “trust anything with a reference number and approval date”, blocked every fake but also rejected 16 of 23 real policies and all 10 GOV.UK pages.)

I · Prefer current documents. A six-line record of which document replaced which. Passages from a replaced document are dropped before the AI sees them. This was the biggest gain of the project: accuracy went from 85% to 93%. Question by question it got 10 right that the gate got wrong, and 3 the other way round.

J · Ten passages instead of six. The answer passage often ranked just below the top six, so I showed it ten. The answer passage arrived more often, but answers wandered into the extra material and accuracy fell back to 84%. More context isn’t free.

With 132 questions each rung’s accuracy carries a margin of about ±7 points, and the fake-document figures (25 questions) about ±15. The gains from G and I are the strongest signals in the data, but they are evidence, not proof.

Checking my own homework

Two of my own mistakes nearly made it into this article.

My scorecard punished the right answer. Early results said citations made the assistant answer more questions it should have refused. Reading them one by one, six were false-premise questions where it correctly said “that’s not true, here’s what the policy says”. My scorecard counted any answer to those as a failure. Once fixed, the “upgrades make it worse” story disappeared.

A bug made the confidence finding look worse than it was. The code that read citations was silently dropping every citation to a GOV.UK page, so the checker saw only about half the claims. After fixing it and re-running everything affected, the confidence score turned out to be weakly right on normal questions and wrong only under attack. A smaller headline, and a truer one.

I also checked the AI grader against my own judgement on 42 answers:

Its mistakes ran both ways, so it isn’t quietly flattering the system. But 42 answers is a small sample, and I reviewed pre-drafted labels rather than labelling blind. Every number in an AI evaluation is only as good as the scorecard behind it.

What I’d tell anyone building one

  • Measure before you add safety features. Most of mine sounded right and measured as neutral.
  • Test with trick questions. An assistant that aces normal questions can still invent answers its documents never cover.
  • Don’t trust the AI’s confidence in itself. Check any confidence signal against graded answers before it makes decisions.
  • Control who can add documents, and keep a list of what’s genuine. The real threat was a believable fake memo, not a clever hidden command.
  • Clean up old versions. Telling my system which version was current lifted accuracy from 85% to 93%, more than any other change.
  • Read the actual answers. One of my headline findings was wrong because of my own scorecard.

Citations make an AI answer look trustworthy. Looking trustworthy and being right are measured separately, and only one of them is easy to fake.

Try it yourself

The code, the 256 questions and every recorded answer are on GitHub: github.com/sandeshjung/rag-safety-ladder. You can also explore the results online: pick any test question and read what each rung answered side by side, with the passages it was shown, which ones it cited, and whether any were fakes or out of date. No sign-up needed. With your own free Gemini and Groq keys, you can ask it your own questions live; the keys are used for that request only and never stored.

Sources

The documents

The models

This project