Four rules that stop a support bot making things up

A support bot that makes up an answer does more damage than one that says "I don't know". The customer acts on it, it turns out to be wrong, and now you have a support ticket and a customer who trusts your help less than before.

We build Deacon, an AI support agent that answers from a company's own docs. This post covers the four rules we use to keep it honest, why each one matters, and how to test your own bot for the same failures. You can apply all of them to any retrieval-based bot, whatever it's built on.

Why bots make things up

A language model writes the most likely next words given everything in front of it. It has no built-in sense of which of those words came from your docs and which came from its training.

Say your docs contain two true statements in different places. The first says you accept Apple Pay, and the second says refunds take 14 days. A customer asks how long an Apple Pay refund takes. Nothing in your docs answers that, but the model can join the two facts into a confident claim that Apple Pay refunds take 14 days. Both sources are real, and the answer is still invented.

Retrieval on its own doesn't fix this. Giving the model the right passages helps, but it will still fill gaps with general knowledge unless you tell it not to and give it an honest way out.

Rule one. Answer only from what was found for this question

Every answer is written from the passages retrieved for that specific question, and nothing else. We number the passages and keep track of which page each came from, so the answer can link the source.

In practice that means the system prompt treats the passages as the only allowed material. A simplified version looks like this.

Answer the customer's question using only the numbered passages below.
If the passages don't contain the answer, say you don't know. Never use outside knowledge.
Cite the passage numbers you used.

Citing passage numbers matters more than it looks. It gives you something to check automatically. An answer that cites nothing, or cites a passage that doesn't contain the claim, is a failure you can catch in testing.

Rule two. Only the facts the passages state

Prices, numbers, dates and policy details are where invented answers do the most harm, so the model may only repeat them when a passage states them.

The subtle part is combination. Two passages must never be stitched into a claim neither of them makes alone. That's the Apple Pay failure from earlier, and it's the one most bots miss, because every fact in the answer can be traced to a real source.

We tell the model this directly, and we test it with questions designed to tempt a join between two passages.

Rule three. Ask a clarifying question first

Plenty of questions are answerable, just not yet. "How do I export my data?" might have three different answers depending on the customer's plan.

When the docs hold the answer but it depends on something the customer hasn't said, the bot asks one question back instead of guessing or giving up. If it still can't answer after three rounds, it hands the conversation to a person.

This keeps "I don't know" for questions the docs really don't cover, so it stays meaningful.

Rule four. Write the missing answer once

An honest bot will say "I don't know" more often than a reckless one. That's only useful if someone fixes the gap.

Every question the bot couldn't answer goes on one list, with a box to write the answer. The answer you write becomes part of the knowledge base, so the next customer who asks gets it. Over time, the list of unanswered questions shrinks and the bot handles more on its own.

What you give up

Some questions will get "I don't know" where a less careful bot would have guessed right. We think that's the correct trade. A wrong answer costs more than a handoff, and each "I don't know" tells you exactly which page of your docs to write next.

How to test the bot you already have

You can check any support bot for these failures in about ten minutes.

1. Ask something your docs don't cover. A good bot says so and offers to pass you on.
2. Ask for a specific fact that isn't written anywhere, like a price for a plan you don't sell. A good bot declines.
3. Find two true statements in different places in your docs, and ask a question that joins them. A good bot answers only what each passage says.
4. Ask a question whose answer depends on the customer's plan or setup. A good bot asks which one before answering.

If your bot fails any of these, tightening the prompt helps, but you'll get further by testing these cases every time you change the prompt or the docs. We run a set of questions our docs can't answer, and the test fails if any of them gets an answer.

If you'd like to see these rules working, Deacon has a free plan with 50 answers a month. Point it at your docs and ask it something they don't cover.