Why Every AI Answer Needs a Source: Notes on Building RAG Chatbots

An AI assistant should show where each answer came from, and it should be allowed to say it does not know. Those two behaviours matter more than which model you choose, because they are what let a person trust the answer or catch it when it is wrong.

At Octarnal we build assistants that answer from a company's own documents. This is what that work has taught me so far.

Why is a demo not enough?

The demo always goes well. Someone asks a question, a confident paragraph appears, and everyone in the room nods. The trouble starts later, when a person acts on a paragraph that was wrong.

Language models are fluent by nature. They produce a plausible sentence whether or not there is a fact underneath it. In a demo you ask questions you already know the answers to, so you never see the failure. Real users ask the questions you did not think of.

What does a source actually fix?

A source turns an answer into something you can check. If the assistant says the notice period is thirty days and points to page four of the employment policy, the reader can open page four. If the page says something else, the mistake is caught in seconds instead of weeks.

This is most of what retrieval is for. The point of fetching documents before answering is not to make the model sound clever. It is to keep the answer tied to something the company actually wrote down.

In practice that means every reply should carry the document and the passage it drew on, with one tap to open it. Trust then rests on the document, not on how confident the model sounds.

Why let it say "I don't know"?

Because a missing answer is cheaper than a wrong one.

When retrieval finds nothing solid, a system has two choices. It can tell the user it could not find the answer, or it can let the model fill the silence from its general knowledge. The second feels more helpful and is far more dangerous, because the invented answer looks exactly like a real one.

The safer design is a threshold. If the retrieved passages are not relevant enough, the assistant says so and, where possible, points to a person who can help. Users forgive an honest gap. They do not forgive being misled.

How do you know it is working?

You measure it against real questions. Collect the questions people actually ask, write down the correct source for each, and check two things: how often the right passage is retrieved, and how often the cited passage really contains the answer given. The second number is the one that exposes confident nonsense.

It is slow, unglamorous work, and it is the difference between a demo and a product.

A small example on this site

The assistant on this site follows the same rule on a small scale. It answers only from a fact sheet about me, and when a question falls outside that sheet it says so instead of inventing something. I would rather it admit a gap than describe a version of me that sounds better than the real one.

I am still a student and I am still learning how to build these systems well. But this one principle has held up on every project so far: an answer without a source is a guess, and people deserve to know which one they are getting.