Why Your RAG Chatbot Gives Wrong Answers (and How to Fix It)

Posted on Oct 22 • Originally published at aqibjavaid.site

You built a chatbot on your own documents, and it still answers with confident nonsense. Customers get wrong prices, invented policies or a polite "I don't know" when the answer is on your website. This is the most common complaint I hear about RAG chatbot wrong answers, and the good news is that most of the time the model is not the problem. The retrieval and the rules around it are.

I build and tune these systems as part of AI integration and AI-powered applications work. This guide walks through why a retrieval-augmented chatbot gives bad answers and the fixes I try, in the order I try them.

RAG stands for retrieval-augmented generation. When a user asks a question, the system:

That means a wrong answer can come from three places: the right passage was never found, the passage was found but was poor or incomplete, or the model ignored or went beyond the material it was given. Fixing the wrong layer wastes weeks, so diagnose first.

Take 20 to 30 real questions that the bot got wrong. For each one, log what was retrieved and look at it yourself.

This simple audit usually shows that one layer causes most of the failures. Everything below maps to one of these three layers.

Splitting documents into fixed-size pieces often cuts a policy in half or separates a heading from its answer. Split on headings, paragraphs and list boundaries, keep a little overlap, and store the page title and section name with each chunk so the passage carries its own context.

Semantic search finds related meaning, but it can miss exact terms such as product codes, names and error messages. Combining it with keyword search (hybrid search) and merging the results catches both kinds of question. If you store vectors in PostgreSQL, the pgvector extension works alongside normal full-text search, and my comparison of HNSW and IVFFlat indexes covers the index choice.

Fetch a wider set of candidates, then use a reranking step to pick the few most relevant passages for the model. This often improves answer quality without changing anything else.

Users ask things like "what about the cheaper one?" A search for that phrase finds nothing useful. Before searching, rewrite follow-ups into standalone questions using the conversation so far.

Instruct the model to answer only from the supplied passages and to say so when they don't contain the answer. Anthropic's guidance on minimizing hallucinations describes techniques such as allowing the model to say it doesn't know and asking it to ground claims in quotes from the source.

Make the bot show which page or document each answer came from. Citations let users verify, and they let you see at a glance when an answer is not supported by a source.

If the best retrieved passage is only weakly related, don't force an answer. Return a fallback such as offering to connect the user with a person. A bot that says "I'm not sure" at the right moment is more trusted than one that always answers.

Define what the bot may and may not discuss, and how it should speak. On the Henceforward platform I built, the admin suite included controls for brand persona, topic restrictions and AI guardrails, so non-developers could adjust behaviour without a deploy. See the Henceforward RAG chatbot case study for how that was structured.

If the answer isn't in your content, no setting will conjure it. Look at the questions the bot can't answer and treat them as a to-do list for your knowledge base:

Many "AI problems" turn out to be documentation problems.

The single most useful habit is a fixed list of test questions with the answers you expect. Run it after every change to chunking, search or prompts, and compare results. Without it, you fix one question and silently break three others.

Re-run it whenever you change anything, and add every new failure you find.

Log each question, the passages retrieved, the answer and any user feedback. Review a sample every week. Thumbs-down ratings and "I don't know" responses point you straight to content gaps and retrieval failures. My article on making AI features production-ready covers the logging, cost controls and failure handling that keep a bot trustworthy after launch.

If you have limited time, this is the order I would follow, because each step is cheap and builds on the last:

Changing one thing at a time, and measuring after each change, is what separates steady improvement from guesswork. It also gives you evidence to show stakeholders that answer quality is actually improving.

If you are planning a chatbot from scratch, my guide to AI chatbot development cost explains what drives the budget, and building an AI chatbot with Laravel and OpenAI shows one implementation.

If your chatbot is giving answers you can't trust, book a call and I will review how it retrieves and answers, and tell you what to fix first.

Originally published at aqibjavaid.site.