How to Evaluate a RAG Chatbot Before Your Customers Do: Build a Test Set That Catches Real Failures

Posted on Oct 30 • Originally published at znovastudio.com

To evaluate a RAG chatbot before launch, test it against an evaluation set: real questions with known good answers, including the ones it should refuse, run again after every change. A demo can't tell you the bot is ready. A test set can.

Demos always go well. Someone asks five questions they know the documents answer, the bot replies with citations, and everyone agrees it's ready. Then real customers ask about the old pricing, a product you discontinued or something your documents don't cover, or paste in a question with three typos. That's where the bot invents a refund policy.

OpenAI's own evaluation guide calls the alternative, "it seems like it's working", an anti-pattern. Below is how we'd build the set, step by step. It assumes you've chosen RAG; if you haven't, start with RAG vs fine-tuning. Tracking answer quality with test questions and user feedback is part of our LLM integration and chatbot service.

The best questions come from people who've actually asked them:

Aim for 50 to 200 questions to start. Fewer and you can't see patterns; many more and nobody keeps grading them.

Then cover the ways chatbots actually fail. This mix separates a useful test set from a flattering one:

Around a quarter of the set should be questions the bot must not answer confidently. That's where hallucinations hide.

For each question, record in a spreadsheet:

Don't write full model answers. Two correct answers can be worded very differently, and you want to grade facts, not phrasing.

A RAG system has two halves: retrieval finds passages, generation writes the answer. When an answer is wrong, you need to know which half failed, because the fixes are different.

For every question, check whether the right source passage is in the top results retrieved (say, the top 5). If it isn't, no prompt will save the answer. The fix lies in chunking, search settings, document cleanup or adding keyword search alongside vector search.

Open-source tools such as Ragas measure both halves with named metrics: context recall (did retrieval find what was needed?), context precision (were the relevant passages ranked near the top?), faithfulness (is every claim in the answer supported by the retrieved passages?) and answer relevancy (does it address the question?). Context recall needs a reference answer to compare against, and the key facts from step 2 can serve as one.

One caution about faithfulness: it checks the answer against what was retrieved, not against the truth. If retrieval pulls the outdated policy, a faithful answer is faithfully wrong. That's why the out-of-date traps in step 1 matter.

So calibrate before you trust it. Grade 30 to 50 answers by hand, run the model judge on the same ones and compare. Where they disagree, tighten the grading instructions. Tell the judge to ignore length, and if you compare two versions side by side, run it twice with the order swapped.

A score is only useful if it decides something. Before launch, agree rules like these:

If a change fails the gate, it doesn't ship, however good the demo looked.

Chatbots regress quietly. Re-run the full set whenever any of these change:

Keep every run's results. When a customer reports a bad answer six months from now, you'll want to know when it started.

The OWASP Top 10 for LLM Applications (2026 edition, published in August 2026) keeps prompt injection as the number one risk, including indirect injection, where the malicious instruction sits inside a document the bot retrieves rather than in the user's message. It also lists vector and embedding weaknesses, such as retrieval returning documents the user shouldn't see.

The stakes rise if the assistant can also take actions, such as issuing refunds or updating records. Our guide to AI agents for small businesses covers the extra risks of that step, prompt injection included.

Log questions (with personal data removed), add thumbs-up and thumbs-down buttons, and review the thumbs-down answers every week. Each confirmed failure becomes a new test case. After a few months, the set reflects your actual customers, not your guesses about them.

Start with 50 to 200 real questions, covering answerable, unanswerable, out-of-date, restricted and adversarial cases. Grow the set from real conversations after launch, adding every confirmed failure.

Yes, for qualities code can't check, but calibrate it first. Grade a sample by hand, compare it with the model's grades and adjust the instructions. LLM judges are known to favour longer answers and the answer shown first.

Measure retrieval and answers separately. For retrieval, context recall (did it find what was needed?) and context precision (were the relevant passages ranked near the top?). For answers, faithfulness (is every claim supported by the retrieved passages?) and answer relevancy (does it address the question?). Faithfulness isn't correctness: if retrieval returns an outdated document, an answer can be faithful and still wrong, so also check answers against your key facts.

A RAG chatbot is only as good as the tests it has passed. Start this week: collect 50 real questions, including the ones it should refuse, and run them before every release. It's less exciting than the demo, and it's what makes the bot safe to put in front of customers.

Planning an assistant on your own documents? Book a free consultation and we'll help you build the evaluation set before you build the bot.

Originally published on znovastudio.com.