How to Test AI That Never Gives the Same Answer Twice

Posted on Oct 28 • Originally published at Medium

Most software tests work on a simple idea. You give the system an input, and you check that you get the right output. If the answer matches, the test passes. If it doesn't, something broke.

AI doesn't play by that rule. Ask the same question twice and you may get two different answers. Both might be fine. One might be wrong in a way that's hard to spot. This is one of the first things that catches teams out when they move an AI feature from a demo into daily use.

Here's how I think about testing AI features, in plain terms.

You can't check for one fixed answer. You need to check for qualities instead. Does the answer use the right facts? Does it stay on topic? Does it follow the format the user needs? Does it avoid things it should never say?

Write these down as a short list of checks for each feature. This list becomes your test, not a single expected reply.

Collect a set of real questions or tasks your users actually bring. Include the easy ones, the tricky ones and the odd ones. Keep the set small enough that a person can review the results, then grow it over time.

Every time the AI gets something wrong in real use, add that case to the set. This way your tests get better as the product does.

Because answers change from run to run, one pass tells you very little. Run the same case several times and look at the spread. If it gets it right every time, great. If it gets it right most of the time, you need to decide if that's good enough for this task. For a draft email, it might be. For a change going into a live SAP system, it isn't.

Not every AI feature needs the same level of care. A feature that suggests a summary can have lighter checks. A feature that takes an action, or helps someone approve one, needs much tighter checks and a person in the loop.

I find it useful to sort features by what happens if the AI is wrong. The worse the outcome, the stricter the test and the more human review you keep.

Automated checks are fast and catch a lot. They can flag missing fields, wrong formats or banned words. But some things only a person can judge, like whether an answer is actually helpful.

Set up a simple review habit. Have someone who knows the work look at a sample of results on a regular basis. Keep the review light so it actually happens.

Log what the AI was asked, what it said and what happened next. When something goes wrong, this is how you find out why. In large companies, this record also matters for audit. People need to see what the AI did, just as they would for a person.

A new model version, a change to your prompts or new data can all shift how the AI behaves. Treat each of these like a code change. Run your sample set again before it goes live.

Testing AI is less about proving it's perfect and more about knowing how it behaves. Clear checks, real examples, repeat runs and human review will get you most of the way. That's what lets people trust an AI feature enough to use it every day.

I'm Deshan Ranchhod, an AI Product Manager in Melbourne. You can find more of my work at deshanranchhod.com, on LinkedIn, on Medium and on my blog.