From experienced tester to AI testing: the roadmap, in order

Posted on Oct 28 • Originally published at ai-testing-in-practice.ai-testing-in-practice-site.workers.dev

Short answer: learn it in this order — the non-determinism mindset, prompts as test inputs, evals (the new assertion), LLM/RAG failure modes, then verifying AI-written tests with mutation testing. Skip the math-heavy ML theory; you're testing behavior, not training models.

I keep seeing the same post on r/softwaretesting: "10 years of manual and automation under my belt, and the AI stuff makes me feel like a junior again. Where do I even start?" The honest answer is that you're closer than you think. Maybe 70% of what makes you a good tester transfers directly. The other 30% is a handful of new ideas, and they stack in a specific order. Learn them out of order and it feels like chaos. Learn them in order and each one clicks.

Everything new starts here. In normal testing, one input has one correct output, so you assert equality. In AI testing, one input has a set of valid outputs — ask the same question twice and the wording changes, both answers fine. That single fact breaks every exact-match assertion you've ever written and is why your first naive AI tests will flake.

Don't move on until this feels natural, because the rest of the roadmap is just tools for dealing with it. If a test flaps on identical input, your instinct should flip from "the system is flaky" to "I'm asserting the wrong thing."

Everyone treats prompting as a party trick. For you it's controlling the test input. A vague prompt is an uncontrolled variable; you can't tell a real regression from your own sloppy setup. Spend a week writing prompts that pin down format, scope, and constraints, and reading back exactly what the model did with them. You're not learning to be clever. You're learning to make the input reproducible enough to test against.

This is the step most testers skip, and it's why their AI tests feel unreliable later.

This is the real new skill, and it's where I'd spend the most time. An eval replaces "did the output equal X?" with "is this output acceptable?" Three flavors, in increasing effort:

def test_support_reply (): out = bot ( " my card was declined " ) assert out. intent == " billing " # property: deterministic assert " $ " not in out. text # property: no invented amounts assert judge ( out. text, rubric = " acknowledges, asks one question, no blame " ) >= 0.8 Enter fullscreen mode Exit fullscreen mode If you learn one thing from this whole list, learn to write evals. It's the skill job posts are circling around without naming.

Now get specific about how these systems break, because it's not how normal software breaks — and how to tell a real bug from the model just drifting:

You don't need to know how a transformer works to test any of this. You need to know what wrong looks like, which is exactly what testers are good at.

You'll let Copilot or Claude generate test suites. Fine. But generated tests have a specific trap: they tend to assert that the code does what it currently does — bugs included. A green suite that was written against buggy code is just the bug, notarized.

The fix is mutation testing: deliberately break the code and check your tests scream. If you flip a > to >= and every test still passes, those tests prove nothing. Run this on AI-generated suites especially — it's how you actually earn confidence in AI-written tests. This is the whole point of the book, and the free lab is a 10-minute demo of it — a suite that's green against four real bugs.

You do not need: the math behind training, building models, PyTorch, or a data-science course. You're testing behavior at the boundary, the same place you always have. Picking up ML theory first is the most common way people stall for three months and learn nothing testable. Skip it until something forces you to care.

Realistically this is a few focused weeks, not a bootcamp — and you keep every bit of your existing automation, CI, and bug-sense. Don't let anyone sell you "AI testing" as a reset. It's an extension.

Do I need to learn Python? If you already automate, you're set — most eval and LLM tooling is Python-first, but it's the same testing you do now with different assertions.

Is this going to replace traditional QA? No. It sits on top of it. CI, coverage, API and E2E testing all still apply; AI adds non-determinism and a new set of failure modes to cover.

Where does the book fit in step 3 and 5? Those two steps — evals and verifying AI-written tests — are the spine of AI Testing in Practice. It walks the full version with runnable examples, so you're not piecing it together from scattered blog posts.