Techniques for Making AI Agents Your QA Team for Reliable Modernization

Posted on Nov 3 • Originally published at cobuildx.ai

Agents can port a legacy screen in minutes, so the bottleneck moves to proving the new code behaves like the old. These are the techniques that turn agents into a QA team you can trust: layered gates, characterization tests, maker-checker review and visual diffs.

AI agents can now port a legacy screen in minutes. That moves the bottleneck. The hard question in a legacy system migration is no longer "can we write the new code?" but "how do we know it behaves exactly like the old code?"

Manual QA can't keep up with agent-speed porting, and a passing build proves very little. The answer is to point agents at verification too, and to set them up so they find problems instead of approving them. Here are the techniques that make an agent QA team you can trust.

Normal QA asks whether software meets a spec. Modernization QA asks whether the new system matches the old one, and the old one usually has no spec. Its real behavior is spread across templates, watchers, global events, server quirks and years of bug fixes nobody wrote down.

So the goal is parity: same outputs, same side effects, same edge cases, unless a difference is a deliberate, recorded decision. Every technique below serves that goal.

“Program testing can be used to show the presence of bugs, but never to show their absence.” — Edsger W. Dijkstra, Notes on Structured Programming

Don't send every change to the most expensive check. Stack the gates so cheap, deterministic checks catch the obvious problems before an agent or a person spends time on them:

Nothing merges on the porter's word. Each layer is a gate, and a failure sends the unit back.

No single layer is close to perfect, and none needs to be. With modest catch rates for each layer, the model above lets through about 6% of behavior bugs, against 28% when you stop at tests. The layers only multiply like this when they look at different things, which is why each one has a distinct job.

Have a test-writer agent read the old unit and write tests that pin down its current behavior: rendered output for known inputs, events fired, API calls made, validation messages shown, what happens on empty, error and slow responses. These tests run against the old code first and must pass there.

Then the same tests run against the new code. Because they were written from the original, not from the port, they can't inherit the porter's misunderstandings.

Lock the tests once they pass on the old code. If a porter edits a characterization test to make it pass, that is a finding, not a fix.

“To me, legacy code is simply code without tests.” — Michael Feathers, Working Effectively with Legacy Code

An agent reviewing its own work mostly agrees with itself. It already has a story for why every line is right. The reviewer has to be a separate agent with a fresh context that sees only the evidence: the original code, the new code and the test output.

Give the reviewer a narrow job and a strict output format. Ask for concrete findings, each with a file, a line and a failing scenario, and nothing else:

Compare src/legacy/orders/OrderList.js with src/app/orders/OrderList.tsx. Report only behavior differences a user or API consumer could observe. For each: file:line, the input or scenario, old result, new result. Do not comment on style. Do not suggest refactors. If you find none, say so. Enter fullscreen mode Exit fullscreen mode Run a second pass on the findings themselves. A verifier agent tries to reproduce each one, and only confirmed findings go back to the porter. This keeps false alarms from wasting porter time and teaching the team to ignore the reviewer.

The same logic applies to choosing models for each layer. Use a different model family for the reviewer than for the code it checks, put cheap fast models on high-volume checks, and keep top-tier models for the judgment calls.

Logic review misses things a user notices immediately: a shifted layout, a missing loading spinner, a date in the wrong format. An agent driving a browser can run the same flow on the old and new app, take screenshots at each step and compare them.

Record network traffic too. If the old screen made three API calls with certain parameters and the new one makes five, or drops a header, something changed. Network diffs catch problems that never show up on screen until production traffic hits them.

Agents make human review cheaper, not unnecessary. Have the orchestrator rank units by risk: payments, auth, permissions, data writes and anything with a history of incidents go first. People review those in full, and a sample of the rest.

In this model, risk ranking cuts senior review time by about two thirds while still reviewing every high-risk unit in full. Agents run the first four layers on every unit regardless.

Every deliberate difference, such as a bug you chose to fix instead of copying, goes into a decisions log with a reason. When parity tests fail later, the log tells the reviewer which differences are expected.

An agent QA team needs QA too. Seed a few known bugs into ported units, such as a dropped validation rule, a changed sort order or a missing error state, and check that the layers catch them. If a seeded bug slips through, you know which layer to strengthen before a real one does.

Track the reviewer's precision as well: the share of its findings that the verifier confirms. A reviewer that cries wolf gets ignored, and a reviewer that never finds anything is probably not looking hard enough. Both numbers belong on the project dashboard next to units migrated.

Our MigrateX plugins build this in. Each framework plugin has a read-only parity-reviewer agent that checks one migrated piece against its original for behavior, markup, data and timing differences before the old code is removed. It can't edit code, so it can only report. The porter fixes, the reviewer checks again, and an engineer signs off.

Related reading: how multi-agent AI speeds up application migration, 10 legacy modernization mistakes to avoid and software migration lessons from Shopify's AI rewrite.

Originally published on the CobuildX blog.