Your AI Review Bot Reviewed a Diff the AI Itself Wrote. That's Not a Review.

Posted on Oct 9 • Originally published at powerduck.com

The bot leaves a comment 12 seconds later: LGTM, looks clean, no issues found.

The diff it reviewed was generated by the same model that's now reviewing it.

This is the most common failure mode in AI-assisted development right now, and almost nobody is calling it what it is: it's not a review. It's the author reading their own essay out loud to a friend who already agrees.

When you hook a model into GitHub and point it at a pull request, it reads the changed lines, the surrounding context, and your commit message. It then produces a plausibly written summary of what the code does, plus a few stylistic nits.

If the diff came from the same model (or a related one), the bot shares the same blind spots. It misses the same off-by-one. It agrees with the same questionable abstraction. It calls the same clever workaround "clean code" because it wrote the workaround.

The review loop closes. The LGTM lands. The PR merges. Three days later, production breaks in a way that was visible in the diff all along — to a human who wasn't invested in the solution being correct.

Useful code review doesn't come from another model's opinion about the model's output. It comes from something the code didn't generate itself:

These signals don't care what the model intended. They don't read the commit message. They just run and report.

An LGTM from a model is worth roughly the confidence the model already had in its own output. Which is, by construction, higher than the evidence supports.

This doesn't mean AI code review is useless. It's useful for the things a tired human reviewer skips:

But it should be positioned as a second pair of eyes on style and edge cases, not as a correctness check. When the bot says LGTM, the human reviewer should treat that as "the stylistic pass is done," not as "this is safe to merge."

After watching this play out across a few repos, we landed on a rule that costs almost nothing:

Never let the bot's LGTM be the only automated signal on a PR. If the diff was AI-generated, require one independent check that the model didn't generate: a green type check, a contract test, or an integration test against a live endpoint.

For API work specifically, the independent signal is usually the contract. A spec file that existed before the agent touched the code is the one artifact the model can't rationalize. When the agent's generated handler returns a response that doesn't match the spec, the contract check fails — regardless of how confident the model sounds.

That's the reason we ended up building this into Powerduck: the spec stays the source of truth, and the agent has to re-run the request against the live endpoint and compare the actual response against the contract, not against what it thinks it should have returned. The bot can't talk its way out of a failed assertion.

Teams that skip this step usually tell themselves they'll "just review the PR more carefully." But the whole point of using an AI coding agent is that the PR volume goes up. You can't review twice as many PRs at the same depth as before.

Which means you need the independent signal to do the work the human used to do — not another model's reassurance.

The next time an AI review bot stamps LGTM on a diff the AI wrote, ask one question: what didn't the model generate that just ran and passed? If the answer is nothing, the review didn't happen.