Testing an AI Code Reviewer Against Real GitHub Pull Requests

Building an AI code reviewer is one thing. Trusting it with real pull requests is another.

I've been developing CodeGuard AI, a self-hosted AI-powered GitHub PR reviewer built with Java 21 and Spring Boot 3.

Instead of relying only on sample code and happy-path demonstrations, I've been testing it against deliberately created GitHub pull requests with different bugs, test scenarios, and edge cases.

So far, the test lab has covered scenarios across several areas:

Builds that succeed without executing tests

Incorrect implementations hidden behind passing tests

Harmless code that should not trigger unnecessary findings

Whether tests actually validate changed behavior

A few results have been particularly useful:

CodeGuard correctly identified a genuine failing assertion and showed the expected versus actual values.

It detected an incorrect discount calculation even though the existing test passed.

It correctly classified a timed-out test execution as TIMED_OUT.

It avoided flagging valid Optional.orElse() usage.

It identified missing regression coverage without claiming that the production implementation contained a confirmed bug.

However, testing also exposed areas that need improvement.

For example, some failing test scenarios received a warning instead of a blocking merge recommendation. In another scenario, test execution was unverified while the merge decision was still marked as safe.

These are exactly the kinds of inconsistencies I want to uncover before calling the product reliable.

An AI code reviewer should not simply produce more findings.

Distinguish real bugs from false positives.

Understand whether tests actually executed.

Separate verified evidence from AI inference.

Avoid suggesting fixes that weaken valid tests.

Make merge recommendations consistent with verification results.

A green build does not automatically mean the tests passed. And a suggested fix is not necessarily a correct fix.

I'm working through a 100-scenario validation plan.

The next focus areas include repository context, regression detection, team policies, suggested-patch validation, review consistency, and merge decisions.

I'm documenting the failures alongside the successful results because both matter.

CodeGuard AI is available as a source-code product for developers interested in exploring the implementation.

I'd especially like feedback from developers who maintain CI/CD pipelines or GitHub PR review workflows:

What should an AI code reviewer absolutely verify before recommending that a pull request is safe to merge?

Built with Java 21, Spring Boot 3, GitHub integration, and AI provider support. Explore CodeGuard AI and get the source code here: https://javacoder716.gumroad.com/l/codeguard-ai?utm_source=devto&utm_medium=article&utm_campaign=codeguard_testing_100