AI code review: what the papers and tools say, compared to my skill

Posted on Nov 2 • Originally published at web-developpeur.com

When you change code, a colleague reads your changes before they go live. That is code review. I have AIs do it, with a tool I wrote for Claude Code, Anthropic's programming assistant. That tool is a skill: a folder of instructions that Claude Code reads before a specific task.

My skill, senior-review, launches several reviewers. A reviewer is an instance of Claude launched to read the code (you can also call it an "agent"). It does not know the author of the code, the person who wrote it ( I explained why the reviewer must never be the author ). Each defect a reviewer flags is called a "finding". I use this skill almost every working day: 236 reviews since August 14.

The skill has three settings, called tiers. "quick" is light, "standard" is the default, "deep" is supposed to be more rigorous, with more reviewers. It keeps a journal: for each review, it records how many findings were raised and how many pass verification. Verification demands proof for each defect and tries to refute the most serious ones, meaning to show that they are false.

The journal says otherwise. On "deep", 74% of the findings raised survive verification. On "quick", it is 89%. Apparently, I had paid more for reviewers that miss more often.

Those numbers made me want to check whether my skill is solid or a hack I happen to like. I launched two research agents: one in the scientific literature, the other among existing tools.

I compare my skill to other AI review tools. There is Anthropic's, commercial tools (CodeRabbit, Greptile, Qodo, Bugbot) and community skills. I kept nine criteria.

I added three criteria that were missing: externally validated measurement, integration and cost. Externally validated measurement: has someone other than the vendor tested the tool? Integration: does the tool work with no setup in "pull requests", the requests you post to have your changes reviewed?

Each column of the table is a tool or a group of tools, each cell a score from 0 to 5 (5 is the best). It is my judgment, based on the documentation I read. "Not documented" in a source does not mean "absent", and I scored accordingly. "1-2" covers the tools of the group. Look at proof (5 for my skill) and external measurement (2).

"Official Claude Code" covers the managed review service from Anthropic and its code-review extension. That extension launches four agents at the same time. Other agents then check the defects found, one per defect: an unconfirmed defect is dropped. Its documentation cites a confidence threshold of 80, which is no longer in the code.

My strengths are the required proof, the ticket, the loop with a limited number of rounds and the memory. My weaknesses: high cost, no native pull request integration, and above all weak external measurement. The commercial tools display benchmarks (numbered tests), but each vendor wins on its own measure, such as "comments resolved before merge". Those numbers cannot be compared. I cite none of them.

The research agent opened the page of about thirty papers. The figures come from their abstracts, except one, taken from the body of the Cihan study. Several are "preprints", texts not yet reviewed by other researchers. Six results matter for a review skill.

One last figure comes from a real tool, CodeRabbit. Out of 31,073 reactions to its reviews, across 239 repositories, 56.3% of the comments were rejected. They were mostly false alarms, repeats or off-topic remarks (Lin et al., arXiv 2607.03316). That is a single tool, but the order of magnitude is a warning: more than one comment in two ends up rejected.

Here is the output of the learn command, which analyses the journal. It keeps only the reviews whose counts add up: 189 out of 236, of which 182 fall in the three tiers of the table. The last column gives the cost of a finding, in tokens.

The cost depends on the number of agents. A finding costs about 66,000 tokens with 1 agent, 50,000 with 4 to 5, then 87,000 from 6. Nothing shows that adding reviewers helps beyond five.

Be wary of these readings, though: the reviews are not about the same changes. A "diff" is the list of lines changed by a change. Its median size (the middle value) is 158 lines in quick against more than 1,000 in deep.

Single-agent reviews mostly re-read a small change, round after round. The comparison "one big reviewer against several", at an equal budget, remains to be done.

My real gap is elsewhere: the way to read that "74%". Here is the path of a finding.

The path of a finding: reviewers flag defects, verification demands proof, refutation tries to show they are false, findings without proof or refuted are discarded, the ones left go through my hands to be fixed, rejected or moved to another ticket. The reviewers they do not know who wrote the code findings Verification proof required for each finding Refutation we try to refute the most serious Discarded no proof or refuted Findings left they survived "precision" counted here then My triage fixed, rejected or moved to another ticket "acceptance" counted here

To picture it, think of a dictation. A first corrector underlines words: the reviewers. A second one strikes out those it judges correct: the verification. The words still underlined look like mistakes. Yet the pupil, who is me, may refuse one: "it is a character's name, it is on purpose". Counting them measures the second corrector, not what the pupil accepts.

My journal did the same. Its "precision", the share of findings that survive verification, measured what my own filter lets through, not what I accept. A confirmed finding may well be rejected by me ("that is intended"). The 236 lines of the journal did not carry what became of each finding. A tool that grades itself with its own filter measures only that filter.

The journal and the research made me change three things, published in the skill's public repository (version 1.26).

A fourth change, version 1.25, comes from a book on agent architectures (how to organise several agents) that I was reading in parallel. A reviewer can crash or return an empty answer. The skill then relaunches it once. On the second failure, the report writes "not covered" next to the aspect of the review it was meant to check. The final verdict can no longer be "approved". Before, a reviewer that dropped out left a hole the report did not show.

A separate agent read the code for these changes. It found six defects, including a case where two simultaneous writes to the journal could make a line disappear. Each got a failing test before the fix.

I did not plug the skill into pull requests. It is interactive, built for my workstation: I launch it myself. Plugging it in would need a program that launches itself on every code push. It would need setup, passwords and a charge on every push. For a solo use, it is not worth it.

I did not add a reviewer from another model family either. It would need a second account, with the vendor of that other model. The study that justifies it is about a language task, not code.

What remains is the experiment the research asks for: one big reviewer against the group of reviewers, on the same diffs, at an equal budget. By my estimate it would cost about 300,000 extra tokens per trial review. I am not running it right away, because the journal first needs a few weeks of recorded outcomes so that the comparison speaks about acceptance and not about the filter.

My recall, meaning the share of defects a reviewer finds, is measured on only seven cases with injected bugs. An injected bug is a defect planted on purpose to test the reviewer. Depending on the bug, it ranges from 3 out of 3 to 1 out of 3. Seven cases settle only a coarse gap.

No verified paper gives the ideal diff size either. None gives the point from which several agents earn more than they cost. The token ratios I had read about teams of agents are found in none of the papers the research agent opened.

The most useful part of this research was neither a paper nor a tool. It was discovering that my own measurement graded itself with the filter it was supposed to judge. The only score that matters for a review is what the reader of the review does with the findings: fix them, or dismiss them. Everything else, the architecture included, is a hypothesis as long as that number is missing.

If you have an AI review your code, note what you do with each defect (fixed, rejected, moved to another ticket). Ten lines of log are enough to see whether your review is worth anything.

The book on agent architectures mentioned above has its page in my library. The skill is public: github.com/ohugonnot/claude-skills, senior-review folder, with the history of every version.