317 gemergte AI-Agent-Pull-Requests gegenuegstellt: Wie gut sind die Tickets wirklich erledigt?
Eine Analyse von 317 oeffentlichen Pull Requests aus ebenso vielen Organisationen prueft, ob die jeweils referenzierten Tickets nach dem Merge tatsaechlich abgeschlossen sind. Alle PRs stammten von Coding-Agents wie Claude Code, Devin oder Copilot.
Posted on Nov 2 • Originally published at lians.ai • AI-assisted
Coding agents now open a lot of pull requests that say Fixes #123. I wanted to know how often the ticket is actually done when that PR merges.
So I took 317 public pull requests from 317 different organisations. Each one was written by a coding agent (Claude Code, Devin, Copilot's coding agent or Cursor), was merged, and closed at least one issue. I checked every requirement in each issue against the diff.
Most agent PRs do what the ticket asks. The ones that don't are hard to spot in review, because the reviewer reads the diff, not the ticket.
Of the 70 requirements that were missed or only partly done:
Each PR was checked by Lians check. Claude reads the issue the PR closes and its discussion, splits it into requirements, and marks each one met, partial, missed or "can't tell" from the diff.
My first pass was wrong. It graded every issue a PR so much as mentioned, including "#123 stays separate", and it ignored scope changes in the issue's comments. That run said 44% of PRs missed something. Hand-checking a sample showed most of those were false alarms. I fixed both problems and re-ran everything. On the hand-checked findings, about 3 in 4 of the new run's "missed" or "partial" calls held up. So read 15% as an upper bound, and the true rate as closer to 1 in 9.
Another 323 requirements came back "can't tell", usually because the evidence was a CI run, a sibling PR or code outside the diff. They aren't counted as gaps. The sample is public repositories found through GitHub search, so it leans toward smaller teams and open-source companies. I'm not naming any repository.
When you review an agent's PR, read the ticket as well as the diff, and look hardest at PRs whose ticket changed after the work started.
If you'd rather not do that by hand, Lians check does it on every PR. It runs in your own GitHub Actions, posts one comment listing each requirement, and re-checks the open PRs linked to a ticket when someone edits it. Here's what a check looks like on a real PR: https://github.com/ebeirne/lians-check-demo/pull/2
I'm looking for a few teams using coding agents on real repos to try it and tell me what's useful and what's noise. Full write-up and method: https://www.lians.ai/agent-prs