METR's AI Coding Speed Study Broke: Too Many Developers Wouldn't Work Without AI

In early 2025, METR paid 16 experienced open-source developers $150 an hour to work 246 real issues in large open-source repositories they had worked on for years. Each issue was randomly allowed or disallowed AI. With AI allowed, the tasks took 19% longer. Before starting, the developers forecast a 24% speedup. After finishing, they still believed AI had sped them up by 20% ( METR, July 2025 ).

Then METR tried to run the study again, with newer tools and more people. It couldn't get a clean answer. Too many developers wouldn't agree to work without AI.

That second result matters more than the first.

The new experiment started in August 2025: 57 developers, 143 repositories, more than 800 tasks. In February 2026, METR published what happened ( METR, February 2026 ). Its verdict on its own data: it "gives us an unreliable signal of the current productivity effect of AI tools."

The main reason was recruitment. METR saw a sharp rise in "developers choosing not to participate in the study because they do not wish to work without AI." The ones who did join filtered their own work too: "30% to 50% of developers told us that they were choosing not to submit some tasks" because they didn't want to do them without AI. METR thinks both effects make its numbers understate how much AI speeds developers up.

Two more problems stacked on top. Pay dropped from $150 an hour to $50, which METR believes added its own selection effect. And for "the fraction of developers who use multiple AI agents concurrently," METR says its time-per-task measurements are unreliable. If you kick off an agent and switch to something else while it runs, how long did the task take?

The numbers METR did get, in its own wording: for returning developers, "a speedup of -18% with a confidence interval between -38% and +9%." For newly recruited developers, "the estimated speedup is -4%, with a confidence interval between -15% and +9%." The same post gives the 2025 result as 19% longer, with an interval of +2% to +39%, so on this scale a minus sign means tasks took less time with AI. Both estimates lean toward AI making developers faster, the reverse of 2025. The intervals also cross zero, so neither settles the question. METR's own read is that developers are probably more sped up now than in early 2025, and that this data is very weak evidence for how much.

A randomized trial needs both sides. In METR's follow-up, fewer developers would take the no-AI side.

So a randomized trial built to measure AI coding speed lost its control condition. Working without AI became something many developers refused to do, and at $50 an hour instead of $150, fewer were willing to try.

METR also asked people. Randomized trials are expensive and surveys are cheap, so it ran one from February to April 2026, covering 349 technical workers, 87 of them software engineers. The median self-reported speed change was 3x. The median self-reported change in the value of their work was 1.4x to 2x ( METR, May 2026 ). The median respondent also said they would give up 29% of their salary to keep AI tools for a month, which METR flags as weak evidence.

METR put its own warning right next to those numbers: "survey results are not necessarily grounded in reality." The same post recalls that in the 2025 trial, "people overestimated AI's effect on their time spent on tasks by 40 percentage points on average."

I'm not arguing that AI makes you slower. The 2025 trial used early-2025 tools, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, on large, mature repositories the developers knew well. METR itself thinks things have moved since. As of mid-October 2026, I haven't found a newer METR trial result that replaces either estimate.

My argument is narrower. The instrument everyone reaches for, how fast it feels, was checked once against a stopwatch and missed by about 40 points. The stopwatch is now hard to run, because the people being timed won't give up the tool. That leaves feel. And feel is the one measurement with a known error bar, and the error bar is huge.

The 2026 version of "AI makes me faster" is "I run five agents in parallel." It sounds like a number. It's an input.

Look at what broke METR's stopwatch: developers running several agents at once and doing other work while they ran. That's the workflow people often cite as proof of speed. Launching more agents is easy to count and easy to feel good about. It tells you nothing about what reached your main branch.

An agent that runs for forty minutes and hands you a diff you throw away still cost you the review. An agent whose change lands and gets reverted two days later cost you more. Neither shows up in "agents launched."

You can count three things without a stopwatch or your gut:

None of these measures speed. They count output that survived review, which is closer to what a team pays for. They're crude and they lag, but they don't depend on how the week felt.

If your agent work ends in pull requests, the GitHub CLI and git can give you the three numbers. This script counts the current repository's last seven days:

#!/usr/bin/env bash # weekly-landed.sh: what landed in the last 7 days since = $( date -d '7 days ago' +%F 2>/dev/null \ || date -v-7d +%F ) opened = $( gh pr list --state all --limit 500 \ --search "created:>= $since " \ --json number --jq length ) landed = $( gh pr list --state merged --limit 500 \ --search "merged:>= $since " \ --json number --jq length ) dropped = $( gh pr list --state closed --limit 500 \ --search "is:unmerged closed:>= $since " \ --json number --jq length ) reverted = $( git log main --since = " $since " \ --oneline --grep = '^Revert' | wc -l | tr -d ' ' ) echo "since: $since " echo "opened: $opened " echo "landed: $landed " echo "dropped: $dropped " echo "reverted: $reverted " Enter fullscreen mode Exit fullscreen mode The date line covers both GNU ( -d ) and macOS ( -v ). The revert count only catches commits that kept git's default Revert "..." subject, and it assumes your trunk is called main and your local copy is up to date. Branches you abandoned without opening a PR don't show up in dropped.

The PR counts include every pull request in the repo, human or agent, which is fine: the team's output is what you're tracking.

Run it every Friday and write the numbers down. Compare this week with last week, not with another team. If you add a second agent and "opened" doubles while "landed" stays flat, you have your answer.

I build Hermes IDE, so weigh this section with that in mind.

Counting landed work is easier when every agent task has a clear start and a clear end. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on a fresh hermes/<name> branch and worktree, with the task as the agent's first prompt. Every session shows one status: working, needs approval, asked you, done, error or idle, and Hermes says when that status is a guess.

Two agent sessions side by side in Hermes: Codex on orders-types and Claude Code on orders-api, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.

A task stays a task until you decide what happens to it. Hermes saves every agent turn, so you can diff or restore any single turn. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows the changes by turn or by file, with risk flags, and a line comment goes back to the agent that wrote the line.

The Review Desk (⌘G) by turn: two turns of a Claude Code task in orders-api, turn 1's diff, the changed files, and a commit message drafted from the turns.

Then there's Land. It commits, opens a pull request or merges locally in one step, into the branch the task started from, and it can be undone. Your own checks can decide when a task counts as done. When Land opens a pull request and it merges, the script above counts it. A local merge won't show up in the PR numbers, so count those from git log if that's how you land.

Hermes has no productivity dashboard and doesn't measure your speed. The count still comes from git and GitHub. Every agent runs its own interface in a real terminal, with your sign-ins, so Hermes only adds the branch, the status, the review and the Land step around it. If you use one vendor, its own tool may be all you need: Claude Code moves background sessions into worktrees under.claude/worktrees/ by default, and the Codex app runs parallel chats in worktrees under $CODEX_HOME/worktrees.

Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It isn't affiliated with Nous Research, Anthropic, OpenAI or any agent vendor.

For most of us: I don't know. METR, with randomization, a stopwatch and a budget, couldn't get a reliable number this year. You won't get one from how a Tuesday felt.

Run the script this Friday, and again next Friday before you add another agent.

Would you have taken METR's no-AI tasks, even at $150 an hour? And if you track agent output on your team, what do you count? Tell me in the comments.

If you want to try the count with Hermes, download it, give one task to two agents, and see which one lands. If it helps, star it on GitHub.

Counting what lands only works if your agents are built to finish a task and stop. The AI Agents Pocket Guide covers the loop, tool and guardrail patterns behind that, from a single agent to several working in parallel.