I benchmarked 14 AI coding agents on the same task — same model, wildly different results

In my last post, benchmarking a small local LLM exposed a bug in my own harness — the "weak model" was partly my code. That raised the obvious next question: how much does the tool around the model matter, measured across many tools?

So I ran 14 agent-type AI coding tools through the same task set, on the same ordinary PC (Windows 11, Ryzen 5 4500U — no GPU, nothing special), with two model backends:

Scope: agent-type tools only — things that autonomously read files, edit files, and run commands. Chat assistants and subscription products (Copilot, Claude Code, Cursor, Devin) are out of scope.

Real tasks: create a file, search/read and answer, perform a precise edit, diagnose+fix a failing test. Success rates count only the last 2 attempts per task, and results identical to baseline ("did nothing but claimed done") are marked failed.

Same model, same tasks — and the spread is 100% → 0%. The model is constant; the harness (prompt assembly, tool-call format, retry logic, context handling) is what varies.

The weak local model is a harsher judge: tools whose harnesses were tuned for strong cloud APIs fall apart (brittle tool-call formats, no retry on malformed output), while tools with tolerant parsing and explicit command protocols still complete tasks.

If you're choosing an agent tool, don't ask "which model is it using" — ask "what does the harness do when the model misbehaves." That's where the success rate lives.