Stop telling Claude to "handle edge cases". I ran it 504 times.
You've probably written this into a Claude Code prompt: "Make sure you handle all the edge cases." "Write the tests first." Or you bumped effort to high. So did I. So I measured whether it does anything. 504 runs later: it mostly makes the bill bigger. Haiku 5.5: "handle edge cases" + "read the exi
Posted on Nov 3 • Originally published at tokensave.app
You've probably written this into a Claude Code prompt:
"Make sure you handle all the edge cases." "Write the tests first."
Or you bumped effort to high. So did I. So I measured whether it does anything. 504 runs later: it mostly makes the bill bigger.
21 real bugs fixed upstream between August and October 2026 in 12 open-source projects: cachetools, zod, marked, js-yaml, anyio, undici, fastify, p-queue, python-dateutil, Python-Markdown, qs, tomlkit. Claude Code 2.1.295, headless, Haiku 5.5 and Opus 5.5, four conditions, 3 runs each. 21 × 2 × 4 × 3 = 504.
Scoring used the maintainers' own tests for each fix. Before the first run I labelled every test case as "shown in the report" or "left out" and pre-registered it on GitHub with sha256 hashes, so I couldn't move the goalposts later. The bar for calling something an effect: 20 points or more.
Share of the left-out cases passed (mean per bug):
Median time per run went from 121s to 299s for Haiku and 120s to 271s for Opus. The models did exactly what they were told: listed variants, wrote tests. It just didn't turn into better fixes.
21 bugs, 3 runs per cell. A 20-point effect is ruled out; a few points is not. One prompt wording per condition; a different wording or agent might do better.
All 504 runs (scores, cost, time, tokens) are public under CC BY 4.0, with the pre-registration and the scoring code:
What do you add to your Claude prompts that actually works? Drop it in the comments and I'll test it in the next round.