I benchmarked my own agent skill, and today's Claude barely needed it

I wanted to write an agent skill that makes coding agents better at data analysis. The kind of mistakes I had in mind are the ones that produce a confident, wrong number: a "Total" row summed together with the countries, a join that duplicates orders, averaged p99s, timed-out requests that vanish from the latency percentiles, an A/B test that was supposed to split 50/50 and came back 50.7/49.3.

Every rule in the skill comes from a paper or official documentation. But before claiming it helps, I wanted to measure it.

I built 30 realistic cases, each with one planted trap and a correct answer computed by code:

Each case ran with and without the skill on Claude Haiku 5.5 and Sonnet 5.5, graded by an LLM judge (Sonnet, 3 votes per grader).

Today's Claude models already avoid nearly all of these traps, including the harder performance ones. The skill fixed every baseline miss ( -1 timeout sentinels, which both models missed, plus a closed-loop load test hiding a stall and an overhead number reported without uncertainty on Haiku) and introduced no new failures. But with one run per arm, that's "catches occasional slips", not a large effect. That's what the README says.

1. Skills help most where the model's knowledge is weakest. The same week I wrote a skill for upgrading to TypeScript 7, which shipped after the models' training cutoff. There the gains were real: graders passed went from 10/18 to 17/18 on Sonnet and 7/18 to 15/18 on Haiku, on small samples with offline agents. Mainstream analysis practice is already in the model. Post-cutoff releases and exact operational details are not.

2. Check your test cases before you trust a baseline failure. Across both projects, several of the "failures" I first saw were flaws in my own cases: an effect I hadn't intended in the data generator, a grader that was too strict, a premise that contradicted itself. Once, Haiku correctly pointed out that a case's premise was wrong.

3. A skill that doesn't activate doesn't help. In the benchmark the skill activated by itself in 22/30 Haiku runs. Its description listed data types and trap names. The prompts it missed asked for an action: "should we roll back?", "confirm it's fixed", "measure the overhead". I rewrote the description to say when to load it ("at the start of any task whose answer will include a number … before reading files, running benchmarks or editing code"). In a short trigger test on Haiku (first 3 turns, 2 rounds), activation on 10 queries the description was never tuned on went from 7/20 to 20/20, with no activations on 10 unrelated queries.

The benchmark runs against any agent, with or without the skill. If you run it on another model or agent, I'd like to see the results: https://github.com/boranfurkan/trustworthy-data-analysis