Evals für ein Claude Code Plugin entlarven Fehler in der eigenen Doku

Every new project I started, I wrote the same Claude Code setup by hand: a CLAUDE.md, a few guard hooks, some reviewer agents. Every template I tried gave me a web-app config with brackets to fill in, which is the wrong shape for a CLI, a Terraform repo or a smart contract.

So I built a plugin that does it for me. You describe the project in one paragraph, it works out the project's traits (language, whether there's a UI, a database, a trust boundary, whether it's published), and it writes the harness that follows from those. A CLI with no database gets no persistence guard. A published library gets a semver reviewer that an app doesn't need.

I tested it by hand on ten very different projects, from a game server to a C++ audio plugin, and fixed what broke each time. It felt done. Then I wrote an eval suite, and it kept telling me things I didn't want to hear.

Claude Code has a built-in way to test plugins: claude plugin eval. Each eval case is a real Claude Code session. You give it a prompt, it runs, and graders check what came out. I started with eight cases, one for each thing the plugin claims to do:

Graders come in two kinds. Some are deterministic, like this one, which just checks the skill was invoked:

--- type: tool_used tool: Skill input_match: ' "skill"\s*:\s*"(?:[\w-]+:)?init"' weight: 1 --- Enter fullscreen mode Exit fullscreen mode Others are an LLM reading the output against a written rubric, for things a regex can't judge, like "is this agent actually in the roster, or only mentioned as something to add later?"

The README said you could just describe your project and the setup would start. The first eval run said otherwise. Given only a project description, with no mention of Claude Code or setting anything up, the skill never fired.

Its trigger condition only matched requests like "set up Claude Code for this". The description was narrower than the job. I had never noticed, because when I tested by hand I always knew what to type. I widened the trigger, and self-triggers-on-bare-description now makes sure it stays fixed.

A failing eval is a claim about either your code or your grader, and you have to check which.

In September all eight cases passed. In October, on a newer Claude Code version, vague-description failed twice in a row. Given "I want to build an app", the plugin said it was guessing, then guessed a very specific stack anyway: Next.js, Prisma, Auth.js, Vercel. It also ended half-built, with a CLAUDE.md describing agent files that were never written.

The fix: a description that names no kind of app now gets exactly one question, and before finishing the plugin checks that every file CLAUDE.md mentions actually exists on disk. Back to 8 of 8.

This is the one I least wanted to see. Every run so far had used my account's default model. When I added a --model flag to my eval runner and ran the same plugin on the smaller models, version 0.6.1 passed 3 of 8 on Haiku, and 3 of the 6 Sonnet cases that finished before the run was cut short.

The misses had patterns. On both models, the skill didn't trigger for "set up a harness for this existing repo", because its description leaned on describing a new project. Haiku also missed detail in the spec (dev commands, which reviewers to include, keeping CLAUDE.md short), which makes sense: the spec it follows is 55KB.

After fixing the trigger and adding four new cases, version 0.7.0 passes 9 of 12 on Haiku and 9 of 12 on Sonnet. It's not 12 of 12, and the README says so. Most of the remaining misses share one cause: in an eval nobody can answer the two setup questions, and smaller models sometimes ask and then stop without writing anything. Taking the recommended answer when no reply can come is the next change.

All of those are one run per case, which is noisy. I treat a single pass or fail as a signal, not a verdict.

Evals have a blind spot here. Eval runs aren't allowed to write into.claude/, so the graders can check CLAUDE.md and what the session tried to write, but never the hooks that actually guard your project.

The hooks are just scripts, though. Claude Code sends each one a JSON description of the tool call it's about to make, and the hook decides whether to block it. So I wrote a test runner that skips the model entirely: it feeds each hook 13 attack payloads and checks that they're blocked.

The 13 are what coding agents actually get hit with:

Each attack has a harmless twin that must not be blocked, so a guard that blocks everything fails too. Every attack runs from the project root and again from a subfolder. With no model calls, the whole suite runs in seconds.

The result for the seven example harnesses the plugin had generated: 0 of 13 blocked by hooks, 2 of 13 counting permission rules. The guards worked as written; they just didn't cover any of this. The three examples regenerated with 0.7.0 now block 8 or 9 of 13 with hooks, and 13 of 13 counting the permission rules.

It's open source, MIT licensed, and listed in Anthropic's plugin directory. Install it in Claude Code:

/plugin marketplace add sagaga144/claude-harness-bootstrap /plugin install harness-bootstrap@harness-bootstrap Enter fullscreen mode Exit fullscreen mode Then run /harness-bootstrap:init in any project folder, or just describe your project.

The repo has the eval suite and per-model results, the attack suite and its matrix, a demo GIF, and seven example harnesses for projects from a Rust monorepo to a JUCE audio plugin: github.com/sagaga144/claude-harness-bootstrap

If you try it on a kind of project it hasn't seen, I'd really like to hear what it got wrong. And if it saved you setup time, a star helps other Claude Code users find it.