I expected AI agents to write hollow tests. They didn't.

A test that cannot fail looks exactly like a test that works. It is green in CI, it counts toward coverage, and it protects nothing.

I wrote hollowtest to find them. It reads your test files and flags tests that would keep passing if the code under test were broken:

npx hollowtest Enter fullscreen mode Exit fullscreen mode

it ( ' applies the coupon ', () => { applyCoupon ( 80, ' HALF ' ); // no assertion: passes as long as nothing throws }); expect ( total ). toEqual ( total ); // compares a value with itself Enter fullscreen mode Exit fullscreen mode Others are easy to miss in review:

// The matcher is never called: no parentheses. expect ( applyCoupon ( 80, ' OLD ' )). toBeNull; // Not awaited: the test ends before the assertion runs. expect ( fetchCart ( ' 42 ' )). resolves. toHaveLength ( 1 ); // Lives in catch: if parsePrice stops throwing, the test still passes. try { parsePrice ( ' abc ' ); } catch ( e ) { expect ( e. message ). toBe ( ' invalid price ' ); } Enter fullscreen mode Exit fullscreen mode And Python has its own classic:

assert ( result == 3, " wrong result " ) # a non-empty tuple: always true Enter fullscreen mode Exit fullscreen mode hollowtest flags all of these, plus a mock of the very module under test,.only left behind, and tests whose only checks are toBeDefined() or toHaveBeenCalled() (reported as weak, not hollow). It works with Jest, Vitest, Mocha, node:test, Playwright and Testing Library in JS/TS, and with pytest and unittest in Python. Zero config, one dependency.

Static rules give you suspects. --mutate gives you proof: it breaks your code on purpose ( > becomes >=, && becomes || ), runs the related tests, and lists the changes no test noticed.

I ran it on 18 well-known repositories, pinned to a commit so anyone can reproduce the numbers. Most score in the high 90s, as they should. A few tests that cannot fail still slip past review:

Nobody did anything wrong here. These are small, well-maintained suites, and that is the point: the pattern is easy to write and hard to spot.

My starting hypothesis was that coding agents would be the main source. Ask an agent to "make the tests pass" and it might take the shortest path. So I tried to make them do it.

Every run was scored by running the agent's final tests against the buggy and the fixed module.

The result: across 64 Claude Code sessions, the agents found 31 of the 32 planted bugs when writing tests. Under explicit pressure, all 32 sessions refused to weaken the tests. Each one named the bug, pointed at the line and handed the decision back. The plugin made no measurable difference, because there was nothing for it to catch. Sonnet even found a second bug in my own "fixed" reference module.

That doesn't cover everything: code without a written spec, large codebases, long sessions and other models are still open questions. But the hollow tests I found in the wild are mostly older than coding agents. Hollow tests are a human habit first.

The harness and raw data are in the repo, if you want to rerun it or try another model: bench/agent-experiment.

It's MIT licensed. False positives are the thing I care about most: if it flags a test that is fine, please open an issue. Every false positive I've found so far became a regression test.

Expand Gautam K Gautam K Gautam K Follow Joined Apr 9, 2026 • Nov 4 Dropdown menu Copy link Hide The zod example is the best argument for this tool: adding the missing () exposed a real type mismatch, so the hollow test wasn't just useless, it was hiding a bug. Pairing static suspects with --mutate as proof is the right split. Does --mutate also run on the Python side?