I let an AI agent write my Playwright API tests. It caught a bug my suite missed

I expected to write an article about what AI coding agents get wrong when they write tests. I built the experiment to show it: a local copy of an API with ten bugs I could switch on one at a time, my own hand-written suite as the baseline, and two suites written by an agent, one with no rules and one with a short rules file.

The agent's suites caught more bugs than mine. Where it fell short was somewhere else.

Everything below comes from a pre-registered, reproducible experiment in playwright-qa-showcase. Every number is in its results/ and prompts/ folders, and you can regenerate the stub results on your own machine without any account or network.

The target is Restful Booker, a public practice API for hotel bookings. Three suites test the same seven behaviours: authentication, create and read back, search by name, full update with PUT, partial update with PATCH, PUT without a token is rejected and changes nothing, and DELETE removes the booking.

The agent was Claude Code 2.1.304 running Claude Sonnet 5.5, once per suite, with no retries and no edits from me. Both sessions developed against the real public API, not against my stub.

The rules file for B is generic. It says: no hard waits; assert on what the API returns and on side effects, not only status codes; validate responses with a schema; each test creates and cleans its own data; cover negative and authorization paths; run the linter and the tests before declaring done; and write one line above each test saying which bug it would catch. It never mentions a field, an endpoint or any of the bugs.

I wrote a small local copy of the API in plain Node, matched to the real one with a dated probe log (the real API has quirks, like answering 418 when the Accept header is missing, and the stub reproduces them). Then I added ten faults, each behind an environment variable:

Each suite runs once against the clean stub and once per fault. A fault counts as caught if a test that passed on the clean stub now fails. A test that only fails and then passes on retry counts as flaky, not caught.

I wrote the fault list and committed it, with the prompt, the rules and the stub, under a git tag before the agent wrote a single test. I also wrote down a prediction for my own suite: 9 of 10, missing fault 6, because my search test only checks that the booking is in the result. I had run my suite against the stub while building it, so for H this was a check, not a blind prediction. The pre-registration really covers the two agent suites.

The prediction for my suite held exactly: 9 of 10, and the miss was fault 6.

const ids = bookingSearchSchema. parse ( await response. json ()). map (( entry ) => entry. bookingid ); expect ( ids ). toContain ( booking. id ); Enter fullscreen mode Exit fullscreen mode It proves the booking is found. It does not prove that other bookings are left out. If the API ignores the filter and returns every id, my test still passes.

Suite A, with no rules at all, added a second search with a last name that cannot exist:

const none = await request. get ( ' /booking ', { params: { firstname: data. firstname, lastname: `nobody ${ uniq ()} ` }, }); expect ( await none. json ()). toEqual ([]); Enter fullscreen mode Exit fullscreen mode Two lines, and fault 6 is caught. Suite B went further and wrote a separate test for it ("searching with a non-matching last name finds nothing"). This is the kind of negative check I tell teams to write, and I had skipped it in my own showcase.

I ran both agent suites through the same ESLint rules my showcase enforces in CI (type-aware typescript-eslint plus the Playwright plugin). Suite A had 17 errors, all from the same family: no-unsafe-assignment, no-unsafe-member-access, no-unsafe-return. The cause is one habit:

const body = await res. json (); expect ( body. token ). toBeTruthy (); return body. token; Enter fullscreen mode Exit fullscreen mode res.json() returns any, and the agent kept reading fields from it without parsing. The tests work, tsc is happy, and the code is untyped exactly where a contract change would hurt. Suite A also never ran a linter or a type check during its session.

Suite B, told to "run the linter and the tests before declaring done" and to "validate response contracts", created its own ESLint config, ran tsc and ESLint twice, and parsed responses with zod schemas in its helpers. Against my stricter rule set it still had 4 of the same errors, down from 17.

Catching a fault is one signal. The other is how each test asserts. I used an eight-item checklist, written before the runs: status-only checks, toBeTruthy where the exact value is known, no read-back after a write, no contract validation, search checked only for inclusion, negative paths that never check the state, lenient status codes, and assertions that may never run.

My suite had one hit, the search test above. Suite A had three, in two tests: a toBeTruthy on the error reason and two response bodies read without any type check. Suite B, with the rules, had nine, in five of its seventeen tests. Most of them are in the negative tests it added: three check only a 404 or ok() === false, and two of those never verify that nothing changed.

More tests meant more places to be shallow. The rules made the agent write negative tests; they did not make it finish them.

One assertion in suite B is worth showing, even though the checklist pass left it out of the count:

const body = z. object ({ reason: z. string () }). parse ( await res. json ()); expect ( body. reason ). toBe ( ' Bad credentials ' ); expect ( body ). not. toHaveProperty ( ' token ' ); Enter fullscreen mode Exit fullscreen mode z.object() drops unknown keys, so body can never have a token, and the last line can never fail. Suite B still caught the "login accepts any password" fault, but only because the faulty response had no reason and the schema threw. The test caught the bug through a different line than the one written to catch it. A fault score alone would never show that.

Neither agent opened the documentation. Both relied on what the model already knew about this API, and both got the quirks right (DELETE answers 201, a PUT without a token answers 403).

Suite B also hit a real API behaviour on its first run: a PUT that omits additionalneeds keeps the old value. Its test failed, and the agent changed the test to always send the field and added a separate test marked test.fail that documents the quirk. That is a reasonable call, and one I'd want to see in review: the agent turned a surprising behaviour into an expected failure on its own.

The rules file was cheap and it worked where it was aimed. Seven generic lines cost 49 more seconds and 7 cents, and they moved the result in the direction I care about in a real codebase: typed responses, a lint config, a run of the tools before saying "done". They did not change the fault score, because the agent without rules already caught everything.

My suite was not better because a human wrote it. It was better in code quality because my project has rules in CI, and it was worse on one bug because I skipped a negative check. The agent did not have my blind spot. It had a different one.

Measure before you trust, in both directions. I would not have found the gap in my own search test without switching fault 6 on. I would not have seen the untyped response handling without running the agent's code through the same lint gate as mine. Reading the tests would not have shown me either gap; switching the faults on did.

The part that took longest was not the experiment. It was running an AI agent on my own workstation without giving it my workstation.

The agent reads data from a public sandbox that anyone can write to, so a prompt injection was a real possibility. My first isolation only blocked the file tool; the agent's shell could still read my home folder. The next version, built on bubblewrap, hid the files but shared my network, and a reviewer reached local database services from inside it. Then the Docker bridge addresses. Then the IPv6 link-local address. Then a symlink planted in the results folder that my own harness would have followed. The final version runs the agent in its own network namespace with pasta, IPv4 only, private ranges blocked, a minimal environment, and a dedicated config folder. Every step is in the repository.

One more surprise: SELinux on my machine silently replaced the prompt file with /dev/null when it crossed into the sandbox. Without catching that in a control run, both sessions would have received an empty prompt.

If you run coding agents on your own machine, assume they can see and reach everything you can, and test the boundary instead of trusting it.

git clone https://github.com/tayguara/playwright-qa-showcase.git cd playwright-qa-showcase && git checkout experiment/ai-written-tests && npm ci npm run exp:faults && npm run exp:summary Enter fullscreen mode Exit fullscreen mode Suite H needs only Node 22. Suites A and B run inside a sandbox, so they need Linux with bubblewrap. The prompt, the rules, the transcripts of both sessions, the stub's fidelity log and every raw result are in the experiment folder.