An AI agent tested Toolshop from just the URL: 23 issues, 8 real

Toolshop's sign-up page says a password needs at least 8 characters. The form enforces 8. The API enforces 8. Type 7 and the error message says the minimum is 6.

That's the best of 8 real bugs our AI testing agent found on Toolshop, the demo hardware store a lot of testers learn Playwright and Cypress on. It got only the URL. No test plan, no credentials, no hints.

This is Weekly Teardown 2. Each week we point TestOptim, the agent we build, at a public app and publish what it finds, including what it got wrong. Teardown 1 was ParaBank, whose admin console had no login.

In 30 minutes the agent explored 17 pages and mapped 16 user flows and 29 business rules. It wrote 289 tests, ran 71 of them before the time limit, and flagged 23 issues.

A list of 23 issues isn't worth much if you have to check all 23 yourself. So I did, on the live site.

1. The password rule changed, and its error message didn't. The sign-up page says a password must "Be at least 8 characters long." The form validator enforces 8:

// sprint5/UI/src/app/auth/register/register.component.ts PasswordValidators. minLength ( 8 ) Enter fullscreen mode Exit fullscreen mode So does the API:

// sprint5/API/app/Http/Requests/Customer/StoreCustomer.php Password:: min ( 8 ) Enter fullscreen mode Exit fullscreen mode But the error text lives in a separate translation file, assets/i18n/en.json, under fields.password.errors.minlength, and it still says "Password must be minimal 6 characters long."

The rule says 8 in three places and 6 in one. Each piece looks fine on its own, which is why this kind of drift is easy to miss in a manual review and easy to catch for an agent that reads the page, submits the form and compares the two.

2. The wrong error for a missing symbol. Leave the special character out and the error says "Password can not include invalid characters." Nothing is invalid; something required is missing. A user who reads that deletes characters, which is the opposite of the fix.

3. A menu link to a category that doesn't exist. Categories → Special Tools opens "There are no products found." The public API has no Special Tools category at all.

4. An empty cart breaks checkout. Open /checkout with nothing in the cart and nothing renders under the step bar. The console says why:

TypeError: Cannot read properties of null (reading 'cart_items') Enter fullscreen mode Exit fullscreen mode 5. A product that doesn't exist gives a blank page. Mistype a product URL and you get an empty page with a "Related products" heading instead of a "not found" message.

6. "Forgot password" says nothing. Enter an email, click "Set New Password", and the page stays as it was: no confirmation, no error.

7. "Show differences only" only half works. It filters the specifications, but rows that are identical for every product, like Brand, Category and Availability, stay on screen.

8. "2years". The product page prints the warranty as "2years". The comparison page prints the same value as "2 years".

9 findings didn't survive the re-check. They're the more useful half of this post if you're evaluating any AI testing tool.

6 more need an account to check (a stale cart total, an HTTP 500 on registration, the payment step), so I've left them out rather than guess.

The agent was strongest where people are weakest: a rule written down in several places that drifts in one of them, and pages that break in states nobody visits on purpose, like an empty cart or a mistyped URL.

It was weakest where it had to guess a business rule the site never states, or tell a demo's shortcuts from real bugs. Closing that gap is what we're working on next.

If you want to try it on your own app, give TestOptim your staging URL. It explores the pages, maps the flows and business rules, writes and runs the tests, and sends each bug with screenshots, repro steps and a severity to Slack, Jira or GitHub.

Got a public app for next week's teardown? Drop the URL in the comments.