How to Review AI-Generated Playwright Tests (Before They Break in CI)

I asked an LLM to write a Playwright test for a simple login flow. It passed the first time. Then I changed the button text from “Sign in” to “Continue” and re-ran it. The test failed. The model had used a CSS selector tied to a class name and a fixed sleep. Classic brittle patterns — just generated faster than a junior engineer would have written them by hand. This is becoming normal. Teams are using AI to draft Playwright tests, and the same failure modes from the pre-AI era are still showing up. The difference is volume and speed. If you don’t review the output, you’re shipping flaky tests into CI on purpose. Here’s what to look for and how to fix it.

Most models default to patterns they see a lot in training data:

The generated code often looks plausible and even passes once. That’s what makes it dangerous.

Before you accept an AI-written test, run it through these questions:

You don’t need to rewrite the whole test. Spot the weakest locator or wait and fix that first. One solid change often removes the flakiness.

Here’s a typical AI-generated version of a login test:

test('user can log in', async ({ page }) => { await page.goto('https://example.com/login'); await page.waitForTimeout(2000); await page.locator('#username').fill('test@example.com'); await page.locator('div.form-group:nth-child(2) input').fill('password123'); await page.locator('.btn-submit').click(); await page.waitForTimeout(3000); await expect(page.locator('.success-message')).toBeVisible(); });

Problems: fixed sleeps, CSS and positional locators, and a weak assertion. Revised version: test('user can log in', async ({ page }) => { await page.goto('https://example.com/login'); await page.getByLabel('Email').fill('test@example.com'); await page.getByLabel('Password').fill('password123'); await page.getByRole('button', { name: 'Sign in' }).click(); await expect(page.getByText('Welcome back')).toBeVisible(); });

Same flow, but the locators target what the user (and assistive tech) sees, and the assertion waits for the actual outcome. This version survives a class rename, a DOM reorder, and a button text change more reliably.

Reading checklists is useful. Doing the review under pressure is better. On LocatorLeague the Foundations track is free and lets you write and run Playwright locators against real pages with instant pass/fail feedback and a video of the run. The upcoming tracks will include reviewing AI-written tests so you can practice exactly this skill.

What’s the worst locator or wait an AI has written for you recently? Drop it in the comments — the patterns are usually familiar once you see a few of them.

This is roughly the same length and tone as the previous article. You can publish it as-is on Dev.to (or Hashnode) and cross-post. The link to LocatorLeague sits naturally after the practical content.