It passes on my machine and fails in CI: a practical checklist (and where AI actually helps)

If you work in test automation long enough, you get a very specific kind of message from your pipeline. Red build. You pull the branch, run the same test locally, and it's green. You run it again. Green. You run it ten times. Still green.

This week I asked other testers for their strangest "only fails in CI" bugs, and the replies were a mix of comfort and horror. One person described a Selenium failure on a VM that devs, DevOps and testers had been chasing for over a year. Nobody had cracked it.

So this isn't a "five easy fixes" post. Some of these are genuinely hard. But most CI-only failures I've seen fall into a handful of buckets, and knowing the buckets makes the search a lot shorter.

CI usually isn't broken. Your machine is quietly giving the test something for free, and CI isn't.

Once you think about it that way, the question changes from "why is CI flaky?" to "what does my laptop have that the runner doesn't?"

Your laptop is probably faster than the shared runner, and nothing else is competing for its CPU.

So any check that doesn't wait will pass locally by luck and fail in CI. The classic one in Playwright:

// Checks once, right now. No waiting, no retry. if ( await page. getByText ( ' Saved ' ). isVisible ()) {... } // Retries until it passes or times out. await expect ( page. getByText ( ' Saved ' )). toBeVisible (); Enter fullscreen mode Exit fullscreen mode Same goes for reading text straight after a click, or a fixed waitForTimeout(2000) that happens to be long enough on your machine.

Quick check: run the test many times locally, ideally while your machine is busy with something else. If it starts failing, you've found your bucket.

npx playwright test my.spec.ts --repeat-each = 20 Enter fullscreen mode Exit fullscreen mode 2. Leftover state Locally, you've been logged in to that test account for weeks. The test data already exists. Some feature flag got switched on months ago and nobody remembers.

If your test only passes because something already happened before it ran, it'll pass on your machine forever and fail on a clean runner. The fix is boring but reliable: every test sets up what it needs (seed data, log in through an API, saved storage state) and doesn't assume anything else.

In Playwright you can pin a lot of this in the config, so local and CI share it by default instead of hoping they match:

// playwright.config.ts export default defineConfig ({ use: { locale: ' en-GB ', timezoneId: ' Europe/London ', viewport: { width: 1280, height: 720 }, trace: ' on-first-retry ', }, }); Enter fullscreen mode Exit fullscreen mode One of the replies I got put it well: the most useful habit is reproducing the CI environment as closely as possible (same OS, headless, browser version, locale, timezone, clean state) before you start guessing. If your CI runs in a container, run that same container locally.

This one is the most invisible, and someone in the thread raised it better than I would have.

Your laptop may be on a corporate VPN, behind a proxy, using split DNS, or trusting a company certificate authority. The CI image may have none of those.

And here's the trap: a curl or Node health check passing in CI doesn't prove the browser is using the same proxy settings. They can take completely different routes.

To tell a network problem apart from a slow element:

And please don't "fix" a certificate error by turning on ignoreHTTPSErrors and moving on. The test goes green, but you've hidden exactly the difference you were looking for. Get the CA into the CI image properly.

Tests that pass alone and fail together. Two workers editing the same user, the same basket, the same row in a table. Locally you're often running just the one spec you're debugging, so you never see the collision.

Quick check: run the failing test on its own in CI. If it passes alone and fails in the full run, look for shared data.

Retries have their place. But as one reply put it: retries can hide the cause, but they rarely explain it.

npx playwright show-trace path/to/trace.zip Enter fullscreen mode Exit fullscreen mode One caution: traces capture network requests, headers and page content. Check them for tokens and internal hostnames before attaching one to a public ticket.

I use AI assistants for this kind of debugging most days now, and they've changed how fast I get through the boring parts. But they're better at some bits than others.

The way I think about it: AI shortens the time between "the build is red" and "I have three good theories". Proving which theory is right is still your job, and the trace is still the evidence.

The year-long Selenium VM mystery is still bugging me. What's the strangest CI-only failure you've tracked down, and what turned out to be the cause? Drop it in the comments. I'm collecting them.

I'm building Locator League, browser-based Playwright challenges for practising this kind of thing (waiting properly, mocking, network checks) without setting up a project.