The Biggest Problem With AI Test Automation Isn't Writing Tests

A few years ago, creating automated tests was one of the biggest bottlenecks in software testing.

You needed someone who understood the application, knew how to write Selenium or Playwright scripts, and had enough patience to deal with selectors, timing issues, and browser differences.

AI is making the first part considerably easier.

You can describe a user journey in plain English and have an AI agent generate a test. Some tools can even execute it, analyze failures, and attempt to repair broken steps automatically.

But I think we're about to discover that creating tests was never the most expensive part of test automation.

And AI doesn't automatically solve that problem. In some situations, it introduces new problems that are harder to detect.

Imagine an automated checkout test that clicks the "Place Order" button.

A developer changes the application, and the original selector stops working.

The testing platform uses AI to find a replacement element. It selects another button, executes the action, and marks the step as successful.

Except the replacement button was "Save for Later."

The automation worked. The test may even pass if its final assertions are weak.

But it no longer verifies what it was supposed to verify.

This is why I believe the next major challenge in test automation isn't test generation.

It's knowing whether your automated tests are still testing the right things.

Self-healing has become a popular feature in AI-powered testing platforms.

When an element changes its ID, class, or location in the DOM, the testing tool attempts to identify the correct replacement instead of failing immediately.

That can eliminate a significant amount of routine maintenance.

But there's a distinction between repairing a locator and changing what a test does.

The application still has the same login button, with the same label and functionality. Only its CSS class changed.

An automated repair is relatively low risk, provided the replacement identifies the same control.

Scenario B: Two buttons have identical labels.

The application now contains two "Submit" buttons in different sections.

Which one? And how do we know it made the right choice?

The application now requires an additional confirmation step.

The AI decides to skip that step or modify the test to accommodate the new behavior.

At this point, we're no longer talking about locator repair.

We're talking about changing the test's intended behavior.

Those decisions shouldn't be treated as equivalent.

A useful benchmark methodology for evaluating AI testing platforms separates three important capabilities:

These are practical requirements, especially for teams maintaining large test suites.

If a platform changes a test automatically, I want to see the original step, the replacement, the evidence behind the decision, and the final outcome.

And if the platform cannot reliably distinguish between a harmless locator change and a change in application behavior, a human should be involved.

One of the most frustrating things about browser automation is that a perfectly functional application can still produce unreliable tests.

Imagine an application displaying 10,000 customer records.

Rendering every record in the DOM would be unnecessarily expensive. Many modern interfaces solve this by rendering only the rows needed for the current viewport.

As the user scrolls, rows are added, removed, or recycled.

But it creates an interesting problem for automated tests.

Suppose your test expects all 10,000 rows to exist in the DOM.

The application isn't broken. The test simply made an incorrect assumption about how the interface works.

The same thing can happen with infinite scrolling, lazy-loaded content, and scroll restoration.

I've found that a useful way to approach browser automation is to separate user-visible behavior from implementation details.

For a virtualized table, I wouldn't start by asserting an exact number of DOM elements.

There's a detailed guide to testing virtualized tables, infinite scroll, and scroll restoration without flaky viewport assertions that explores these problems.

The principle is applicable far beyond virtualized tables.

Test the behavior the user depends on, not the incidental details of how the browser renders it.

Playwright's own locator documentation recommends locating elements using roles, labels, text, and other meaningful attributes.

That's generally a better starting point than brittle CSS paths or positional selectors.

But even good locators can't fix a test that makes the wrong assumptions about the application.

Another area that gets underestimated is infrastructure.

It's easy to get a few Playwright tests running locally.

Running hundreds or thousands of them reliably across browsers, operating systems, and CI environments is a different problem.

These don't sound like test automation problems.

But they directly affect how much time engineers spend maintaining the test suite.

Consider a test that passes locally and fails on a remote browser.

Without useful debugging artifacts, someone may need to rerun the test, capture logs, reproduce the environment, and investigate what happened.

That can become expensive when failures occur frequently.

Both BrowserStack and Sauce Labs support Playwright execution in cloud environments. Their official documentation describes their respective integrations and execution models.

The interesting question is which one creates less operational work for your team.

This BrowserStack vs Sauce Labs comparison for Playwright teams evaluates them through setup friction, debugging artifacts, CI integration, and maintenance overhead.

I think that's a more useful approach than comparing marketing feature lists.

If I were evaluating either service, I'd take an existing test suite and measure four things:

I'd also make sure the comparison uses equivalent browser versions, test data, concurrency, and application builds.

Otherwise, you're not comparing infrastructure fairly.

And I'd avoid concluding that either provider is universally better without running the workload that actually matters to your organization.

The cheapest browser cloud on paper can become expensive if every unexplained failure consumes an engineer's afternoon.

Mobile testing introduces another layer of complexity.

You're no longer dealing with just browsers.

You have device models, operating system versions, permissions, network conditions, system dialogs, orientation changes, and interactions between native components and embedded web content.

Tools such as Appium provide an automation foundation for mobile applications.

But executing an automated mobile test and determining whether the application is ready for release aren't the same thing.

Imagine a banking application where a confirmation screen appears after a transaction.

The automated test verifies that the expected text is present.

But on a particular device, the confirmation button is partially hidden behind the system navigation area.

Or the application displays a technically correct message that's so poorly formatted that users misunderstand it.

An assertion checking for text won't necessarily detect either problem.

And even when AI identifies a possible visual defect, someone needs to decide whether it's a genuine issue, an acceptable variation, or a false positive.

This is why human review still has a place in mobile QA.

A recent evaluation of mobile testing services combining AI, human review, and traceable evidence discusses services such as Applause, Testlio, Global App Testing, and HeadSpin.

The useful distinction is between finding a potential problem and producing evidence that allows someone to act on it.

For an actionable mobile defect report, I'd want:

Depending on the issue, I'd also want the network conditions, permissions, and app state captured.

Without that context, a developer might spend more time reproducing the defect than fixing it.

AI can help identify suspicious behavior and organize evidence.

But for ambiguous failures, a person still needs to make a judgment.

That doesn't make the automation less valuable.

It makes the overall testing process more reliable.

Let's talk about the inevitable Cypress vs Playwright discussion.

And both can become expensive to maintain if your test architecture is poorly designed.

Cypress provides an interactive runner with command logs and snapshots that let developers inspect what happened during execution.

Playwright provides tracing capabilities that can capture actions, DOM snapshots, network activity, and console output.

You can see these capabilities directly in the official Cypress documentation and Playwright Trace Viewer documentation.

The tradeoffs become more interesting when you're testing an AI-powered application.

Imagine a customer support dashboard with an embedded AI assistant.

The assistant generates responses asynchronously. Sometimes it calls multiple backend services. Sometimes its responses change because of updated prompts or retrieved information.

You need to test whether the interface behaves correctly without assuming that every generated sentence will be identical.

You might also need to test iframes, authentication flows, multiple browser contexts, or network failures.

At that point, debugging ergonomics and framework capabilities become important.

This Cypress vs Playwright comparison for AI-powered web applications examines those tradeoffs, including network control, iframe handling, execution parallelism, and debugging.

My preference would be to evaluate both using the same application scenarios rather than treating either framework as a default winner.

And I'd separate two questions that often get mixed together.

How quickly can a developer create the initial test?

How much work will it take to keep that test useful over the next year?

But if the tests require substantial maintenance every time the application changes, the initial convenience may not matter much.

I don't think a vendor demonstration is enough to evaluate a testing platform.

Most demonstrations use applications and test cases selected to make the product look good.

That doesn't tell you how the product will behave when your application changes unexpectedly.

I'd rather run a small, controlled proof of concept.

Choose scenarios that expose different types of difficulty.

These are proposed experiments, not published benchmark results.

The goal is to see what happens when the tool encounters something less predictable than a straightforward login flow.

Before running the tests, document the expected behavior.

For the ambiguous button scenario, identify which button is correct.

For the virtualized table, define which records should appear and what should happen when they're selected.

For the mobile scenario, specify what constitutes a real defect.

This matters because you can't evaluate whether an AI system made the right decision unless you already know what the right decision was.

A single successful execution doesn't prove reliability.

I'd repeat each scenario across the relevant browser or device configurations, using controlled application builds and test data.

That fourth category, incorrect passes, is particularly important.

A flaky test is annoying because it creates additional work.

A test that passes while checking the wrong behavior is dangerous because it creates false confidence.

For each failure or automated repair, I'd use a simple scoring system.

The tool reports success or failure without enough context to determine what happened.

The tool provides enough artifacts to understand the outcome, such as steps, screenshots, logs, or trace data.

A reviewer can reconstruct the behavior, inspect automated changes, identify the environment, and approve or reject the result.

This isn't an industry-standard benchmark. It's a proposed rubric for comparing tools consistently.

I'd use it alongside execution accuracy and maintenance effort, rather than replacing those measurements.

A tool can provide excellent debugging evidence and still execute the wrong action.

Here's the part I think deserves more attention.

When companies evaluate test automation, they often concentrate on licensing costs and test creation speed.

But those are only part of the equation.

Total cost of test automation = Test creation + Maintenance + Infrastructure + Failure investigation + Human review

You can make that model more detailed by adding training, integrations, and migration costs.

But even this simplified version captures an important distinction.

Suppose Tool A generates tests almost instantly but regularly requires engineers to investigate ambiguous failures.

Tool B takes longer to configure but produces more predictable executions and better debugging artifacts.

You can't answer that question from the subscription price alone.

You need data from your actual application.

And that data should include the cost of investigating failures, not just the number of tests created.

This is especially relevant now that AI is making test generation so accessible.

If creating a test becomes nearly effortless, teams will naturally create more tests.

But unless those tests are reliable, understandable, and maintainable, we're simply increasing the number of things that can break.

If I were responsible for improving an existing automation program, I wouldn't start with a goal of doubling the number of tests.

First, I'd make sure every critical test has meaningful assertions. A successful click isn't proof that the intended business process worked.

Second, I'd improve failure evidence. When something breaks, the responsible engineer should have enough context to investigate it without immediately rerunning the test.

Third, I'd put controls around AI-generated modifications. Any system that automatically changes tests needs a way to explain those changes, preserve the original intent, and allow review when the decision is ambiguous.

Only after those foundations were in place would I aggressively expand automated coverage.

Because the goal isn't to have the largest possible test suite.

It's to have a test suite that consistently tells you something useful about your application.

AI is going to make automated testing more accessible.

It means smaller teams can automate workflows that previously required dedicated automation engineers.

But I don't think the industry should confuse easier test creation with better software quality.

Are the tests checking the correct behavior?

And can the team maintain the tests without spending an unreasonable amount of time on them?

A testing platform that helps answer those questions is valuable.

One that simply makes everything look green is considerably less useful.

I'd rather have 200 automated tests I can trust than 2,000 tests that I don't fully understand.

And I suspect that distinction is going to become more important as AI-generated tests become the norm.

I approach this topic from the perspective of working on test automation software, where the operational side of testing matters as much as test creation. That perspective can introduce a preference for maintainability and traceability, so I've tried to make the evaluation criteria explicit.

This is a technical analysis and proposed evaluation framework, not a new hands-on benchmark of the vendors mentioned. The comparisons linked throughout the article provide additional perspectives; official Playwright, Cypress, Appium, BrowserStack, and Sauce Labs documentation should be used to confirm specific supported capabilities and configurations.

What's been more expensive for your team: writing automated tests or maintaining them? I'd be interested to hear how that has changed since you started using AI.