Using Cypress Cloud CLI and Test Replay for AI-Assisted CI Debugging

Signup Write New Story Using Cypress Cloud CLI and Test Replay for AI-Assisted CI Debugging by Raju Dandigam by Raju Dandigam | @ rajudandigam Staff Software Engineer at Navan Staff Software Engineer at Navan, Docker Captain, Cypress Ambassador, and creator of AgentInspect, writing about production AI agents, TypeS

Staff Software Engineer at Navan, Docker Captain, Cypress Ambassador, and creator of AgentInspect, writing about production AI agents, TypeS

A checkout test fails in CI with this message:

expected '[data-cy="cart-total"]' to have text '$45.00'

The screenshot shows $50.00. An AI assistant can invent several plausible explanations: the discount request failed, React rendered stale state, the selector found the wrong cart, or the assertion ran early.

The quality of AI-assisted debugging depends less on how confidently the assistant explains the screenshot and more on whether it can inspect the evidence that produced it.

Cypress now exposes two useful but distinct agent workflows:

I have written separately about giving an agent access to a live Cypress session. This workflow starts at the opposite end: the failure already happened, perhaps only in CI, and the recorded attempt is the evidence.

The Cypress Cloud CLI returns JSON for runs, specs, tests, commands, network requests, and console events. The local Test Replay viewer is available through cy-cloud replay view in CLI version 0.4.0 or newer.

The current documentation requires Node.js 22.21.0 or newer and recommends using the latest CLI. Install it, record the resolved version, and pin that exact version in repeatable CI automation after validation:

npm install --global @cypress/cloud@latest cy-cloud version cy-cloud status

For a failed run, the shortest useful path is:

cy-cloud run list \ --projectId <projectId> \ --branch main \ --status failed \ --limit 1 cy-cloud test list \ --projectId <projectId> \ --runNumber <runNumber> \ --status failed cy-cloud replay timeline \ --testId <testId> \ --commands \ --aroundFailure 5 \ --network \ --logs

Do not begin by dumping an entire organization's run history into an agent. Scope the query to one project, run, test, attempt, and timeline window.

Suppose the test failed on attempt one and passed on attempt two. A screenshot from the failed attempt tells us the symptom. Comparing the attempts can expose the first causal difference.

cy-cloud replay timeline \ --testId <testId> \ --attempt 1 \ --commands --network --logs \ > attempt-1.json cy-cloud replay timeline \ --testId <testId> \ --attempt 2 \ --commands --network --logs \ > attempt-2.json

The evidence does not yet prove whether the product or test is wrong, but it eliminates several guesses. The assertion can run before the successful response. The next review question is whether the UI should block until the response arrives, or whether the test failed to wait for an application event that defines completion.

The timeline gives structure. The local replay viewer lets you inspect the recorded DOM and styles around one command:

cy-cloud replay view open --testId <testId> cy-cloud replay view move \ --testId <testId> \ --attempt 1 \ --pin '<commandId>|before' cy-cloud replay view inspect \ --testId <testId> \ --selector '[data-cy="cart-total"]' cy-cloud replay view move \ --testId <testId> \ --attempt 1 \ --pin '<commandId>|after'

This is more diagnostic than asking an agent to infer causality from the final screenshot. It can see whether an earlier command changed an attribute, collapsed a container, or left the DOM stale.

replay view requires a Chromium-family browser and Cloud CLI 0.4.0 or newer. Because Cloud CLI is Beta, scripts should pin a compatible version and treat output schemas as versioned dependencies.

An unrestricted prompt such as “fix this test” encourages the assistant to optimize for a green run. It may add a hard wait, broaden a selector, remove an assertion, or mask a product regression.

Analyze attempt-1.json and attempt-2.json for test <testId>. 1. Identify the first material difference before the failed assertion. 2. Cite the relevant command, network, or console event IDs. 3. Classify the likely cause as product, test synchronization, test data, environment, or unknown. 4. Propose the smallest code change, but do not edit files. 5. Do not add fixed waits or weaken the business assertion. 6. List the local verification commands and one negative control.

The request separates observation, diagnosis, proposal, and execution. A human can reject the classification before any code changes.

Cloud CLI access follows the organizations, teams, projects, and roles available to the authenticated Cypress Cloud user. OAuth is recommended locally; a personal access token can be supplied through CYPRESS_CLOUD_TOKEN in CI. Stored credentials are owner-readable on the local machine.

Those are useful controls, but they do not mean every AI environment is automatically safe. If an agent can read arbitrary environment variables, home-directory files, or shell output, it may reach more than intended.

Use the ordinary least-privilege questions:

Cypress supports masking and redaction for Test Replay network data; configure those boundaries before making replay evidence broadly available.

After reviewing a proposed fix, run the affected spec locally. A useful sequence is:

The negative control matters. If changing the selector makes the test pass even when the discount is not applied, the “fix” destroyed the test.

Before blaming the test or product, rerun the candidate change in the same declared Node and browser environment.

An official cypress/browsers or cypress/included image can make that boundary repeatable, but only when you pin a full version tag or digest:

export CYPRESS_IMAGE='cypress/browsers@sha256:<approved-digest>' docker compose run --rm \ -e CYPRESS_SPEC='cypress/e2e/checkout.cy.ts' \ cypress

The Compose service should run the same repository revision and application fixture used in the investigation. Record docker image inspect "$CYPRESS_IMAGE", node --version, npx cypress version, and the selected browser with the evidence packet. A green result in a different image is not a reproduction; it is a new experiment.

Recorded Test Replay evidence describes one captured run. It does not prove that a failure is statistically representative or that the proposed patch is correct. A passing retry can differ because of data, timing, infrastructure, or an actual product race.

Cloud CLI is also not an autonomous remediation system. It is a read-oriented evidence interface. Code changes, approval, and verification remain responsibilities of your development workflow.

That turns the assistant from a confident guesser into a supervised participant in a debugging process whose claims can be reviewed

Building End-to-End Cypress Tests Across Browser, API, and Worker Boundaries