AI-Generated Tests Can Agree With the Bug

An AI coding agent implements a feature, generates tests, runs them, and opens a pull request with a green build.

That looks like independent verification. It often is not.

If the implementation and the tests came from the same prompt, repository context, and interpretation of the requirement, they can repeat the same mistake. The code is wrong. The tests confidently prove that the wrong behavior works exactly as implemented.

The failure is not that AI cannot write tests. The failure is treating correlated output as independent evidence.

Apply a 20% discount to the order subtotal, then add tax to the discounted amount.

subtotal = 100.00 discount = 20.00 taxable amount = 80.00 tax = 8.00 final total = 88.00 Enter fullscreen mode Exit fullscreen mode An agent misreads the requirement and applies the discount after tax:

export function calculateTotal ( subtotal, taxRate, discountRate ) { const tax = subtotal * taxRate; const discount = subtotal * discountRate; return subtotal + tax - discount; } Enter fullscreen mode Exit fullscreen mode For the simple values above, both formulas happen to return 88.00. A generated test passes:

import { describe, expect, it } from " vitest "; import { calculateTotal } from "./billing.js "; describe ( " calculateTotal ", () => { it ( " applies a discount and tax ", () => { expect ( calculateTotal ( 100, 0.1, 0.2 )). toBe ( 88 ); }); }); Enter fullscreen mode Exit fullscreen mode The test looks reasonable. It even uses a concrete expected value.

But it does not distinguish the intended formula from the incorrect one.

Now introduce a fixed discount, tax exemptions, jurisdiction-specific rounding, or items with different tax classes. The shared assumption becomes a production billing bug.

Code coverage answers questions such as:

A generated test suite can reach 100% line coverage while asserting the implementation's current behavior back at itself.

const result = calculateTotal ( order ); expect ( result. subtotal ). toBe ( order. subtotal ); expect ( result. tax ). toBe ( result. taxableAmount * order. taxRate ); expect ( result. total ). toBe ( result. subtotal + result. tax - result. discount ); Enter fullscreen mode Exit fullscreen mode The test reconstructs the same formula used by the implementation. If the formula is wrong, both sides remain wrong together.

A test needs an oracle: an independent reason to believe the expected result is correct.

When one agent writes the code and tests in the same session, both artifacts may inherit:

Regenerating the test in the same conversation may not add meaningful independence. The model still sees its earlier reasoning and implementation.

Even using a second model is not automatically enough. If both models receive the same ambiguous prompt and derive expectations from the same implementation, they can converge on the same attractive mistake.

The goal is not merely a different generator. The goal is a different source of truth.

expect ( formatPrice ( input )). toBe ( formatPrice ( input )); Enter fullscreen mode Exit fullscreen mode This extreme example is obvious, but indirect versions are common. A test may call the same helper, parser, mapper, or formula as the production path.

expect ( calculateInvoice ( order )). toMatchSnapshot (); Enter fullscreen mode Exit fullscreen mode The first snapshot records whatever the implementation produced. Approving it proves that someone accepted the output, not that the output matches the requirement.

Snapshots are useful for detecting change. They are weak at establishing initial correctness when nobody verifies their meaning.

expect ( cache. internalEntries. size ). toBe ( 1 ); Enter fullscreen mode Exit fullscreen mode The test confirms structure rather than observable behavior. It may pass while users receive stale data, and fail during a safe refactor.

If an issue contains one example, an agent will often reproduce it as one happy-path test. That example may not distinguish competing interpretations.

paymentGateway. charge = vi. fn (). mockResolvedValue ({ status: " paid " }); Enter fullscreen mode Exit fullscreen mode The test proves behavior for a dependency that always succeeds instantly. It says nothing about timeouts, duplicated responses, uncertain outcomes, or retries after the remote side processed the charge.

Examples verify selected points. Invariants describe properties that must hold across many inputs.

For a discount calculation, useful invariants might include:

Property-based tests can exercise these rules across generated inputs:

import fc from " fast-check "; import { expect, it } from " vitest "; import { calculateTotal } from "./billing.js "; it ( " never increases the total when the discount increases ", () => { fc. assert ( fc. property ( fc. double ({ min: 0, max: 100 _000, noNaN: true }), fc. double ({ min: 0, max: 0.3, noNaN: true }), fc. double ({ min: 0, max: 1, noNaN: true }), fc. double ({ min: 0, max: 1, noNaN: true }), ( subtotal, taxRate, firstDiscount, secondDiscount ) => { const lower = Math. min ( firstDiscount, secondDiscount ); const higher = Math. max ( firstDiscount, secondDiscount ); const totalWithLowerDiscount = calculateTotal ( subtotal, taxRate, lower ); const totalWithHigherDiscount = calculateTotal ( subtotal, taxRate, higher ); expect ( totalWithHigherDiscount ). toBeLessThanOrEqual ( totalWithLowerDiscount ); } ) ); }); Enter fullscreen mode Exit fullscreen mode This still does not prove every domain rule. It does make it harder for one convenient example to hide a broken relationship.

The strongest acceptance tests should come from artifacts that existed independently of the implementation:

For the billing example, create a decision table before asking an agent to implement it:

Case: percentage discount before tax Subtotal: 125.00 Discount: 20% Tax rate: 8% Expected discount: 25.00 Expected taxable amount: 100.00 Expected tax: 8.00 Expected total: 108.00 Enter fullscreen mode Exit fullscreen mode Then convert that independently reviewed fixture into a test:

it ( " taxes the amount after the percentage discount ", () => { expect ( calculateTotal ( 125, 0.08, 0.2 )). toBe ( 108 ); }); Enter fullscreen mode Exit fullscreen mode The unusual values are deliberate. They distinguish formulas that coincidentally agree for simpler inputs.

Mutation testing changes the code in small ways and checks whether the tests fail.

For example, a mutation tool might replace:

const taxableAmount = subtotal - discount; Enter fullscreen mode Exit fullscreen mode with:

const taxableAmount = subtotal + discount; Enter fullscreen mode Exit fullscreen mode or change:

if ( total >= freeShippingThreshold ) Enter fullscreen mode Exit fullscreen mode into:

if ( total > freeShippingThreshold ) Enter fullscreen mode Exit fullscreen mode If the test suite stays green, it did not protect that behavior.

A mutation score is not a perfect quality metric. Equivalent mutations and costly test runs are real constraints. But targeted mutation testing is valuable for authorization, billing, retries, idempotency, limits, and other logic where a plausible one-character error matters.

Meta reported a related change-aware approach in its 2026 work on Just-in-Time catching tests. Instead of only generating tests that pass on the proposed code, catching tests are designed to surface regressions before a change lands. In an evaluation of 22,126 generated tests, change-aware methods improved candidate catch generation, and the reported process found confirmed serious failures. The important idea is practical: generated tests should actively try to falsify the change, not merely document it.

Differential testing runs the same inputs through two independent implementations and compares the outputs.

This is useful during rewrites, migrations, and optimizations:

for ( const order of generatedOrders ) { const legacyResult = legacyCalculator ( order ); const newResult = newCalculator ( order ); expect ( newResult ). toEqual ( legacyResult ); } Enter fullscreen mode Exit fullscreen mode The reference implementation does not need to be fast or elegant. It needs to be trusted for the behavior under comparison.

Be careful: if the legacy behavior contains known bugs, encode those exceptions explicitly. Otherwise, differential testing can preserve defects just as snapshots do.

A safer AI-assisted workflow creates friction between implementation and validation.

Give the coding agent the contract, but do not let its implementation become the source of expected values.

Ask it to find counterexamples, not to improve coverage.

Act as an adversarial test engineer. Given the requirement and code diff: 1. List plausible incorrect interpretations. 2. Create inputs that distinguish them. 3. Identify boundary conditions and invariants. 4. Write tests that would fail for each plausible wrong implementation. 5. Do not derive expected values from the implementation under test. Enter fullscreen mode Exit fullscreen mode Step 4: Run targeted mutation testing Focus on critical decisions rather than mutating the entire repository on every commit.

Reviewers should spend less time asking whether the test is readable and more time asking:

A green agent-generated pull request still needs repository protections, required checks, and human review. GitHub explicitly requires human review before merging draft pull requests produced by its coding agent and layers security scanning and branch restrictions around generated changes.

That is not ceremony. It is recognition that generated code plus generated tests is evidence, not authority.

Suppose an agent adds this rate limiter:

export function isAllowed ( requestCount, limit ) { return requestCount <= limit; } Enter fullscreen mode Exit fullscreen mode It also generates:

it ( " allows requests within the limit ", () => { expect ( isAllowed ( 3, 5 )). toBe ( true ); }); it ( " blocks requests above the limit ", () => { expect ( isAllowed ( 6, 5 )). toBe ( false ); }); Enter fullscreen mode Exit fullscreen mode Both tests pass. Coverage may be complete.

But the critical question is whether the fifth request is allowed or whether limit represents the number of requests already consumed. The tests avoid the exact boundary where the interpretation matters.

it ( " defines behavior at the exact limit ", () => { expect ( isAllowed ( 5, 5 )). toBe ( false ); }); Enter fullscreen mode Exit fullscreen mode Whether false is correct depends on the contract. The key is that a human or independent specification must decide it. The implementation cannot decide its own requirement.

Before trusting AI-generated tests, verify that:

CTA: Pick one AI-generated test from your current project and trace every expected value back to its source. If the trail ends at the implementation itself, replace that assertion with an independent fixture, invariant, or boundary case.

AI-generated tests are useful. They can produce scaffolding, cover routine branches, explore edge cases, and shorten feedback loops.

But tests generated from the same context as the implementation are not automatically independent validation.

They may encode the same misunderstanding twice:

incorrect requirement interpretation ↓ incorrect implementation ↓ matching incorrect tests ↓ green build Enter fullscreen mode Exit fullscreen mode Do not measure confidence only by the number of generated tests or the coverage percentage.

A test suite is valuable when it can disagree with the implementation.

What is the most convincing green test suite you have seen that still allowed a production bug through?

If you found this article helpful, let's connect!