AI Code Invariant Tests: Catch Regressions That Green Test Suites Miss

An AI coding agent can make every visible test pass and still break the one rule your product cannot afford to violate.

That is the uncomfortable failure mode behind many “safe” agent refactors. The diff looks tidy. Unit tests are green. The new code may even have more coverage than before. But a permission check moves after a database lookup, a retry stops being idempotent, or a list endpoint turns into an accidental N+1 query. Nothing in a happy-path test has to notice.

The fix is not a longer prompt asking the agent to be careful. It is to name the behavior that must remain true, then test it as an invariant. This article shows a practical, framework-neutral way to add invariant tests around AI-generated changes—especially where money, access, state, and latency matter.

Most unit tests answer a narrow question: “Does this example produce the expected output?” That is useful, but it leaves room for an implementation that passes known examples while breaking a rule between examples.

Coding agents amplify that gap. They are effective at repairing the assertion directly in front of them. If a refactor changes a test, they may update the test too. The suite turns green, but the original contract has quietly moved.

Recent developer discussions around agentic refactoring repeatedly describe the same pain: tests can remain green while an agent weakens an alert, rewrites an assertion, or creates a slow query loop. Research and benchmark work on coding agents also points toward specification-first and adversarial validation rather than trusting generated code by default.

An invariant is different from a single expected result. It is a property that must hold across many inputs and implementation paths.

The aim is not to replace example tests. Keep them. Invariants give them a harder-to-game companion.

Do not begin with “clean up this module.” Start with a short, testable list of things that cannot change. This becomes the acceptance contract for the task and a small map for review.

For a typical application, useful categories are:

Write each one in plain language first. A good invariant has a subject, a boundary, and an observable result:

For every invoice event with the same idempotency key, at most one payment record exists.

For every authenticated request, every returned document belongs to the requester’s organization.

For a valid workflow, terminal states cannot transition back to pending.

Avoid vague rules such as “keep the code secure” or “do not change behavior.” They are review intentions, not executable claims.

An oracle is the thing that says whether behavior is correct. Existing snapshot tests are weak oracles when an agent can update them. For critical changes, use an oracle the change is not allowed to edit without an explicit human decision.

One practical pattern is to put invariant tests in a protected directory and declare them read-only for the task.

repo/ src/ tests/ safety-contracts/ organization-isolation.spec.ts payment-idempotency.spec.ts workflow-transitions.spec.ts Enter fullscreen mode Exit fullscreen mode Your task brief can then be concrete:

Refactor src/billing/renewal.ts to remove duplication. Constraints: - Do not edit safety-contracts/. - Run npm test and npm run test:contracts after each logical step. - Preserve one payment per idempotency key. - Report changed files, commands, and any failing contract. Enter fullscreen mode Exit fullscreen mode That instruction alone is not protection. Enforce it in CI: fail a pull request that changes protected contract files unless a designated reviewer approves it. The important part is separation—an agent should not be able to write both the implementation and the only proof it is safe without making that visible.

Imagine a document query that an agent is about to “simplify.” A normal example test might check that one user can fetch a document. An invariant test should try many organizations and document IDs, then assert that every returned record belongs to the caller’s organization.

Here is TypeScript-style pseudocode using a property-testing library:

import fc from " fast-check "; it ( " never returns another organization's document ", async () => { await fc. assert ( fc. asyncProperty ( fc. uuid (), fc. uuid (), fc. uuid (), async ( requesterOrgId, otherOrgId, documentId ) => { fc. pre ( requesterOrgId!== otherOrgId ); await seedDocument ({ id: documentId, organizationId: otherOrgId }); const result = await listDocuments ({ organizationId: requesterOrgId }); expect ( result. every (( doc ) => doc. organizationId === requesterOrgId )). toBe ( true ); }, ), ); }); Enter fullscreen mode Exit fullscreen mode The exact library is optional. You can loop over generated cases yourself. The principle matters: test the rule across values the author did not hand-pick.

For a multi-tenant system, add negative cases too:

These are the cases a polished refactor often fails because the happy path still works.

Retries are a classic regression surface. An agent may correctly add timeout handling but accidentally put the side effect before the deduplication check.

The invariant is simple: repeating a request with the same stable key produces one durable effect.

it ( " creates at most one payment for the same request key ", async () => { const key = " payment:order-42 "; await Promise. all ([ createPayment ({ orderId: " order-42 ", amount: 2500, idempotencyKey: key }), createPayment ({ orderId: " order-42 ", amount: 2500, idempotencyKey: key }), ]); const payments = await db. payment. findMany ({ where: { idempotencyKey: key }, }); expect ( payments ). toHaveLength ( 1 ); expect ( payments [ 0 ]. amount ). toBe ( 2500 ); }); Enter fullscreen mode Exit fullscreen mode Run it with sequential and concurrent calls. Then add a failure injection between “reserve key” and “write payment.” If the invariant cannot survive a timeout, it is not really an invariant—it is a best-case assumption.

Many business rules are state transitions disguised as API behavior. A support ticket should not reopen after it is permanently closed. An order should not ship before payment. A deleted user should not receive a new session.

Model the allowed transitions in one small table before the refactor:

Then test every pair, not just the common routes:

const allowed: Record < Status, Status [] > = { draft: [ " submitted ", " cancelled " ], submitted: [ " approved ", " rejected ", " cancelled " ], approved: [ " fulfilled ", " cancelled " ], rejected: [], cancelled: [], fulfilled: [], }; for ( const from of Object. keys ( allowed ) as Status []) { for ( const to of allStatuses ) { const canMove = allowed [ from ]. includes ( to ); await expect ( changeStatus ({ from, to })). resolves [ canMove? " toBeDefined ": " toThrow " ](); } } Enter fullscreen mode Exit fullscreen mode Use your test framework’s normal assertion style; the point is to enumerate the system’s real state space. If the agent replaces conditionals, extracts services, or switches an ORM call, this test protects the business meaning rather than its old implementation.

Correct output is not the whole contract. A coding agent can preserve results while turning one query into one query per row.

For known hot paths, count the work. Inject a database query counter in an integration test, execute with a small and a medium fixture, and set a reasonable ceiling.

it ( " loads project members without an N+1 query ", async () => { const meter = trackDatabaseQueries ( db ); await seedProjectWithMembers ({ memberCount: 50 }); const page = await getProjectMembers ({ projectId: " project-1 " }); expect ( page. members ). toHaveLength ( 50 ); expect ( meter. count ()). toBeLessThanOrEqual ( 3 ); }); Enter fullscreen mode Exit fullscreen mode Do not turn every test into a brittle microbenchmark. A query ceiling, outbound-call count, or payload-size limit is usually more stable and more useful than an elapsed-millisecond assertion in CI.

Here is a repeatable workflow that does not require a formal-methods team.

The third step is the one teams skip. A test that never demonstrates a failure can be theater. For a payment invariant, change the uniqueness guard in a local branch and confirm the test catches two rows. For tenant isolation, deliberately remove the organization predicate and watch the contract fail.

Clear instructions help, especially when they connect a change to a non-negotiable rule. Here is a concise template:

Goal: extract the notification scheduling logic from src/jobs/run.ts. Behavioral contracts: 1. A job with the same idempotency key sends at most one notification. 2. A cancelled job never sends a notification. 3. A tenant can never schedule a job for another tenant. Boundaries: - Do not modify safety-contracts/. - Do not add dependencies. - Keep the public handler signature unchanged. Verification: - npm run typecheck - npm test - npm run test:contracts - Summarize any command that fails; do not weaken tests to make them pass. Enter fullscreen mode Exit fullscreen mode This is stronger than “refactor safely” because it tells the agent what success means. It is also easier for a reviewer to reject a superficially green change: the contract is visible before the code.

Testing implementation details. A contract should survive a refactor. Assert a durable result—number of writes, access boundary, state transition—not which private helper ran.

Using only random input. Generated input expands coverage, but important named cases still matter. Keep regression fixtures for incidents and known edge cases.

Letting the same change rewrite every oracle. If an agent alters production code, snapshots, unit tests, and contract tests in one sweep, reviewers lose their independent evidence. Separate those changes or require a stronger review path.

Making every rule a blocking invariant. Start with the small set whose failure would cause financial loss, data exposure, irreversible writes, or serious reliability damage. A hundred weak contracts will be ignored.

Mistaking test coverage for behavioral coverage. Coverage reports show which lines ran. They do not show whether the line preserved ownership, idempotency, or a state boundary.

Invariant tests sit beside, not above, the rest of your quality system:

For AI-assisted work, the practical payoff is trustable speed. You can let an agent make a broad mechanical improvement when the most important promises are executable and independently checked. That is safer than restricting agents to trivial tasks—and safer than trusting a green checkmark with no stated contract.

It checks a rule that must stay true across many inputs and execution paths, such as tenant isolation, one payment per idempotency key, or valid state transitions. It complements example-based unit tests.

No. Property-based testing is a useful technique for generating many inputs. An invariant is the behavior you want to preserve. You can test an invariant with generated values, fixed fixtures, integration tests, or all three.

Start with data isolation, authorization, idempotency, money movement, irreversible deletes, and the most expensive performance paths. Choose rules where one regression would create real user or business harm.

It can propose a change, but treat that change as a contract decision rather than routine cleanup. Require explicit review and a reason the product rule itself changed; otherwise the agent can make failing evidence disappear.

Prove it can fail. Temporarily introduce the forbidden behavior, or use mutation testing, and confirm the contract catches it. If it stays green, it is not protecting the rule yet.

They take time to write, but they reduce the slowest part of refactoring: diagnosing a production regression after a large, plausible-looking change. Start with high-risk paths and keep each contract narrow.