Mutation Testing with TypeScript: How to Verify Your Tests in the Age of AI Coding Agents

Your tests pass. But would they catch a bug?

Imagine asking an AI coding agent to implement a feature. A few minutes later, it has generated the implementation, written unit tests, and opened a pull request.

The CI pipeline is green. All tests pass. Code coverage looks good.

The implementation and its tests may share the same misunderstanding of the requirements. The tests might execute every line without checking the behavior that matters. And a test suite that passes today might not detect a regression tomorrow.

As AI makes code generation faster, we need to think more carefully about how we verify the code we produce.

In this tutorial, we'll explore one technique that can help: mutation testing.

We'll build a small TypeScript project, introduce a bug deliberately, install StrykerJS, and use mutation testing to discover weaknesses in our tests. Then we'll explore how the same technique can help us verify AI-generated pull requests.

By the end, you'll understand not just how mutation testing works, but why it can become an important part of a multi-layer verification strategy.

Imagine we're building an online shop. One of our business rules is that orders worth £100 or more qualify for free shipping.

Let's implement that rule in TypeScript.

export function calculateShipping ( orderTotal: number, ): number { return orderTotal >= 100? 0: 10; } Enter fullscreen mode Exit fullscreen mode The function is straightforward:

import { describe, expect, it } from ' vitest '; import { calculateShipping } from './shipping '; describe ( ' calculateShipping ', () => { it ( ' charges shipping for small orders ', () => { expect ( calculateShipping ( 50 )). toBe ( 10 ); }); it ( ' offers free shipping for large orders ', () => { expect ( calculateShipping ( 150 )). toBe ( 0 ); }); }); Enter fullscreen mode Exit fullscreen mode Run the tests:

npx vitest run Enter fullscreen mode Exit fullscreen mode Both tests pass.

Great! We have an implementation, some unit tests, and a green test suite.

But let's ask a slightly different question.

What would happen if somebody accidentally changed >= to >?

export function calculateShipping ( orderTotal: number, ): number { return orderTotal > 100? 0: 10; } Enter fullscreen mode Exit fullscreen mode Now an order worth exactly £100 is charged shipping, even though the requirement says it should qualify for free shipping.

Why? Because we tested orders worth £50 and £150, but we never tested the boundary at £100.

Our tests execute the function, but they don't distinguish the correct implementation from this incorrect one.

This is the problem mutation testing helps us investigate.

Mutation testing is a technique for evaluating how effectively your tests detect certain defects.

Instead of simply running your tests against your implementation, a mutation-testing tool deliberately makes small changes to your production code and runs the tests against each modified version.

These changes are called mutations. Each modified version is called a mutant.

For example, a mutation-testing tool might make changes such as:

The tool then observes whether your tests detect the change.

There are two particularly important outcomes.

Killed mutant: At least one test fails because of the mutation. Your tests detected the change.

Survived mutant: The tests still pass. Your tests didn't detect the change.

Let's apply this to our shipping example.

return orderTotal >= 100? 0: 10; Enter fullscreen mode Exit fullscreen mode A mutant changes it to:

return orderTotal > 100? 0: 10; Enter fullscreen mode Exit fullscreen mode Our existing tests still pass, so the mutant survives.

That's a useful finding. It tells us that the test suite doesn't distinguish these two implementations.

Notice what we're evaluating: we're not simply asking whether the implementation works for our current examples. We're asking whether the tests can detect selected changes that might introduce defects.

This is a different question from ordinary unit testing and code coverage.

You may already measure code coverage in your projects.

Coverage tells you which parts of your code were executed during testing. It's useful, but it doesn't necessarily tell you whether your assertions are strong enough.

it ( ' calculates shipping ', () => { calculateShipping ( 100 ); }); Enter fullscreen mode Exit fullscreen mode The function executes. Depending on the coverage measurement, the test may contribute to branch or line coverage.

But the test doesn't assert the expected result.

The implementation could return £10 instead of £0, and the test would still pass.

it ( ' offers free shipping at exactly £100 ', () => { expect ( calculateShipping ( 100 )). toBe ( 0 ); }); Enter fullscreen mode Exit fullscreen mode This test checks the actual business rule.

It also kills our mutant because the mutated implementation returns £10.

These techniques complement each other. Mutation testing doesn't replace coverage, and a mutation score doesn't prove correctness. Together, however, they can provide more useful evidence about the effectiveness of a test suite.

We'll assume you already have Node.js and npm installed.

If you're following along from scratch, create a project:

mkdir mutation-testing-demo cd mutation-testing-demo npm init -y Enter fullscreen mode Exit fullscreen mode Install TypeScript and Vitest:

npm install --save-dev typescript vitest Enter fullscreen mode Exit fullscreen mode Create the src directory and add the two files from the previous sections.

npx vitest run Enter fullscreen mode Exit fullscreen mode Make sure your tests pass before continuing.

npx stryker init Enter fullscreen mode Exit fullscreen mode Follow the prompts to configure Stryker for your project. Select TypeScript and the Vitest test runner where applicable.

Stryker's setup options can vary with the version and project configuration, so follow the current prompts rather than assuming every project requires an identical configuration.

Official documentation: StrykerJS — Getting Started.

Once initialization is complete, inspect the generated configuration and make sure the mutation scope includes our production file:

src/shipping.ts Enter fullscreen mode Exit fullscreen mode Our test file, src/shipping.test.ts, should not be treated as production code to mutate.

You can configure the mutation scope explicitly if necessary. For example, in a TypeScript Stryker configuration, the relevant pattern might be:

mutate: [ ' src/**/*.ts ', '!src/**/*.test.ts ' ], Enter fullscreen mode Exit fullscreen mode Keep the configuration generated by your installed version as the starting point, and adapt it to your repository's conventions.

npx stryker run Enter fullscreen mode Exit fullscreen mode Stryker will run your tests, generate mutations in the configured production code, and evaluate the results.

The first run may take longer than your ordinary unit tests because Stryker needs to evaluate multiple mutants.

Once it finishes, inspect the terminal output and generated HTML report.

Look for surviving mutants associated with the comparison in calculateShipping.

If the comparison changes from >= to > and all our tests still pass, we've found the weakness we deliberately introduced.

Congratulations: you've just used mutation testing to discover a gap in your test suite.

A mutation report contains more than just a score.

The most useful outcomes to understand are:

A test detected the mutation and failed. This is generally what we want for a meaningful behavioral change.

The tests passed despite the mutation. Investigate whether this reveals missing test coverage or an assertion that doesn't adequately verify the requirement.

The mutated code wasn't exercised by the relevant tests. You may need to investigate whether the code is tested at all.

The tool couldn't obtain a normal result within its configured time limit. This doesn't necessarily mean your tests detected the mutation.

The mutation doesn't change observable behavior for the relevant inputs. No meaningful test can distinguish it from the original implementation.

Not every surviving mutant is a defect, and not every mutant needs to be killed. Some mutations are equivalent, some aren't meaningful for your requirements, and some may reveal behavior you don't actually need to constrain.

The purpose of the report is to help you make better decisions about your tests.

Stryker calculates a mutation score based on the results of mutation testing.

Mutation score = killed mutants / valid mutants × 100%

For example, suppose a run produces 100 valid mutants and your tests kill 80 of them. The mutation score is 80%.

That can be a useful signal, but be careful about treating it as a universal quality metric.

Two projects can have the same mutation score while having very different levels of risk. The score depends on the mutations generated, the code being analyzed, and which results are excluded.

A surviving mutation in payment authorization may matter much more than a surviving mutation in an unimportant formatting helper.

Focus on understanding meaningful surviving mutants, not maximizing a percentage at any cost.

Our business rule is explicit: orders worth £100 or more qualify for free shipping.

it ( ' offers free shipping at exactly £100 ', () => { expect ( calculateShipping ( 100 )). toBe ( 0 ); }); Enter fullscreen mode Exit fullscreen mode Run the ordinary test suite:

npx vitest run Enter fullscreen mode Exit fullscreen mode The test passes against the correct implementation.

npx stryker run Enter fullscreen mode Exit fullscreen mode The mutation that changes >= to > should now be killed by the boundary test.

We've improved the test suite because it now detects a meaningful defect.

Notice that we didn't add a test just to improve a score. We added one because the requirement explicitly defines what should happen at the boundary.

A mutation-testing report is a source of questions, not an instruction to write more assertions blindly.

This approach fits naturally into Test-Driven Development.

TDD helps us implement behavior from requirements. Mutation testing challenges whether our tests can detect selected deviations from that behavior.

So far, we've run mutation testing manually.

The next step is to integrate it into CI.

Imagine an AI coding agent creates a PR that modifies calculateShipping and adds several unit tests.

Now Stryker runs against the configured production code. It discovers that the tests don't detect a meaningful mutation in the shipping calculation.

The PR report can expose that weakness before the change is merged.

This is particularly useful for AI-generated code because the implementation and its tests may have been produced by the same agent, based on the same interpretation of the requirements.

The tests passing is evidence that the implementation satisfies the assertions the agent wrote. It isn't independent evidence that those assertions capture every important requirement.

Mutation testing adds another challenge: it deliberately changes the implementation and checks whether the test suite notices.

Stage 1: Report findings without blocking merges.

Run mutation testing on a small set of important modules. Learn how long it takes, inspect surviving mutants, and identify noisy or irrelevant results.

Introduce stronger requirements for areas where correctness is particularly important, such as permissions, pricing, validation, or state transitions.

Use scheduled runs and historical reports to identify weaknesses across the broader codebase.

For larger projects, Stryker's incremental mode can reuse eligible previous results to reduce the work needed on subsequent runs. See the incremental testing documentation.

One important distinction: mutating changed production files is not the same as testing only the new tests in a PR. You want relevant existing tests to participate in verifying the changed behavior as well.

You should also avoid making a repository-wide mutation score an immediate merge requirement. A small PR shouldn't necessarily be responsible for fixing every weakness in a legacy test suite.

We can take the workflow a step further.

Instead of using mutation testing only as a CI gate, we can feed its findings back to the AI agent.

For our shipping example, the agent might receive a report showing that changing >= to > doesn't cause any tests to fail.

The agent can then inspect the business rule and recognize that the boundary at £100 is important.

It adds the boundary test, reruns the suite, and verifies that the mutation is now killed.

This creates a feedback loop in which the agent can improve not just the implementation, but also the evidence we have about that implementation.

Don't let the agent blindly optimize the mutation score.

An agent could make a mutant fail by adding an assertion that encodes an incorrect assumption. It could also change production code simply to make the existing tests pass.

The correct behavior must come from explicit requirements, domain rules, or other trustworthy sources—not from the mutation report alone.

The agent can propose a fix, but the final verification should run independently in CI against the revised commit.

This brings us to the broader reason mutation testing matters.

No single verification technique catches every defect.

Type checking can detect certain classes of errors, but it doesn't establish that a business rule is correct.

Unit tests can verify specified behavior, but they only check the cases and assertions we've written.

Mutation testing can challenge the effectiveness of those tests, but it can't guarantee that the original requirements were understood correctly.

Integration tests can expose problems between components, but they won't necessarily identify every defect in an individual component.

Code review can identify design and requirement problems, but human reviewers also have limited time and attention.

This is the idea behind a Swiss cheese model of verification: arrange multiple imperfect defenses so that a defect that passes through one layer has other opportunities to be detected.

Catch incompatible interfaces, invalid types, and selected classes of programming mistakes.

Check specified behavior, interactions, and component integration.

Challenge whether tests detect selected changes to the implementation.

Layer 4: Contract and architecture checks

Verify boundaries, invariants, and integration assumptions.

Layer 5: Independent review and production feedback

Identify misunderstood requirements, broader design risks, and failures that automated checks miss.

The layers are complementary. They aren't completely independent: mutation testing relies on the test suite, and both can be based on the same misunderstood requirement.

But combining different forms of verification can provide stronger evidence than relying on any one of them.

This connects directly to the principles discussed in Revisiting the Toyota Production System in the Age of Coding Agents. In particular, the idea of building quality into the production process is relevant here: detect abnormalities as early as possible, rather than allowing them to move downstream.

Mutation testing can be one of the automated checks that helps us detect weaknesses before a pull request is merged.

AI changes the economics of software development.

An agent can produce a substantial implementation and its accompanying tests much faster than a human might write them manually.

But generating code is not the same as establishing that the code is correct.

If the volume of changes increases dramatically, manually reviewing every implementation and every test may become increasingly difficult.

We need verification mechanisms that scale with the rate of change.

This is the connection to When Code Gets Cheap, Verification Becomes Expensive.

Mutation testing is not a complete solution to that problem. It has a computational cost, and it only evaluates the mutations it generates. It cannot establish that the implementation satisfies every requirement or that every important defect is represented by a mutation.

What it can do is make one important question cheaper to investigate:

Can our tests detect plausible defects in the code we're producing?

And because the process can be automated, we can apply that challenge repeatedly as code changes.

The objective isn't to maximize code generation or mutation scores in isolation. It's to increase our ability to produce and verify valuable changes without letting the volume of generated code overwhelm our verification capacity.

Mutation testing is a relatively simple idea with important consequences.

We deliberately modify production code, run our tests, and observe whether those tests detect the changes. When meaningful mutants survive, we investigate whether our tests are missing important behavioral checks.

With TypeScript and StrykerJS, it's possible to introduce this technique into an existing project without replacing your test runner or abandoning your current TDD workflow.

And when AI agents generate code and tests, mutation testing provides a useful additional challenge: rather than accepting a green test suite at face value, we can investigate whether it detects selected mistakes.

It doesn't prove that our code is correct. No single testing technique does.

But that's precisely why it belongs in a multi-layer verification strategy.

When code generation scales, verification must scale with it. Mutation testing helps us challenge one of the most important defenses we already have: our tests.