Why Green Tests Are Not Enough for AI Coding Agents#

Over the last few months, I’ve been spending more time working with AI coding agents. They are fast, useful, and in many cases they can fix small bugs or implement focused changes much faster than doing everything manually.

But while using them, I kept coming back to the same problem: how do I know that the evidence around the fix is actually meaningful?

It is easy to ask an agent to change some code, run a test, and return a green result. The problem is that a green test only tells me that the final version passed. It does not necessarily tell me whether the test was capable of detecting the original bug.

That difference sounds small, but it matters a lot.

Imagine an AI agent fixes a bug and adds a regression test. The final test run passes, so the change looks good. Now imagine that we take the exact same test and run it against the broken version of the code, and it passes there too.

In that case, the test is green, but it did not really prove the bug was fixed.

That is the kind of problem I wanted to handle more systematically.

Broken baseline + regression probe = FAIL

Fixed result + same regression probe = PASS

If the same probe fails on the broken version and passes on the fixed version, the evidence is much stronger.

If it passes on both, then the test may not actually be testing the bug we care about.

This idea became the basis for a small tool I’ve been building called AI Coding Agent Reliability Kit.

It is a Linux-first local workflow for reviewing AI-assisted bug fixes with stronger mechanical evidence.

The goal is not to replace code review, security review, or engineering judgment. The goal is to make the evidence produced around an AI-generated fix easier to inspect and harder to misinterpret.

When I started working on this, I did not want another wrapper that simply runs a command and prints success or failure.

I wanted a workflow that would answer several questions around the change.

Did the final verification command really pass?

Does the regression probe fail on the broken baseline?

Can the result be reproduced in a clean replay?

Can another person review the evidence without having to reconstruct everything manually?

These are basic questions, but once coding agents start making larger or more frequent changes, they become important.

One of the things the kit does is define the task and the allowed paths before the coding work begins.

This is useful because an agent can sometimes fix the requested problem while also changing unrelated files. The final result may still work, but the change surface becomes larger than necessary.

For a focused bug fix, I usually want a clear boundary.

For example, if a task only needs a change in one source file and one regression test, then a modification to an unrelated README, configuration file, or another package should be visible immediately.

The kit tracks the declared scope and checks whether the final change stayed inside it.

If something changed outside the allowed boundary, that is reported instead of silently accepting the result.

A normal workflow often ends after something like this:

pytest tests/test_bug.py Enter fullscreen mode Exit fullscreen mode The command exits successfully, so the task is considered done.

That is useful, but for a regression fix I want one more piece of information.

I want to know what happens when the same probe is run against the known broken baseline.

A good regression probe should normally distinguish the two states.

Broken version: FAIL Fixed version: PASS Enter fullscreen mode Exit fullscreen mode That is a much stronger signal than:

Broken version: PASS Fixed version: PASS Enter fullscreen mode Exit fullscreen mode The second result does not necessarily mean the code is wrong. It means the evidence is weak.

The workflow should not pretend that a test proves something it does not prove.

This was one of the main reasons I built the kit.

A test can look perfectly valid and still fail to discriminate between the broken and fixed versions.

For example, imagine a function that is supposed to multiply two numbers but currently adds them.

def calculate ( a, b ): return a + b Enter fullscreen mode Exit fullscreen mode The fix changes it to:

def calculate ( a, b ): return a * b Enter fullscreen mode Exit fullscreen mode Now imagine the test only checks this:

def test_calculate_exists (): assert callable ( calculate ) Enter fullscreen mode Exit fullscreen mode The test is green after the fix, but it was also green before the fix.

It proves that the function exists. It does not prove that the bug was fixed.

This type of result should not be treated the same as a real regression test.

The kit detects this condition and reports the evidence as non-discriminating instead of treating the final green result as sufficient.

Another case I wanted to handle is when the actual code change is correct, but something outside the allowed scope is modified.

This can happen accidentally, or because an automated workflow touches another file during execution.

src/formatter.py tests/test_formatter.py Enter fullscreen mode Exit fullscreen mode The source fix and test may both be correct, but if the final change also modifies:

README.md Enter fullscreen mode Exit fullscreen mode then the workflow should report that separately.

The important point here is that correctness and scope are different things.

A correct source fix does not automatically make an unrelated change acceptable.

The kit keeps those concerns separate so the reviewer can see exactly what happened.

Another feature I wanted was the ability to reproduce a result from a clean state when needed.

Sometimes a change passes because of something left behind in the working environment: generated files, caches, local state, or another uncommitted change.

A clean replay helps answer a simple question:

Can the same result be reproduced without depending on the current working directory state?

Not every task needs a full clean replay, but for changes where reproducibility matters, it gives the reviewer another useful piece of evidence.

One problem with automation is that it can generate too much output.

Raw logs are useful when investigating something, but they are not always the best way to review a change.

The kit produces a structured review summary that makes the main results easier to inspect.

Scope: PASS Final verification: PASS Broken baseline probe: FAIL Fixed probe: PASS Differential evidence: PASS Replay: REPRODUCIBLE Enter fullscreen mode Exit fullscreen mode If something fails, the summary should show that clearly instead of hiding it behind a successful final command.

The idea is to reduce the amount of reconstruction the reviewer needs to do.

The workflow can also export proof material and verify its integrity.

This is useful when the evidence needs to be moved, archived, or reviewed later.

Instead of only saying that a command passed at some point, the workflow can preserve the relevant result records and verify that the expected material has not changed.

This is not meant to turn the tool into a security certification system. It is simply a way to make the evidence around a run more inspectable and reproducible.

One thing I wanted to keep very clear is the meaning of the final status.

READY_FOR_REVIEW Enter fullscreen mode Exit fullscreen mode it does not mean:

The code is definitely correct. Enter fullscreen mode Exit fullscreen mode It does not mean:

The code is secure. Enter fullscreen mode Exit fullscreen mode It also does not mean:

This is automatically production-ready. Enter fullscreen mode Exit fullscreen mode It means that the configured mechanical evidence gates passed and the result is ready for a human reviewer to inspect.

I do not think an automated workflow should silently upgrade mechanical evidence into a semantic engineering judgment.

A human reviewer still needs to look at the actual change and decide whether it is correct for the system.

A human developer usually has some context in their head while making a change. They know what they touched, what they were trying to fix, and what assumptions they made.

With an AI coding agent, more of that work can happen automatically and quickly.

That speed is useful, but it also means it becomes easier to accept a result simply because the agent says the task is complete and the final test is green.

The faster the implementation becomes, the more useful it is to make the verification process explicit.

I do not see this as an argument against coding agents. I use them.

The point is that if agents are going to make more changes for us, then the evidence around those changes should become better as well.

The first version of the kit is intentionally simple in its environment assumptions.

It is designed for Linux and uses a local workflow with:

I wanted the first version to stay focused instead of claiming support for platforms I had not properly validated.

Windows and macOS are not part of the validated V1 environment.

That may change later, but I prefer to be explicit about what has actually been tested.

This tool does not try to prove that a program is correct.

It does not replace unit tests, integration tests, static analysis, security review, or a good human code review.

It also does not make a weak test strong automatically.

If the regression probe is poorly designed, the workflow can detect some types of weak evidence, such as a probe that passes on both the broken and fixed versions, but it cannot understand every semantic requirement of every project.

What it does is provide a stricter process around the evidence that already exists.

It is intentionally not a fully autonomous approval system.

def slugify ( text ): return text. strip (). lower (). replace ( " ", "" ) Enter fullscreen mode Exit fullscreen mode For this input:

Hello World Enter fullscreen mode Exit fullscreen mode the broken result is:

helloworld Enter fullscreen mode Exit fullscreen mode The required result is:

hello-world Enter fullscreen mode Exit fullscreen mode The fix might be:

def slugify ( text ): return " - ". join ( text. lower (). split ()) Enter fullscreen mode Exit fullscreen mode And the regression probe:

def test_slugify (): assert slugify ( " Hello World " ) == " hello-world " assert slugify ( " Two Words " ) == " two-words " Enter fullscreen mode Exit fullscreen mode Now the useful part is not only that this test passes after the fix.

Broken baseline: FAIL Fixed result: PASS Enter fullscreen mode Exit fullscreen mode That tells the reviewer that the probe actually detects the behavior being corrected.

The main reason is that I want to use coding agents more, not less.

But if I am going to delegate more implementation work to them, I want a better boundary between “the agent says it is done” and “there is enough evidence for me to review the result seriously.”

That is the gap this project is trying to cover.

It is still a first version, and I expect the workflow to evolve as I use it on more projects and find more edge cases.

For now, the focus is narrow: bounded changes, stronger regression evidence, reproducibility, and human review.

If that sounds useful for the way you work with coding agents, I published the first version here:

https://sovereign-system.itch.io/ai-coding-agent-reliability-kit