Your AI Coding Agent Passed the Tests. Did the Tests Actually Run?

A coding agent finishes a task, runs pytest, and reports success.

But here's a question I've become increasingly interested in:

What exactly did that green result prove?

Because there's a difference between a test suite passing and the intended verification actually taking place.

And that distinction is becoming increasingly important as we delegate more software development to autonomous agents.

While investigating coding-agent verification, I ran a small controlled experiment against a repository after a clean test run.

I introduced five changes to see which ones the test suite would catch.

Four of the five left the suite passing.

The implementation failure turned the suite red.

But several changes affecting what the tests asserted or executed still produced successful exit codes.

That experiment wasn't a measurement of how frequently coding agents behave this way. These were deliberate changes designed to probe the verification boundary.

What interested me was something more fundamental.

A successful test command is not automatically evidence that the intended test obligations were satisfied.

Developers often protect their acceptance tests from modification.

But with pytest, the files in tests/ aren't the only things influencing what executes.

Configuration and collection behavior matter too.

For example, a repository-level conftest.py can participate in pytest's collection process. A pytest.ini can alter discovery settings. Plugins and environment configuration can also influence execution.

Depending on the setup, those mechanisms can change what gets evaluated without modifying a protected test assertion.

Protecting the authority that determines which tests execute.

And even protecting collection configuration isn't a complete solution. The interpreter, dependencies, environment variables, and loaded plugins can introduce additional assumptions.

An exit code of zero tells you that the command completed successfully according to the rules of that execution.

And some require different kinds of evidence.

For example, comparing collected test identifiers may catch missing tests, but identifiers alone won't reveal that a required test was skipped.

Similarly, independently rerunning an unchanged test suite doesn't establish that its assertions represent the correct product requirements.

Execution integrity and test adequacy are different problems.

A common response is to rerun the tests somewhere else.

An independent CI runner or clean environment that the coding agent cannot modify can remove important opportunities for interference.

But the boundary still needs to be defined carefully.

Which repository revision gets evaluated?

Which test definitions are authoritative?

Can the agent change the configuration governing collection?

Which dependencies and plugins are trusted?

What happens when a legitimate task actually requires modifying the tests?

Simply moving execution into another process doesn't answer every question. What matters is whether the verification authority is genuinely independent of the work being judged.

And no matter how well that boundary is constructed, passing tests remain limited by what those tests were designed to detect.

The more I've explored this, the more interested I've become in what happens after a verification system refuses to clear a change.

Because a gate that catches interference but repeatedly demands unnecessary human investigation can create another bottleneck.

Situation A: A required test was silently removed from execution.

Situation B: A legitimate, approved configuration update changed how tests are collected.

Both may require withholding clearance until the verification basis is understood.

But the developer shouldn't have to spend twenty minutes reconstructing the entire run to determine which situation occurred.

The useful output isn't merely a red light.

It's actionable evidence: what changed, what executed, what couldn't be established, and what requires approval.

And this creates a second question worth measuring:

Can independent verification reduce the human effort needed to accept agent-generated work, without weakening the checks that justify acceptance?

That's a harder question than detecting one successful test-tampering example.

It's also arguably the more commercially important one.

I'm Cecelia Turnbeau, an attorney by professional background and the founder of ClaraGate.

I've spent the past several months developing an independent verification approach for autonomous coding-agent workflows.

The current alpha is deliberately narrow: it targets a defined OpenCode and Python/pytest workflow, independently reruns a designated test command using a declared interpreter, and checks a frozen verification surface.

It does not guarantee execution in a clean, agent-inaccessible clone. It does not certify test adequacy or prove code correctness.

The larger idea I'm exploring is that an agent's declaration of completion and the authority to accept its work should be separate things.

But an architectural argument alone isn't sufficient to establish product value.

That has to come from actual developer use.

I'm particularly interested in people using coding agents on real repositories, especially Python/pytest projects.

Not what an ideal verification system should do.

When your coding agent reports that a task is complete:

And perhaps the most interesting question:

If an independent verifier could establish that the designated checks ran against an intact verification basis, would that actually change your review workflow, or would it duplicate evidence you already trust?

If your existing setup already solves this effectively, I'd genuinely like to understand how.

If it doesn't, I'd like to understand where the remaining burden lives.

Because the goal shouldn't be to introduce another gate simply because we can.

The goal should be to make autonomous work more reliably acceptable, with evidence that justifies the decision.

I'm documenting this investigation through ClaraGate Insights. I'm particularly interested in candid technical criticism and firsthand development workflows.