What If Your AI Code Reviewer Remembered Bugs That Haven’t Happened Yet!

Your AI reviewer approves the pull request.

Three hours later, a customer gets charged twice.

The bug wasn’t hidden in a complicated algorithm. It lived in an assumption nobody wrote down:

“This operation must happen before that one.”

What if your AI code reviewer could discover that assumption, deliberately break it in a sandbox, and remember the resulting failure—before a customer ever experienced it?

That’s the idea behind the Counterfactual Wiki.

A memory of what could go wrong, backed by executable evidence.

Documentation describes how a system is supposed to work.

Incident reports explain how the system actually failed.

Where do we store plausible failures that we have reproduced experimentally, but that have never happened in production?

A document can describe them, but the description can drift.

A test can catch them, but the reason behind the test can disappear.

An incident report usually arrives too late to prevent the first incident.

Build a codebase knowledge system that discovers assumptions, challenges them in isolated experiments, and stores the resulting evidence alongside executable checks.

For a tool like Greptile, this would be an extension of the living-wiki idea discussed earlier—not simply more context for a reviewer, but context the reviewer can actively test.

To be clear: this is a proposed product direction, not a claim about an existing Greptile feature.

“What would happen if something were different?”

“What small, plausible change would violate an important property of this system?”

Read the code and its history ↓ Identify a candidate assumption ↓ Create a controlled change in a sandbox ↓ Try to reproduce a meaningful failure ↓ Store the evidence and a regression check ↓ Use that memory when reviewing future changes Enter fullscreen mode Exit fullscreen mode The important word is evidence.

An agent saying “this might be unsafe” is a hypothesis.

An agent producing a minimal failing scenario gives a developer something concrete to inspect.

The wiki should preserve that distinction.

Imagine a pull request with this description:

Reduce payment latency by recording the request identifier after the payment completes.

# Before reserve_request ( request_id ) charge_customer ( payment ) mark_completed ( request_id ) Enter fullscreen mode Exit fullscreen mode # After charge_customer ( payment ) record_request ( request_id ) Enter fullscreen mode Exit fullscreen mode A conventional review might discuss error handling or naming.

Our proposed reviewer would ask a different question:

Is reserving the request identifier before charging part of the duplicate-prevention mechanism?

In an isolated environment, using a simulated payment provider, it could:

If the experiment reproduces duplicate charging, the review comment could look like this:

This change appears to break duplicate-request protection.

In a controlled concurrency test, two requests with the same identifier both reached the charge operation before either recorded the identifier.

The original implementation blocked the second request in this scenario.

Attached: the execution trace and a proposed regression test.

It is also narrower than claiming the agent has proved the entire payment system correct. It has demonstrated one specific failure under stated conditions.

Each discovery becomes a structured Failure Card.

title: Duplicate charging under concurrent retries status: reproduced_in_sandbox candidate_invariant: A repeated request must not create a second charge. relevant_components: - request reservation - payment execution - retry handling trigger: Two concurrent requests share the same request identifier. experimental_change: Move request recording after payment execution. observed_result: Both requests reach the simulated charge operation. artifacts: - minimal reproducer - execution trace - proposed regression test scope: Controlled test environment with a simulated payment provider. limitations: Does not establish behavior for every deployment configuration. Enter fullscreen mode Exit fullscreen mode The exact schema is less important than the separation between:

A simulated failure must never quietly become a “historical incident” in the system’s memory.

Now imagine another pull request six weeks later.

But it touches the same duplicate-prevention boundary.

The reviewer retrieves the Failure Card and asks:

Does this change reopen the failure mode we reproduced before?

It can rerun the relevant experiment rather than generate a fresh warning from scratch.

Assumption → Experiment → Failure → Memory → Targeted review Enter fullscreen mode Exit fullscreen mode The organization accumulates two distinct kinds of knowledge:

“This failed in production. Here is what happened.”

“This failed in a controlled simulation. Here are the conditions and evidence.”

The proposal deliberately introduces changes and checks whether a failure becomes visible.

But the intended unit of memory is not merely a changed expression or a surviving mutation.

It is a meaningful system assumption, connected to:

The concept could incorporate mutation testing, property-based testing, fault injection, or concurrency testing. It does not need to replace them.

The proposed product layer is the connection between those experiments, a codebase knowledge system, and the pull-request workflow.

The wiki becomes an index of executable reasoning—not a substitute for the underlying testing techniques.

An agent could generate hundreds of tests and still miss the assumption that matters.

For this product, the objective should not be:

“Which important failure can we now detect that we could not detect before?”

A useful experiment needs a clear chain:

Important property ↓ Plausible way to violate it ↓ Observable consequence ↓ Repeatable detection Enter fullscreen mode Exit fullscreen mode Without that chain, the system risks becoming an expensive machine for producing test-shaped text.

“An autonomous agent that discovers every hidden assumption in your software.”

That promise is too broad to evaluate honestly.

Tenant isolation in a multi-tenant application.

The initial system would examine changes affecting:

It would use synthetic tenants and synthetic records, then attempt specific cross-tenant access scenarios in an isolated environment.

Removing the tenant identifier from this cache key allows a value written for tenant A to be returned to tenant B in the attached test.

That is a more credible starting point than universal autonomous review.

Making a system behave differently is easy.

Proving that the new behavior violates an intended requirement is harder.

But perhaps asynchronous execution is intentional.

The product therefore needs an assumption lifecycle:

Candidate assumption ↓ Supporting code or documentation ↓ Experimental result ↓ Developer confirmation or rejection ↓ Maintained knowledge Enter fullscreen mode Exit fullscreen mode Developer feedback should refine the memory, not merely dismiss a comment.

“This is expected behavior” is valuable information.

“This property matters only for enterprise accounts” is valuable information.

“This was true before the migration” is valuable information.

A wiki that cannot revise its assumptions will eventually become a confident archive of obsolete beliefs.

A system designed to break things must have strict boundaries.

Experiments should be reproducible and inspectable.

If an agent cannot explain what it changed, what it ran, and what it observed, its warning should not be treated as a confirmed finding.

Not a spectacular demo where an agent discovers a bug deliberately planted for it.

The key experiment would compare this approach with an ordinary review workflow on the same set of changes.

Does the additional machinery find enough meaningful failures to justify its cost and delay?

That is a question to measure—not a benefit to assume.

Give an AI reviewer a memory of failures it has experimentally reproduced, then let it use that memory to challenge future changes.

A living wiki helps an agent understand what a system does.

A Counterfactual Wiki would help it investigate what the system must not do.

The ambition is not to predict every incident.

It is to make selected failure modes concrete, testable, and reusable before they become somebody else’s bad day.

Because the most useful thing an AI reviewer could say might not be:

“I tried to break this assumption. Here is exactly what happened.”

created by https://almodarresieh.github.io/