I Don’t Trust an AI Fix Until I Can Make the Bug Come Back

Posted on Oct 31 • Originally published at Medium • Fully Autonomous

Imagine asking Claude to fix a bug, getting a tidy patch, and watching the tests turn green. You are already thinking about the next task when somebody asks which test failed before the fix.

You scroll back through the conversation, and suddenly the reassuring summary is doing a lot of work.

Maybe the fix is correct. Maybe the test never exercised the broken behavior. Maybe both the implementation and its new test agree on the same wrong assumption.

Passing result cannot tell those stories apart by itself.

I prefer a slightly awkward question before getting comfortable with an AI generated repair. If we put the original mistake back into a disposable local copy, does the same test complain again?

That question gives us something more useful than another request for the model to double-check its answer. It gives us a small experiment we can understand.

In its account of code review, the Claude Code team describes giving agents runnable checks and asking for their output before human review. My favorite place to start is smaller than a whole review system, with one bug and a test whose expected answer comes from the requirement.

Here is an invented, deliberately small example. A support dashboard lists tickets from oldest to newest, and the user presses Next to continue where the previous page ended.

For this exercise, the dataset stays fixed while we paginate. Each ticket has an integer creation time and a unique integer ID. Sort by creation time first, then ID, and return only tickets strictly after the last one already shown. The page size is a positive integer.

Our entire dataset fits on a sticky note. Ticket 41 was created at time 100, ticket 42 was also created at time 100, and ticket 43 was created at time 101.

With one ticket per page, the reader should see 41, then 42, then 43. No advanced algorithm is hiding here, which is exactly why this makes a useful review exercise.

# Each row is (created_second, ticket_id). def next_page(rows, after, limit): ordered = sorted(rows) if after is not None: ordered = [ row for row in ordered if row > after ] return ordered[:limit] After the first page, our cursor is (100, 41). The function keeps only tickets with a creation time greater than 100, so it jumps straight to ticket 43. Ticket 42 quietly disappears from the walk through the results.

Nothing crashed, and the remaining ticket looks perfectly reasonable. Checking that the response contains one ticket would let this bug stroll right past us.

Before asking an agent to repair anything, preserve the smallest example that demonstrates the problem. In this case, we can justify the answer by hand because ticket 42 follows ticket 41 in the agreed ordering.

def test_next_page_keeps_a_ticket_with_the_same_time(): rows = [(100, 41), (100, 42), (101, 43)] result = next_page(rows, after=(100, 41), limit=1) assert result == [(100, 42)] Following the original code, the actual result is [(101, 43)], so this assertion should fail. That failure matters because it connects our test to the complaint. These are hand-traced outcomes for the example, rather than a claimed test run.

Notice where the expected answer came from. We worked it out from the ordering rule. We did not call another helper from the implementation to calculate what the result should be, because that helper might carry the same mistake.

If the test passes on the original code, pause before applying the patch. Check that it imports the intended function, that the fixture actually contains a tie, and that the test runner discovered the test. If it fails because a dependency is missing, we still have setup work to do before we have evidence about pagination.

An assertion showing ticket 43 where ticket 42 belongs is useful. An import error, a timeout from an unavailable database, or a test skipped by configuration does not demonstrate this particular bug.

Honestly, this is where a confident AI summary can make a messy situation feel cleaner than it is. Keep the actual command and output so somebody else can distinguish a reproduced defect from an environment problem.

Our ordering uses two fields, but the filter remembers only one. The correction is to compare the complete cursor in the same order used for sorting.

def next_page(rows, after, limit): ordered = sorted(rows) if after is not None: ordered = [row for row in ordered if row > after] return ordered[:limit] Python compares these tuples in order. A later creation time qualifies immediately. When the creation times match, a larger ID qualifies. That makes (100, 42) greater than (100, 41), so ticket 42 stays in the result.

Changing the time comparison from > to >= looks tempting, but it lets ticket 41 back into the next page. We need to include the other tickets at the same time while excluding the ticket already shown.

Keep this repair narrow enough to explain. Renaming every variable, introducing a pagination class, and reorganizing the repository would make it harder to see which change addressed the missing ticket.

The small function is a teaching model that sorts an in-memory list. A real database implementation needs a matching order, cursor predicate and suitable index. With ascending non-null fields, its filter would express created_at > cursor_time OR (created_at = cursor_time AND id > cursor_id). It also needs defined behavior when records change between requests, which our fixed-dataset example deliberately leaves outside its contract.

Once the focused test passes on the patch, make a disposable local copy containing both the patch and the test. Restore only the old timestamp only comparison there, leaving the test and its expected answer unchanged.

The same assertion should fail again, with ticket 43 replacing ticket 42. Now we have checked that this test can distinguish the repaired behavior from the original defect.

This is a small mutation check, meaning we deliberately put a known mistake into the code to see whether the test detects it. Keep that experiment away from production, and retain the clean patched version separately.

If the restored bug still passes, investigate before celebrating. The test might be importing a different copy, a mock might be bypassing the comparison, or the assertion might be too weak to notice which ticket returned.

One successful mutation check does not prove the whole program correct. It tells us something narrower and genuinely useful, which is that this test detects this mistake under these conditions.

Our first case protects ticket 42, but the repair also has to preserve the next page, the end of the list and the page-size rule. These nearby cases make it harder for a special-case patch to masquerade as a general correction.

In a real test file, keep the input data fixed and write the expected results explicitly. Here is one compact way to express those cases using pytest parametrization.

import pytest @pytest.mark.parametrize( ("after", "limit", "expected"), [ ((100, 41), 1, [(100, 42)]), ((100, 42), 1, [(101, 43)]), ((101, 43), 1, []), ((100, 41), 2, [(100, 42), (101, 43)]), (None, 1, [(100, 41)]), ], ) def test_page_boundaries(after, limit, expected): rows = [(101, 43), (100, 42), (100, 41)] assert next_page(rows, after, limit) == expected I deliberately shuffled the input so the test also asks the function to honor its ordering rule. For a larger practice exercise, walk all the pages and compare the collected IDs with a hand-written expected list. That catches omissions and duplicates across page boundaries.

Then run the relevant existing suite. A focused regression test can protect our missing ticket while missing a broken route, a changed response field or an authorization check elsewhere.

For a database backed endpoint, add coverage that reaches the actual query and cursor encoding. For a frontend race, control the order in which requests resolve and check what appears on screen. For a permission bug, exercise the relevant user boundary instead of checking only a helper that returns true.

Choose the next check by asking which layer could still break the promised behavior. That is a better guide than trying to make an arbitrary test-count number look impressive.

You can ask an AI assistant to help construct the reproduction, but review the expected answer before letting it repair the implementation. Otherwise, the assistant can accidentally write a test that politely agrees with its own patch.

For this kind of bug, a useful request can stay wonderfully plain.

Reproduce the reported failure before changing application code. Explain why the expected answer follows from the requirement. Run the focused test on the original implementation and show the command, the failure, and why it matches the report.

Make a focused repair without weakening the assertion. If a test expectation needs changing, explain the contract conflict first instead of silently updating it. Check the patched code with the same test and nearby cases.

In a disposable local copy, restore the original defect and check that the regression test detects it again. Report what ran, what did not run, and what remains uncertain. This request does not make an agent infallible.

It makes its work easier to inspect, and it gives you specific places to intervene when a result is missing or an assumption is wrong.

If the project cannot run locally, preserve the reproduction and explain the blocker. A reasoned expectation can still help a reviewer, but label it as unexecuted. There is no need to dress up a prediction as a green test result.

For a follow-up, try an existing API debugging exercise and choose one failing behavior before touching the rest. That exercise covers a different movie-search API, so our ticket-pagination patch is not its answer. The reusable part is the evidence trail.

Keep one note with the requirement, the smallest reproduction, the cause, the focused change and the checks you actually ran. During an interview, that note becomes a clear explanation of your decisions. During a review, it gives the next engineer somewhere useful to start.

Some real bugs are too intermittent to reproduce on demand. For those, capture the relevant timing, versions and observations, then use controlled scheduling, replay or monitoring where possible. Be honest about the uncertainty instead of promising that one neat unit test settles the incident.

What I want from an AI fix is a trail I can follow from the complaint to the corrected behavior. Once I understand that trail, I can explain the change without rereading the assistant’s entire conversation.

Before accepting your next patch, try keeping the test still while bringing the mistake back. If it catches the bug again, you have earned a much more useful kind of confidence.