Flaky Is Not Failed: A Repeat-k Triage Loop for AI-Written C++ Patches

Six golden cases turn red in the same CI run. The team spends two days reverting a patch that was correct, and the single case that had genuinely regressed gets fixed last, almost by accident.

That scenario is a composite — the timeline is illustrative, the pattern is not. Any harness that grades model-written C++ against a fixed set of golden cases will eventually produce a run where the same case disagrees with itself. Two things follow. A two-state pass/fail result cannot describe that world, and a harness without a vocabulary for "I do not know" will convert infrastructure noise into engineering decisions.

The artifacts below are templates. They target Python 3.11 and a CMake/CTest tree, and they were not executed for this article, so treat paths and the runner call as starting points.

A golden case can fail for reasons that have nothing to do with the patch: a rate-limited model call, a build that timed out on a shared box, a runner that lost its cache. Grading those as fail is how noise becomes a revert. The runner below records infra_unknown as a distinct outcome and refuses to reason about it later.

# triage.py — repeat each golden case K times, then classify. Template. import hashlib, os, re, subprocess, tempfile from dataclasses import dataclass K = int ( os. environ. get ( " REPEAT_K ", " 5 " )) INFRA_RE = re. compile ( r " (429|503|connection reset|build timeout|no space left) ", re. I ) @dataclass class Attempt: case: str repeat: int status: str # pass | fail | infra_unknown exit_code: int digest: str note: str = "" def digest ( blob: str ) -> str: # Timing and pid lines track the scheduler, not the behaviour. cleaned = " \n ". join ( line for line in blob. splitlines () if not re. search ( r " (total time|pid=|elapsed) ", line, re.

I ) ) return hashlib. sha256 ( cleaned. encode ()). hexdigest ()[: 12 ] Enter fullscreen mode Exit fullscreen mode Step 2 — Reuse the case, not the process Every repeat must start from a clean state. The environment pinning in this runner is deliberate: LC_ALL=C and TZ=UTC remove two classic sources of "it passed on the other machine", and a per-repeat TMPDIR keeps stale artifacts from leaking across attempts.

def run_case ( case: str, repeat: int ) -> Attempt: with tempfile. TemporaryDirectory () as td: proc = subprocess. run ( [ " ctest ", " --test-dir ", " build ", " -R ", f " ^ { case } $ ", " --output-on-failure " ], capture_output = True, text = True, timeout = 900, env = { ** os. environ, " LC_ALL ": " C ", " TZ ": " UTC ", " REPEAT ": str ( repeat ), " TMPDIR ": td }, ) blob = proc. stdout + proc. stderr if INFRA_RE. search ( blob ): return Attempt ( case, repeat, " infra_unknown ", proc. returncode, "", blob [ - 400:]) return Attempt ( case, repeat, " pass " if proc. returncode == 0 else " fail ", proc. returncode, digest ( blob )) Enter fullscreen mode Exit fullscreen mode Step 3 — Classify into four verdicts The interesting bucket is not flaky. It is stable_pass_varying: a case that passes every single repeat while printing two different digests.

That case is a silent regression waiting for a slightly different machine, and it is invisible to any harness that only reads exit codes.

def classify ( attempts: list [ Attempt ]) -> dict: graded = [ a for a in attempts if a. status!= " infra_unknown " ] infra = len ( attempts ) - len ( graded ) if infra * 2 > len ( attempts ): return { " verdict ": " inconclusive ", " infra ": infra } passes = sum ( a. status == " pass " for a in graded ) digests = len ({ a. digest for a in graded if a. status == " pass " }) if passes == 0: verdict = " stable_fail " elif passes < len ( graded ): verdict = " flaky " elif digests > 1: verdict = " stable_pass_varying " else: verdict = " stable_pass " return { " verdict ": verdict, " infra ": infra, " passes ": passes, " digests ": digests } Enter fullscreen mode Exit fullscreen mode Step 4 — Bind each verdict to a CI action A verdict that does not change what CI does is a log line, not a gate.

That last row carries the most weight. Against a shared model endpoint, inconclusive will be common, and a harness that turns it into a failure teaches reviewers to ignore failures.

Unstable cases need somewhere to live that is not /dev/null and not a permanently ignored test.

# quarantine.yaml - case: containers/unordered_dedup reason: iteration order leaks into stdout digest owner: finley opened: 2026-10-19 expires: 2026-11-02 Enter fullscreen mode Exit fullscreen mode # gate.py — an expired quarantine is a build failure, not a reminder import datetime as dt, sys, yaml today = dt. date. today () bad = [ q for q in yaml. safe_load ( open ( " quarantine.yaml " )) if dt. date. fromisoformat ( q [ " expires " ]) < today ] if bad: sys. exit ( " expired quarantine: " + ", ". join ( q [ " case " ] for q in bad )) Enter fullscreen mode Exit fullscreen mode Step 6 — Treat the source of the nondeterminism Repeat-k is triage, not treatment. Three C++ changes remove most of the churn seen in practice.

// 1. Order leaks into output: per-process hashing, different bytes each run. std:: unordered_set < std:: string > seen; // digest churns std:: set < std:: string > seen_ok; // stable bytes, assertable // 2. Wall clock: inject it, then assert on it. struct Clock { virtual ~ Clock () = default; virtual std:: time_t now () const = 0; }; struct FixedClock final: Clock { std:: time_t t; std:: time_t now () const override { return t; } }; // 3.

Worker count: 3 threads locally, 32 in CI, different interleavings. const unsigned workers = std:: min ( 4u, std:: max ( 1u, std:: thread:: hardware_concurrency ())); Enter fullscreen mode Exit fullscreen mode Add -fsanitize=address,undefined to the triage build rather than the shipping build. Uninitialised reads and signed overflow produce exactly the intermittent failures that repeat-k flags, and a sanitizer report names the bug instead of leaving a pass-rate puzzle. setarch -R is also worth a try when a crash reproduces under one address layout only.

The arithmetic is the binomial one: a 10% flake survives ten repeats with probability 0.9^10 ≈ 0.35. So repeat-k is a coarse filter, and it is honest to say so before anyone wires k=20 into every pull request. Rare nondeterminism is caught better by digest divergence on green runs and by a nightly sweep, not by raising k in the merge path.

A k=10 sweep over 40 golden cases is 400 model calls, and on a metered endpoint that arithmetic appears on an invoice. MonkeyCode's free model access and free server option make such a sweep affordable for a side project or a first pass on a feature branch, which is the cheapest moment to discover that six of your cases were never stable.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two constraints shape how the tool is used here. Free capacity is shared and rate-limited, so the INFRA_RE branch in Step 1 is not decoration — it is the point of the three-state design. The free tier also belongs in the triage loop, not the release gate, because a merge decision should never depend on capacity someone else controls. The offer itself (the operator states 10M free tokens plus a free server option) is worth confirming on the project's own page before planning around it.

To see the classifier fire before wiring it into CI, point it at the two cases you already suspect and spend the first sweep on a scratch branch.

The habit worth keeping is small: a red run should produce a verdict with a name, and flaky and inconclusive are names. Without them every red run looks identical, and the team slowly learns to treat all of them as noise — including the one that was real.