A regex matched my LLM pipeline on clean text. Then synthetic speech broke both.

An operator on a plant radio says "Close valve two-oh-three, line Bravo." The field technician answers "Copy, closing two-oh-eight, line Bravo." Valve 208 is on a different line. The read-back exists so somebody catches that, and sometimes nobody on the channel does.

I spent the AMD Developer Hackathon (ACT III, lablab.ai) building SayAgain, a checker for exactly that moment. This post is not the launch tour. It is the part I didn't expect: on clean text a plain regex did as well as the language model, and when I fed the pipeline synthetic speech, both of them got worse in a way that a headline number hides.

Live replay: https://sayagain.edycu.dev/replay/ · Repo (MIT): https://github.com/edycutjong/sayagain · Demo video: https://youtu.be/6a4IF-iJmxg

The obvious build is "ask an LLM whether the read-back matches the order." I didn't want a model making a safety verdict, so the model's only job is to point at words: which span of the transcript is the action, which is the equipment tag, which is the line. It runs on the AMD Token Factory (an OpenAI-compatible endpoint, JSON mode), and every call writes a receipt.

Then plain Python takes over. The spans must be words really present in the transcript (grounding). Spoken digits and NATO letters become plant tags through alias tables. The plant registry decides whether a tag exists and which line it is on. And a pure function returns the verdict:

def verdict ( instr: list [ CanonCommand ], readback: list [ CanonCommand ] | None, *, span_conf: float | None = None, floor: float | None = None, disagreement: bool = False, ) -> Verdict: """ VERIFIED · MISMATCH · NO_READBACK · ABSTAIN for one paired exchange (canonical inputs only). """ bad = validate_instruction ( instr ) if bad: return Verdict ( ABSTAIN, bad, trace = [ f " instruction: { bad } " ]) trace = [ " instruction: " + ", ". join ( f " { c. tag } line { c. line or c. registry_line } " for c in instr ) + " (registry ok) " ] if readback is None: return Verdict ( NO_READBACK, trace = [ * trace, " pairing: no read-back inside the window " ]) if any ( r. tag_status == " ambiguous " for r in readback ): return Verdict ( ABSTAIN, R_AMBIGUOUS, trace = [ * trace, " read-back: bare number shared by several tags " ]) if disagreement: return Verdict ( ABSTAIN, R_AUDIO, trace = [ * trace, " audio: prompted and unprompted transcripts disagree " ]) if floor is not None and span_conf is not None and span_conf < floor: return Verdict ( ABSTAIN, R_AUDIO, trace = [ * trace, f " audio: span confidence { span_conf:. 3 f } < floor { floor:.

3 f } " ]) diffs = compare ( instr, readback ) if diffs: return Verdict ( MISMATCH, diffs = diffs, trace = [ * trace, " compare: " + "; ". join ( str ( d ) for d in diffs )]) return Verdict ( VERIFIED, trace = [ * trace, " compare: every field equal " ]) Enter fullscreen mode Exit fullscreen mode ( referee/verdict.py — the module imports no model code.)

Notice how many exits lead to ABSTAIN. If the system can't read something, it says so and hands the card to a human. A failed model call is NOT_AUDITED, never a guess. Every card a supervisor sees ends with Confirm or Dismiss.

Before calling any model, I committed an evaluation protocol and tagged it ( prereg-v1 ). The ladder rule: walk up from the cheapest extractor (a regex, no model) through larger models, and pick the smallest rung that lets zero wrong read-backs through as VERIFIED while keeping false alarms within a budget a shift can absorb.

That rule was written for the audio row: real recordings, real radio. I didn't have recordings during the hackathon, so I applied it to the recording cards' written scripts and logged that as a deviation. Everything below says which condition it is.

On 89 held-out scripted exchanges (41 with a seeded error, 48 correct):

The honest reading: by my own pre-registered rule, on scripted words no language model is needed. I put that in the README in those words. It's the kind of sentence a project write-up usually leaves out, and it is the most useful thing the benchmark told me: clean text is not where the model earns its place.

One result did favour the design. When the same model returned normalised values instead of copying words from the transcript, false alarms went from 2/48 to 6/48; in 4 of those it turned a spoken pump or motor-valve tag into a valve tag the registry doesn't have. Making the model copy words, and making code check them, matters.

To exercise the speech half, I ran an exploratory pass: ElevenLabs text-to-speech of the 145 card scripts (two voices), through a simulated radio path, transcribed by local Whisper-large-v3, then the same frozen referee. This was added after the text rows, it is not pre-registered, and it is not human speech. Same 89 held-out cards.

That third bullet is the finding. "0 wrong read-backs passed" stayed true the whole time, and it would make a clean headline. But an error that produces no card is not caught either; it just never reaches a human. The stricter form of my own rule counts those as failures, and the README now says so.

Where the false alarms came from is also worth knowing, because none of it is the model being clever or dumb:

All of these fail toward a human, which is the right direction. They still cost the supervisor's attention, which is the budget the rule protects.

The next step is the row the rule was written for: real people, real radios, real plant noise.

Try the refused one: https://sayagain.edycu.dev/replay/ · Code: https://github.com/edycutjong/sayagain · Video: https://youtu.be/6a4IF-iJmxg

If you work with plant or field radio, I'd value one sentence on whether operators actually read back tag numbers and NATO letters like this.