Make your AI agent prove it: a deterministic delivery gate

Tags: ai, testing, agents, devtools Canonical URL: https://skills.seu.pet/en/blog/make-your-agent-prove-it?utm_source=devto&utm_medium=article&utm_campaign=make-your-agent-prove-it

A real run from this week: the builder reported done, the critic scored 55 and returned a gap list with names and weights. Two checks were failing, one of them a secret scan that found a token in the diff. No opinion was involved — the report attached the raw output of every check. This is what a deterministic delivery gate looks like when you stop asking the model whether the work is good and start measuring.

This skill is now published as a real package. pip install agent-test-loop and npx ai-test-loop both resolve, the repository is open (MIT), and the free version is a real download — not a teaser.

The classic failure: one context writes the code, runs the tests, reads the result and decides whether it is done. The model that produced the bug is the model that judges the fix. The gate breaks that loop by moving the judgement into a separate process that executes real commands and returns numbers, not prose.

Rule zero: builder ≠ critic ≠ reviewer. If one context does all three, the loop is theatre.

{ "checks":[{ "name": "pytest", "cmd": "set -o pipefail; pytest -q", "weight": 40 },{ "name": "no_secret_scan", "cmd": "gitleaks detect --no-banner", "weight": 30 }]} Enter fullscreen mode Exit fullscreen mode Run the critic — it executes every command for real, captures the evidence and never asks a model whether the work is good. python scripts/loop_runner.py --config loop.json --round 1 Enter fullscreen mode Exit fullscreen mode Score, compare, decide — above the threshold goes to the reviewer; below, take the largest gap, fix it and run again. Cap the rounds (8 is plenty): if it never converges, the check set is wrong, not the code.

Independent review before delivery — same artifact, fresh context, no justification from the builder. A panel of 2-3 specialists (quality and security always) returning strict JSON.

Security gate, fail-closed — a secret in the diff or history, a high-severity finding, an auth/path-traversal/injection issue block delivery even with perfect tests. An inconclusive answer counts as blocked, never as pass.

Report with score, evidence and what was not verifiable — the report states what was measured, with raw output, and what could not be checked — no "probably fine".

The package ships loop_runner.py, the scoring rubric, the reviewer prompt and the security checks. Two commands, two ecosystems, same gate.