Five Fail-Closed Gates, One Script: Rehearse Your AI Feature on a Free Tier First

My readiness checklist for small AI features is a Markdown file. It has nine boxes and zero exit codes. Nobody has opened it since August.

That is the whole problem. A checklist you read is a wish. A checklist you run is a gate.

So I rewrote mine as one script. It runs in about two seconds. It exits non-zero when anything is wrong. And it runs first against a deliberately broken fake model, not against a real one.

Here is the harness, the evidence it writes, and the limits of rehearsing on a free tier.

Most teams test the model. Few test the guard around the model.

Ask yourself: if the model returned truncated JSON right now, what happens? Does your service crash, show a blank card, or quietly bill you twice? If you cannot answer in one sentence, you have a doc, not a gate.

A gate needs three parts. A deterministic check. A written evidence record. A fail-closed action that runs without a human.

This is the step everyone skips. You cannot trust a green run until you have seen a red one.

So the harness ships with fixtures that are already broken. Empty string. Garbage prose. Truncated JSON. Oversized output.

Run the suite against those fixtures first. If anything passes, your gate is decoration. Fix the gate before you touch a real provider.

That single habit has saved me more deploy nerves than any prompt tweak.

Pure standard library. No dependencies. Drop it in tools/gates.py.

#!/usr/bin/env python3 """ gates.py - five fail-closed gates for a small AI feature. """ import argparse import json import os import time from dataclasses import dataclass, asdict, field MAX_OUT_CHARS = 4000 MAX_LATENCY_MS = 4000 REQUIRED_KEYS = { " summary ", " confidence " } FIXTURES = { " empty ": "", " garbage ": " lorem ipsum dolor sit amet " * 40, " truncated ": ' { " summary ": " refunds take five days ", " confidence ": 0. ', " oversize ": json. dumps ({ " summary ": " x " * 9000, " confidence ": 0.9 }), " good ": json. dumps ({ " summary ": " refunds take five days ", " confidence ": 0.9 }), } @dataclass class Result: gate: str ok: bool detail: str evidence: dict = field ( default_factory = dict ) def call_fake ( fixture: str ) -> str: return FIXTURES [ fixture ] def call_live ( prompt: str ) -> str: # Wire this to your provider. Read the key from os.environ only. raise SystemExit ( " call_live() is not wired yet " ) def gate_schema ( text: str ) -> Result: try: obj = json. loads ( text ) except json.

JSONDecodeError as e: return Result ( " schema ", False, f " unparseable: { e } ", { " chars ": len ( text )}) missing = REQUIRED_KEYS - obj. keys () return Result ( " schema ", not missing, f " missing: { sorted ( missing ) } ", { " keys ": sorted ( obj )}) def gate_size ( text: str ) -> Result: ok = 0 < len ( text ) <= MAX_OUT_CHARS return Result ( " size ", ok, f " { len ( text ) } chars, cap { MAX_OUT_CHARS } " ) def gate_latency ( ms: int ) -> Result: return Result ( " latency ", ms <= MAX_LATENCY_MS, f " { ms } ms, ceiling { MAX_LATENCY_MS } ms " ) def gate_budget ( used: int, allowance: int ) -> Result: ok = used < allowance * 0.2 return Result ( " budget ", ok, f " { used } of { allowance } tokens, 20% cap " ) def gate_rollback ( flag: str ) -> Result: ok = os. path. exists ( flag ) return Result ( " rollback ", ok, f " kill switch at { flag } " ) def main () -> None: ap = argparse. ArgumentParser () ap. add_argument ( " --model ", choices = [ " fake ", " live " ], default = " fake " ) ap. add_argument ( " --fixture ", default = " garbage ", choices = sorted ( FIXTURES )) ap. add_argument ( " --out ", default = " evidence/gates.json " ) args = ap. parse_args () allowance = int ( os. environ. get ( " TOKEN_ALLOWANCE ", " 0 " )) if allowance <= 0: raise SystemExit ( " set TOKEN_ALLOWANCE to your own plan ' s number " ) start = time. monotonic () if args. model == " fake ": text = call_fake ( args. fixture ) else: text = call_live ( " summarize the support fixture " ) ms = int (( time. monotonic () - start ) * 1000 ) results = [ gate_schema ( text ), gate_size ( text ), gate_latency ( ms ), gate_budget ( len ( text. split ()), allowance ), gate_rollback ( os. environ. get ( " KILL_SWITCH ", ".rollback/feature.flag " )), ] failed = [ r. gate for r in results if not r. ok ] payload = { " when ": time. strftime ( " %Y-%m-%dT%H:%M:%SZ ", time. gmtime ()), " model ": args. model, " fixture ": args. fixture, " passed ": len ( results ) - len ( failed ), " failed ": failed, " results ": [ asdict ( r ) for r in results ], } os. makedirs ( os. path. dirname ( args. out ), exist_ok = True ) with open ( args. out, " w " ) as f: json. dump ( payload, f, indent = 2 ) print ( " failed: " + ", ". join ( failed ) if failed else " all gates passed " ) raise SystemExit ( 1 if failed else 0 ) if __name__ == " __main__ ": main () Enter fullscreen mode Exit fullscreen mode Run it in four commands mkdir -p.rollback && touch.rollback/feature.flag export TOKEN_ALLOWANCE = 100000 python3 tools/gates.py --model fake --fixture garbage; echo "exit= $? " python3 tools/gates.py --model fake --fixture good; echo "exit= $?

" Enter fullscreen mode Exit fullscreen mode The first run must print failed: schema and exit=1. The second must print all gates passed and exit=0.

If both runs look identical, stop. You have a logging script, not a gate.

Then open evidence/gates.json. That file is the thing you attach to a release, not a screenshot of a green checkbox.

Five rows. Five verdicts. No adjectives.

Here is where a free option earns its place. Failing paths cost calls. Empty output, truncation, retries, oversized responses — those are the expensive ones to discover in production.

I rehearse the whole red path against MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free allowance lets me burn calls on intentionally broken fixtures without a card on file.

A free server is also a fine place to host the kill switch check. If curl cannot reach your flag, your rollback is a rumour.

Skip this if your feature already moves money. Skip it if you have a pager rotation and real SLAs; you need runbooks, not scripts. Skip it if there is nobody who can own a red exit code at 2am.

Also skip it if you cannot name your kill switch. That is a design gap, and no harness fixes it.

Give it two hours. If the fake run still cannot go red, the harness is wrong. Delete it and start over rather than keeping a fuzzy gate that always passes.

And if a gate cannot produce a deterministic verdict, remove it. A checklist with nine soft boxes is how we got here.

One question for my next iteration: what is the one input your feature cannot survive losing? I want to turn that answer into the next fixture.