Regression-Test the Docs Prompt: A Fixture Suite and Five Checks Before a Model Drafts Your Handbook

The cheapest place to catch a bad docs draft is a fixture suite that finishes in about ninety seconds, not a reviewer's Friday afternoon. Once a team lets a model draft documentation, the thing under test is not the draft — it is the prompt, the model identifier, and the source tree as one versioned unit. A small directory of fixtures with explicit expectations turns that unit into something you can diff, gate, and roll back. This article lays out the fixture format, five mechanical checks, and a model admission policy, plus the parts a human must keep owning no matter how green the suite turns.

A variance run asks whether a draft is stable across repeated generations, and it is cheap to bolt onto whatever pipeline you already have. A fixture run asks a sharper question: did a prompt edit, a model swap, or an unrelated dependency bump silently break a rule the team already agreed on? Stability tells you how much to trust a single draft; fixtures tell you whether a change to the generator violated a contract you had previously enforced.

Both are useful, but only one of them belongs in a required CI status check. Variance needs repeated calls and produces a distribution you have to interpret; fixtures produce a pass or fail per named rule, which is what a merge button can act on.

A fixture is a miniature repo plus a JSON file that states what the draft must and must not contain. Keep fixtures small enough that a human can read the whole thing in thirty seconds, because a fixture nobody understands becomes a fixture everybody silently disables.

docsgen/ fixtures/ config-timeout/ source/ retry.py pinned/ security-notes.md expected.json harness/ run_fixtures.py reports/ Enter fullscreen mode Exit fullscreen mode # fixtures/config-timeout/source/retry.py DEFAULT_TIMEOUT = 30.0 MAX_RETRIES = 3 def fetch ( url, timeout = DEFAULT_TIMEOUT, retries = MAX_RETRIES ): """ Fetch a URL with a bounded retry budget. """ raise NotImplementedError Enter fullscreen mode Exit fullscreen mode { "doc_class": "api-reference", "must_claim": [ "default timeout is 30 seconds", "retries are bounded" ], "must_not_claim": [ "unlimited retries", "works on python 2", "zero configuration" ], "pinned_blocks": [ "pinned/security-notes.md" ], "check_executable": true, "allowed_sections": [ "Signature", "Timeouts", "Failure modes" ] } Enter fullscreen mode Exit fullscreen mode The two lists do asymmetric work. must_claim guards against omission, which is the failure mode that produces confidently incomplete reference pages. must_not_claim guards against invention, and it should be seeded from real incidents: every hallucinated number or fabricated guarantee that reaches a reader earns a permanent line here.

Checks only mean something if the prompt demands machine-readable structure. Put four rules in the system prompt, and keep them there across every model you evaluate so the comparison stays fair.

Rule 1 is what lets a mechanical check find unstamped facts. Rule 2 is what turns a human-owned security note into a byte-for-byte assertion rather than a hope.

The harness below is a reference implementation, not a benchmark. It uses only the standard library so you can read it end to end, and it talks to any OpenAI-compatible /chat/completions endpoint.

#!/usr/bin/env python3 """ Run docs-drafting fixtures against one OpenAI-compatible endpoint. """ from __future__ import annotations import json import pathlib import re import subprocess import sys import tempfile import urllib.request ROOT = pathlib. Path ( __file__ ). resolve (). parents [ 1 ] FIXTURES = ROOT / " fixtures " FENCE = re. compile ( r " ``` (?:python|py)\n(.*?) ``` ", re.

S ) STAMP = re. compile ( r " <!--\s*human-review\s*-->(.*?)<!--\s*/human-review\s*--> ", re. S ) NUM = re. compile ( r " \d+(?:\.\d+)? " ) SYMBOL = re. compile ( r " `([A-Za-z_][\w.]*\.[A-Za-z_]\w*)` " ) def call_model ( base_url, api_key, model, prompt, timeout = 60 ): payload = json. dumps ({ " model ": model, " temperature ": 0, " messages ": [{ " role ": " user ", " content ": prompt }], }). encode () req = urllib. request.

Request ( f " { base_url. rstrip ( ' / ' ) } /chat/completions ", data = payload, headers = { " Content-Type ": " application/json ", " Authorization ": f " Bearer { api_key } ", }, ) with urllib. request. urlopen ( req, timeout = timeout ) as resp: return json. load ( resp )[ " choices " ][ 0 ][ " message " ][ " content " ] def check_must_claim ( draft, needles ): low = draft. lower () missing = [ n for n in needles if n. lower () not in low ] return { " name ": " must_claim ", " ok ": not missing, " detail ": missing } def check_must_not_claim ( draft, needles ): low = draft. lower () hits = [ n for n in needles if n. lower () in low ] return { " name ": " must_not_claim ", " ok ": not hits, " detail ": hits } def check_pinned ( draft, fixture_dir, pinned ): drifted = [] for rel in pinned: text = ( fixture_dir / rel ). read_text (). strip () if text not in draft: drifted. append ( rel ) return { " name ": " pinned_blocks ", " ok ": not drifted, " detail ": drifted } def check_executable ( draft, timeout = 10 ): failures = [] for index, block in enumerate ( FENCE. findall ( draft )): with tempfile. TemporaryDirectory () as tmp: path = pathlib. Path ( tmp ) / f " block_ { index }.py " path. write_text ( block ) try: proc = subprocess. run ( [ sys. executable, str ( path )], capture_output = True, text = True, timeout = timeout, cwd = tmp, ) if proc. returncode!= 0: failures. append (( index, proc. stderr. strip ()[ - 200:])) except subprocess.

TimeoutExpired: failures. append (( index, " timeout " )) return { " name ": " executable ", " ok ": not failures, " detail ": failures } def check_symbols ( draft, fixture_dir ): declared = set () for source in ( fixture_dir / " source " ). rglob ( " *.py " ): text = source. read_text () for match in re. finditer ( r " ^(?:class|def)\s+(\w+) ", text, re. M ): declared. add ( f " { source. stem }. { match. group ( 1 ) } " ) unresolved = sorted ({ s for s in SYMBOL. findall ( draft ) if s not in declared }) return { " name ": " symbols ", " ok ": not unresolved, " detail ": unresolved } def check_unstamped_numbers ( draft, fixture_dir ): corpus = " ". join ( p. read_text () for p in ( fixture_dir / " source " ). rglob ( " * " ) if p. is_file () ) stamped = " ". join ( m. group ( 1 ) for m in STAMP. finditer ( draft )) prose = FENCE. sub ( "", STAMP. sub ( "", draft )) known = set ( NUM. findall ( corpus )) | set ( NUM. findall ( stamped )) unsourced = sorted ({ n for n in NUM. findall ( prose ) if n not in known }) return { " name ": " unstamped_numbers ", " ok ": not unsourced, " detail ": unsourced } Enter fullscreen mode Exit fullscreen mode The driver reads each expected.json, assembles the prompt from source/ and the pinned files, calls the endpoint once at temperature=0, runs the checks, and writes one report JSON per model. Keep base_url, api_key, and model in environment variables so that a score is always tied to a named endpoint and a named model string.

Add a --stub mode that returns a canned draft so the harness itself can be smoke-tested in CI without any network call.

A report line for one fixture might look like this — the format is illustrative, and the values must come from your own run:

{ "fixture": "config-timeout", "doc_class": "api-reference", "model": "<record the exact model string and date>", "checks": { "must_claim": { "ok": true, "detail": []}, "must_not_claim": { "ok": true, "detail": []}, "pinned_blocks": { "ok": true, "detail": []}, "executable": { "ok": false, "detail": [[ 2, "ModuleNotFoundError: requests" ]]}, "symbols": { "ok": true, "detail": []}, "unstamped_numbers": { "ok": false, "detail": [ "50" ]} }, "verdict": "reject" } Enter fullscreen mode Exit fullscreen mode 5. Convert scores into a model admission table, not a leaderboard A leaderboard invites you to pick one global winner. Admission asks a narrower question per doc class: which model is allowed to produce a first draft here, and which claims still require a human signature?

Write the policy before you look at the scores, so the results cannot talk you into relaxing it.

The last row matters more than the other three. A fixture suite cannot certify a security claim, and a passing score is never a reason to move a compliance page into the autodraft lane.

Evaluating several candidate models usually dies in procurement, not in engineering: you need endpoints for four models before you can fill the admission table. Free model access removes that step, because the harness only needs an OpenAI-compatible base URL, and the free server option means the runner itself — a cron job plus a static report directory — does not need its own provisioned machine.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What that does not change is the method. The fixture suite, the expected.json files, and the admission policy are yours, portable, and useful with any endpoint. Treat the free tier as operator-supplied availability that can change: check the current terms before you make a required CI check depend on it, keep a fallback endpoint in configuration, and sleep between fixtures so you stay inside rate limits rather than discovering them by accident.

#.github/workflows/docs-fixtures.yml (fragment) name: docs-fixtures on: pull_request: paths: [ ' docs/**', ' prompts/**', ' fixtures/**' ] jobs: fixtures: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: python harness/run_fixtures.py --stub - run: python harness/run_fixtures.py --report reports/pr.json env: DOCS_BASE_URL: ${{ secrets.DOCS_BASE_URL }} DOCS_API_KEY: ${{ secrets.DOCS_API_KEY }} DOCS_MODEL: ${{ vars.DOCS_MODEL }} Enter fullscreen mode Exit fullscreen mode Limitations, and who should not adopt this Fixtures are a proxy, not proof. Twenty small fixtures can pass while the four-hundred-page handbook still misleads a reader, because the failures that matter most in prose — a missing precondition, a wrong ordering of steps, a subtly stale migration note — are semantic and resist regex. The executable check only covers pages that carry code, and the unstamped-number check flags anything from a real API limit to a year in a sentence.

Model scores also expire. Free endpoints can change the model behind a stable-looking name, so record the exact model string and the run date in every report, and treat any table older than your last model change as void. Do not adopt this workflow if you have fewer than about a dozen pages and no prompt churn, if you have no CI to hang the check on, or if nobody can name a single human owner per doc class — in that last case the suite will pass and nobody will act on a failure.

Start with three fixtures drawn from real defects your readers have reported, make must_not_claim blocking, and let the suite grow only when an incident earns a new rule. If you want to compare a few endpoints before committing budget to any tooling, running the same fixtures across free model access on a free server is a low-cost way to fill the admission table — but the fixture suite, not the provider, is what makes a draft reviewable.