KI-generierte Dokumentation: Token-Gate soll falsche Zahlen verhindern

A docs PR landed in March with one changed sentence: the queue client's retry backoff was now "250 ms".

The code had moved to 800 ms two releases earlier. Reviewers read the sentence, found it grammatical, and approved it. Three weeks later a support thread quoted our own page back at us while debugging a timeout cascade.

Prose mistakes cost a reader a paragraph. Numeric mistakes cost them an afternoon.

Look at what a language model actually gets wrong in documentation. It rarely produces broken English. It produces plausible constants.

You can measure your own exposure before trusting anything in this post, including my premise:

rg -o --no-heading -N '\b\d+(?:\.\d+)?\s?(?:ms|s|MB|GB|%|x)\b|\bv?\d+\.\d+(?:\.\d+)?\b' docs/ \ | wc -l Enter fullscreen mode Exit fullscreen mode If that number is in the hundreds, human review of prose is not a control. It is a hope.

The working rule I use is narrow. A model may arrange language. A human must own every string a reader could copy into a terminal, a config file, or a capacity plan.

The important part is not the table. It is that the model is not asked to remember a number — it is asked to reference one.

facts.json is the only file a human signs for numeric truth. Each entry carries where it came from and when someone last checked.

{ "facts": { "queue.backoff_ms": { "value": 800, "unit": "ms", "source": "src/queue/retry.ts:41", "verified_at": "2026-10-24", "owner": "runtime-team" }, "api.default_timeout_s": { "value": 30, "unit": "s", "source": "config/defaults.yaml:12", "verified_at": "2026-10-24", "owner": "platform" }, "release.current": { "value": "2.7.0", "source": "package.json:3", "verified_at": "2026-10-24", "owner": "release" } } } Enter fullscreen mode Exit fullscreen mode 2. Let the model draft only slots The prompt contract is short: write Markdown, never write a number, version, or URL directly — emit a token instead. Unresolved needs become a request, not a guess.

The client retries with a backoff of {{fact:queue.backoff_ms}} by default. Requests time out after {{fact:api.default_timeout_s}} unless overridden. <!-- fact-request: queue.retry_jitter_ms — model could not find a source --> Enter fullscreen mode Exit fullscreen mode That trailing comment is the useful failure. The model admitting it does not know is worth more than a confident 50 ms.

claimcheck.py is stdlib-only and runs in two stages. Stage one scans the raw draft; stage two scans the rendered page and confirms every surviving claim traces back to a registered value.

#!/usr/bin/env python3 """ claimcheck.py - only registered claims may appear in AI-drafted docs. python claimcheck.py docs/queue-retry.md --facts facts.json --stage draft python claimcheck.py docs/queue-retry.md --facts facts.json --stage rendered """ from __future__ import annotations import json import re import sys from pathlib import Path FENCE = re. compile ( r " ^```.*?^ ``` ", re. S | re. M ) TOKEN = re. compile ( r " \{\{fact:([a-z0-9_.-]+)\}\} " ) CLAIM = re. compile ( r " \b\d+(?:[.,]\d+)?\s?(?:ms|s|sec|secs|seconds?|min|minutes?|h|hours?

" r " |MB|GB|KB|TB|%|x|rps|req/s|tokens?)\b " r " |\bv?\d+\.\d+(?:\.\d+)?\b " r " |https?://[^\s)\]>\" ' ]+ ", re. IGNORECASE, ) def load_facts ( path: str ) -> dict: return json. loads ( Path ( path ). read_text ())[ " facts " ] def as_text ( fact: dict ) -> str: unit = fact. get ( " unit " ) return f " { fact [ ' value ' ] } { unit } ". strip () if unit else str ( fact [ " value " ]) def strip_examples ( md: str ) -> str: # Fenced blocks are verified by executing them, not by this gate. return FENCE. sub ( "", md ) def check_draft ( md: str, facts: dict ) -> list [ str ]: problems = [] for key in sorted ( set ( TOKEN. findall ( md ))): if key not in facts: problems. append ( f " unknown fact key { key!r} " ) body = TOKEN. sub ( "", md ) for m in CLAIM. finditer ( body ): line = body. count ( " \n ", 0, m. start ()) + 1 problems. append ( f " draft line { line }: naked claim { m. group ( 0 )!r} " ) return problems def render ( md: str, facts: dict ) -> str: return TOKEN. sub ( lambda m: as_text ( facts [ m. group ( 1 )]), md ) def check_rendered ( rendered: str, facts: dict ) -> list [ str ]: allowed = [ as_text ( f ). replace ( " ", "" ) for f in facts. values ()] problems = [] for m in CLAIM. finditer ( rendered ): hit = m. group ( 0 ). replace ( " ", "" ) if not any ( hit. startswith ( a ) for a in allowed ): problems. append ( f " rendered claim { m. group ( 0 )!r} matches no fact " ) return problems def main ( argv: list [ str ]) -> int: try: md_path, facts_path, stage = argv [ 1 ], argv [ 3 ], argv [ 5 ] facts = load_facts ( facts_path ) md = strip_examples ( Path ( md_path ). read_text ()) if stage == " draft ": problems = check_draft ( md, facts ) else: problems = check_rendered ( render ( md, facts ), facts ) except ( IndexError, KeyError, OSError, json. JSONDecodeError ) as exc: print ( f " input error: { exc } ", file = sys. stderr ) return 2 for p in problems: print ( p ) print ( f " { stage }: { len ( problems ) } problem(s), { len ( facts ) } fact(s) " ) return 1 if problems else 0 if __name__ == " __main__ ": sys. exit ( main ( sys. argv )) Enter fullscreen mode Exit fullscreen mode The first run on the draft that started this post looks like this (illustrative output from a synthetic three-page docs set, not a benchmark):

$ python claimcheck.py docs/queue-retry.md --facts facts.json --stage draft draft line 14: naked claim '250 ms' draft line 21: naked claim 'https://internal.example/runbooks/queue' draft: 2 problem(s), 6 fact(s) $ echo $? 1 Enter fullscreen mode Exit fullscreen mode Two lines back to the model, one of them a token, and the stale number is gone — not corrected, removed from the model's authority.

The gate blocks bad pages. It does not tell a reviewer whether 800 ms is still 800 ms. Print that before approval:

python - << ' PY ' import json, re, pathlib facts = json.loads(pathlib.Path("facts.json").read_text())["facts"] md = pathlib.Path("docs/queue-retry.md").read_text() for key in sorted(set(re.findall(r" \{\{ fact:([a-z0-9_.-]+) \}\} ", md))): f = facts[key] print(f"{key:24} {str(f['value']):>8} {f['source']:<28} {f['verified_at']} {f['owner']}") PY Enter fullscreen mode Exit fullscreen mode api.default_timeout_s 30 config/defaults.yaml:12 2026-10-24 platform queue.backoff_ms 800 src/queue/retry.ts:41 2026-10-24 runtime-team release.current 2.7.0 package.json:3 2026-10-24 release Enter fullscreen mode Exit fullscreen mode Now the approval question is answerable in one line: "did the runtime team sign a change to backoff_ms?" If not, the page is wrong or the fact file is stale, and either way it is not a merge.

This is why the split matters more than the script. The script only enforces a boundary a human already agreed to.

The draft stage above is a bounded generation job: fixed template, prose only, tokens required, fact-request list as output. That shape is exactly what you can afford to run on every docs PR rather than on a quarterly cleanup. MonkeyCode's free model access makes that economics trivial for a side project or an unfunded OSS repo. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A weak model is not a blocker here, because its worst habit — inventing constants — is precisely what stage one rejects. What you get instead is a list of unknown fact key failures, each one a question for the owning team, and a page with fewer confident sentences.

The free server option matters for the scheduled half of the workflow. A nightly job drafts one page from a template, runs --stage draft, and opens a PR only if the problem count is zero. That job needs a stable-ish place to run more than it needs CPU. Two caveats before you build that: treat free capacity as a convenience, not an SLA, and never point it at a private repo whose contents you cannot export. Check the current limits on the project's own pages rather than trusting any number in an article — including this one.

If a reader can copy it into a terminal or paste it into an incident doc, a human signs it — the model gets to write the sentence around it.

If you want to try the draft stage without a budget conversation, MonkeyCode's free model access and free server option are a reasonable place to start; the fact registry, the two-stage gate, and the reviewer checklist stay yours either way.