Your AI's knowledge base had 61 pages. 24 of them described a company that no longer exists.

A dental clinic asked me to look at their AI receptionist because it kept quoting a price that was "wrong." It wasn't wrong. It was two years old, and it was in a help-centre article that nobody had deleted.

I pulled their whole knowledge base: 61 markdown pages, last touched between 14 and 1,100 days ago. I checked every page against the live systems — the booking calendar, the price sheet, the staff roster, the insurance list.

24 of the 61 pages (39%) contained at least one statement that was false as of that morning.

Six named a dentist who had left. Five quoted a price that had changed. Four described a service the clinic had stopped offering. Three had opening hours that were correct pre-2024. Two listed an insurance provider they'd dropped. Four were duplicates of other pages with different answers to the same question.

The AI had a 0.94 retrieval confidence on all of them, because retrieval measures similarity, not truth. A page that says "£45 for a check-up" is a perfect match for "how much is a check-up." It doesn't know the price is now £62.

This is documentation rot, and in RAG systems it is the single largest source of confident wrong answers that no amount of prompt engineering fixes.

Most teams treat this as one problem — "we should update the docs." It isn't one problem, and each kind needs a different fix.

The clinic's 24 bad pages broke down as: 11 expired facts, 6 ghosts, 4 forks, 3 orphans. Only the first class is solved by "remind people to update the docs."

Ghost pages are the most dangerous. A discontinued service page doesn't just cause a wrong answer — it causes a wrong answer to a question the customer shouldn't have been able to ask. The AI offers a treatment the clinic cannot perform, and the customer books it.

I've now run this audit on seven knowledge bases: 61 to 840 pages, mostly 100–300. The result is consistent.

Note that last row, and note the fourth. The pages that get retrieved most are usually among the oldest, because they were written when the product was new and nobody has rewritten them since. Volume of retrieval and freshness are inversely correlated in almost every base I've opened.

Quarterly human review fails for a structural reason: you cannot review 200 pages by reading them, because "is this still true?" is not answerable from the page. It's answerable only from the system of record — the price sheet, the calendar, the CRM. A reviewer reading a doc can only judge whether it sounds current.

So the audit has to be built as a diff against the systems, not a reading exercise.

You cannot detect rot without a baseline. Add three fields to the front matter of every page, including the ones you're about to delete.

--- owner: revenue-ops # a team, never a person who can resign source_of_truth: pricing-sheet-v3 # where the facts come from verified_at: 2026-09-30 # last time a human or script confirmed it --- Enter fullscreen mode Exit fullscreen mode owner must be a team or a role. Every base I've audited with person-names as owners had 40%+ orphaned pages within a year, because people leave and nobody inherits the docs. source_of_truth is what makes automated checking possible. If a page's price claims can't be traced to a named table, they can't be verified, only believed. verified_at is not updated_at. Editing a typo is not verification. This distinction is the entire audit. Pages without these three fields are, by definition, unowned and unverifiable. Treat them as suspects, not as documentation.

This runs in about four seconds on 300 pages and produces the worklist. It's deliberately dumb: no LLM, no embeddings. It finds candidates, and a human confirms.

# rot_audit.py — python3 rot_audit.py./kb > rot_report.csv import os, re, sys, csv, datetime, collections ROOT = sys. argv [ 1 ] STALE_DAYS = 180 NUM = re. compile ( r ' (?:[$£€]\s?\d[\d,.]*|\b\d+(?:\.\d+)?\s?%|\b\d{1,2}:\d{2}\b) ' ) def front_matter ( text ): m = re. match ( r ' ^---\n(.*?)\n---\n ', text, re. S ) if not m: return {} out = {} for line in m. group ( 1 ). splitlines (): if ': ' in line: k, v = line. split ( ': ', 1 ) out [ k. strip ()] = v. strip (). strip ( '" \' ' ) return out rows, seen = [], collections. defaultdict ( list ) today = datetime. date. today () for dirpath, _, files in os. walk ( ROOT ): for f in files: if not f. endswith (( '.md ', '.mdx ' )): continue p = os. path. join ( dirpath, f ) text = open ( p, encoding = ' utf-8 ' ). read () fm = front_matter ( text ) body = re. sub ( r ' ^---\n.*?\n---\n ', '', text, flags = re. S ) nums = NUM. findall ( body ) verified = fm. get ( ' verified_at ' ) age = '' if verified: try: age = ( today - datetime. date. fromisoformat ( verified )). days except ValueError: age = ' BAD_DATE ' rows. append ({ ' file ': os. path. relpath ( p, ROOT ), ' owner ': fm. get ( ' owner ', '' ), ' source_of_truth ': fm. get ( ' source_of_truth ', '' ), ' verified_at ': verified or '', ' days_since_verified ': age, ' numeric_claims ': len ( nums ), ' stale ': ' YES ' if ( not verified or ( isinstance ( age, int ) and age > STALE_DAYS )) else '', ' risk_score ': 0, }) # fork detection: same title-ish heading, different numeric content head = ( fm. get ( ' title ' ) or body. strip (). splitlines ()[ 0: 1 ] or [ '' ])[ 0 ] key = re. sub ( r ' [^a-z0-9 ] ', '', str ( head ). lower ())[: 40 ]. strip () if key: seen [ key ]. append (( os. path. relpath ( p, ROOT ), tuple ( sorted ( nums )))) for r in rows: score = 0 if r [ ' stale ' ]: score += 2 if not r [ ' owner ' ]: score += 2 if not r [ ' source_of_truth ' ]: score += 2 score += min ( len ( re. findall ( r ' (?i)\b(currently|now|as of|starting|from january|this year)\b ', open ( os. path. join ( ROOT, r [ ' file ' ]), encoding = ' utf-8 ' ). read ())), 3 ) r [ ' risk_score ' ] = score for key, items in seen. items (): if len ( items ) > 1 and len ({ i [ 1 ] for i in items }) > 1: for f, _ in items: for r in rows: if r [ ' file ' ] == f: r [ ' risk_score ' ] += 3 rows. sort ( key = lambda r: - r [ ' risk_score ' ]) w = csv.

DictWriter ( sys. stdout, fieldnames = list ( rows [ 0 ]. keys ())) w. writeheader (); w. writerows ( rows ) Enter fullscreen mode Exit fullscreen mode What it flags:

The scanner finds suspicious pages. The probes find actual lies. Write twelve questions whose answers you can look up in a system — not opinions, not summaries. For the clinic:

1. What is the price of a standard check-up? 2. What is the price of a filling (composite)? 3. What are Saturday opening hours? 4. Which dentists are currently taking new patients? 5. Do you accept [insurer]? 6. How long is the wait for a hygiene appointment? 7. What is the cancellation window and fee? 8. Is [discontinued service] available? 9. What is the emergency out-of-hours number? 10. How much is a first consultation for a child? 11. Which locations are open on Sunday? 12. What is the deposit for a treatment plan over £1,000? Enter fullscreen mode Exit fullscreen mode Ask the deployed AI each question verbatim. Put its answer next to the system-of-record answer. Every mismatch is a rot event with a fingerprint: the answer the AI gave tells you which page produced it.

On the clinic, 12 probes returned 7 mismatches, mapping to 9 of the 24 bad pages. The other 15 pages had no probe coverage — which is itself the finding: pages that no probe touches are pages you have no reason to keep.

Run the probes weekly. It's fifteen minutes and it catches drift within days instead of at the first angry customer.

The instinct is to fix everything. Don't. Fix in this order, and use the row's class to pick the action.

The clinic's 24 pages took 3 hours 40 minutes total, most of it the six forks. Two weeks later, re-running the 12 probes gave 0 mismatches and the "wrong price" complaints stopped.

The reindex matters: after any deletion, force a re-embed. A deleted file that's still in your vector store is a ghost that survives the exorcism.

Month: ____________________ Total pages: ____ Pages verified this month: ____ (target: ≥25% of base, so full cycle ≤4 months) Pages with no owner: ____ (target: 0) Probe questions live: ____ (target: ≥12) Probe mismatches: ____ (target: 0) Median days since verified, top-20 retrieved: ____ (target: <90) Ghost pages deleted: ____ Forks merged: ____ Enter fullscreen mode Exit fullscreen mode That median row is the one to watch. If your twenty most-retrieved pages are older than 90 days, your AI is confidently describing a previous version of your business — and you'll find out from a customer, not from a dashboard.

[ ] Every page has owner, source_of_truth, verified_at [ ] No owner is a person's name [ ] Every page with numbers names where the numbers come from [ ] No page says "currently", "now", or "as of" without a date [ ] No two pages answer the same question with different answers [ ] No page describes a service that can't be bought today [ ] No page names a person who has left [ ] 12 probes defined, each answerable from a system of record [ ] Probes run weekly against the live agent, mismatches triaged in 48h [ ] Re-embed forced after every deletion Enter fullscreen mode Exit fullscreen mode None of this is prompt engineering. Prompt engineering changes how the model talks. Doc rot changes what it believes, and no instruction can make a model distrust a document that reads perfectly.

The clinic's fix wasn't a better prompt. It was deleting 24 files and adding three lines of front matter.

If you're deploying an agent on top of a knowledge base, the audit template, the scanner, and the 12-probe worksheet are in the kit I use: Agent Audit Pack