A Robots.txt Matrix for AI Crawlers: Allow, Delay or Block by Bot

Different AI crawlers honour robots.txt in different ways, and a single Disallow: / aimed at one vendor can accidentally silence another that your GEO programme cares about. A small matrix in version control beats tribal knowledge in Slack.

From two weeks of access logs, list user-agents that look like AI or training crawlers. Common ones change over time, so do not copy a blog post from last year blindly. Note path volume and status codes for each.

For each bot, pick one policy: allow all public content, allow only /blog and docs, crawl-delay if supported, or block. Training crawlers and live-answer crawlers are different products; treating them the same is a business decision, not a default.

# Policy review: 2026-11-01 # GPTBot: allow public docs + blog only User-agent: GPTBot Allow: /blog/ Allow: /docs/ Disallow: / Enter fullscreen mode Exit fullscreen mode Comments with a review date stop the file becoming archaeology.

Rules at the edge can override origin robots.txt. After a Cloudflare or similar change, curl the live robots.txt and hit a sample of URLs with the relevant user-agent. Log the results next to the matrix.

New bots appear. Old ones rebrand. Put a calendar reminder. The matrix is only useful if someone owns it.

Before you ship the change, walk through a short checklist with someone who did not write the page. Ask them to explain the offer, the price or fee range, the timeline and the main risk in their own words. If they cannot, the draft is not ready for assistants or for humans either.

Document the owner of the page, the next review date and the source of every number you quote. When a platform policy or a fee changes, you want a single place to update rather than a scavenger hunt across blog posts, help articles and sales decks.

Teams usually fail in three ways. They publish once and never revisit dates. They keep conflicting facts on the About page, a directory and a PDF one-pager. They measure vanity traffic instead of whether the answers that matter improved. A lightweight monthly review catches all three earlier than a panicked quarterly audit.

Overseas ads, multilingual sites, GEO and crypto checkout for overseas merchants share the same underlying need: clear facts, consistent entities and pages that can be quoted without guesswork. The implementation details differ by channel; the editorial discipline does not.

How crawler access ties into query fan-out and citation odds is covered in how query fan-out works in AI search.

Published by PandaWM, an overseas marketing team based in Hong Kong.