Say no to AI training, not to AI search: robots.txt per bot

Posted on Oct 20 • Originally published at richi.codes

This note has an interactive demo. Try it in the original

"Block the AI bots" sounds like one switch, but it isn't. Anthropic uses three different crawlers, one for training, one for search and one for requests a person makes in Claude, and robots.txt can treat each one differently. That means you can keep your content out of training without disappearing from the answers. There's one rule in how robots.txt groups work that can quietly undo the whole setup, and the demo above opens right on it.

Anthropic's help article on crawling lists them with what happens when you disable each one:

So if what you want is "don't train on my site, but do cite it", the file is short:

# Training: no. Search and user requests: yes. User - agent: ClaudeBot Disallow: / User - agent: * Disallow: / private / Enter fullscreen mode Exit fullscreen mode That's the "No training, yes search" example in the demo. ClaudeBot has its own group and gets nothing. Claude-SearchBot and Claude-User aren't named, so they fall back to the * group and can read everything except /private/.

The same article has a few practical notes. The file has to go in the top-level directory of every subdomain you want to opt out. Blocking Anthropic's IP addresses instead "may not work correctly or persistently guarantee an opt-out", because it stops them from reading your robots.txt. They also support the non-standard Crawl-delay directive.

Now suppose you want Claude-SearchBot to focus on your blog and you add a small group for it. That's the "The * trap" example:

User -agent: * Disallow: /private/ User -agent: ClaudeBot Disallow: / # Looks harmless... User -agent: Claude-SearchBot Allow: /blog/ Enter fullscreen mode Exit fullscreen mode Look at the /private/x column in the demo. Claude-SearchBot is now allowed there, and the cell gets an orange * badge. The reason is that a crawler follows one group only: the one that names it. If such a group exists, the * group is ignored completely, not merged in. Its only rule says Allow: /blog/, nothing in it matches /private/x, and a path that no rule matches is allowed.

This isn't a quirk of my demo. Google's open-source robots.txt parser, which its README describes as the production code Googlebot uses, has a test for exactly this: with Disallow: /x/ under * and Disallow: /y/ under FooBot, FooBot is allowed into /x/page. In the code, once a group for the agent has been seen, the matcher never looks at the global rules again.

The fix is boring: repeat in each named group the * rules that should still apply ("Trap fixed" in the demo).

User -agent: Claude-SearchBot Allow: /blog/ Disallow: /private/ Enter fullscreen mode Exit fullscreen mode How a rule wins Inside a group, order doesn't matter. Every Allow and Disallow whose pattern matches the path is a candidate, and the longest pattern wins. If an Allow and a Disallow are the same length, Allow wins. Patterns are anchored at the start of the path, * matches any sequence of characters, and $ only means "end of the URL" when it's the last character.

This is the matching function the demo uses, ported line by line from Google's robots.cc. It keeps a list of positions in the path that can still match the pattern read so far: