How AI crawlers read your robots.txt (and why your * rules may not apply)
Most robots.txt files were written for Googlebot and Bingbot. Since then, AI companies have added several new crawlers, and they do different jobs. If you have never looked at how your file treats them, you may be blocking AI search by accident, or letting training crawlers in without ever deciding
Most robots.txt files were written for Googlebot and Bingbot. Since then, AI companies have added several new crawlers, and they do different jobs. If you have never looked at how your file treats them, you may be blocking AI search by accident, or letting training crawlers in without ever deciding to.
This post covers three things that trip people up: which AI bots exist, how a crawler picks the rules it follows, and why a well-meant User-agent: * block often does nothing for them.
It helps to sort AI user agents by what they do, not by company:
Many site owners want the first two (they can send visitors) and want to opt out of the third. That is a business decision, but you can only make it if your rules name the right bots. Vendors also change how their bots behave, so check each one's own documentation.
RFC 9309 (the robots.txt standard) and Google's documentation describe the same basic logic:
Point 2 is the one that catches people out.
User - agent: * Disallow: / admin / Disallow: / cart / User - agent: GPTBot Disallow: / User - agent: OAI - SearchBot Allow: / Enter fullscreen mode Exit fullscreen mode It looks safe, but OAI-SearchBot now has its own group, so it ignores the * group completely. Your /admin/ and /cart/ rules no longer apply to it. The fix is to repeat the shared rules inside every named group:
User - agent: OAI - SearchBot User - agent: Claude - SearchBot User - agent: PerplexityBot User - agent: ChatGPT - User Allow: / Disallow: / admin / Disallow: / cart / User - agent: GPTBot User - agent: ClaudeBot User - agent: CCBot Disallow: / User - agent: * Disallow: / admin / Disallow: / cart / Enter fullscreen mode Exit fullscreen mode Several User-agent lines can share one group, which keeps the file short.
You can test this by hand against each user agent, but it gets tedious with twenty-plus bots. We built a free checker that does it for 22 AI crawlers, for a live site or a pasted draft file, on any path. It also looks for an llms.txt file and writes a block like the one above that keeps your existing * rules (I run 313 Automations):
https://313automations.com/free-tools/ai-crawler-access-checker
If you spot a bot it should cover, tell me in the comments.