robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing)

Every few months this request lands in my inbox: "can we block the AI crawlers but stay in Google?" The answer is yes, and it takes about 20 lines of robots.txt — but only if you stop treating robots.txt as one switch.

This is the walkthrough I run on WordPress sites.

People conflate these constantly. They are different mechanisms with different failure modes.

robots.txt is a voluntary crawl request. It is not access control. If you need enforcement, that belongs at the server or CDN layer. Keep that in mind before you ship a Disallow and assume the problem is solved.

Before writing a single line, decide which bucket each token goes in. Write the decision down — future-you will not remember why PerplexityBot is allowed.

Bucket A — search index crawlers. Allow. Googlebot, Bingbot, DuckDuckBot, Applebot, YandexBot.

Bucket B — training corpora. Usually block. GPTBot, CCBot (Common Crawl), ClaudeBot, Google-Extended, Applebot-Extended, Bytespider.

Note that Google-Extended and Applebot-Extended are not crawlers. They're tokens that control whether already-crawled content is used for model training/grounding. Blocking them has no effect on how your pages rank or whether they're indexed — that's exactly why they're the safe ones to disallow.

Bucket C — AI answer surfaces. This is a business call. OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus user-initiated fetchers like ChatGPT-User and Claude-User.

These fetch pages so an assistant can cite them. Blocking them does not remove you from classic search results. It removes you from being quoted in answers. For a docs site that's a distribution channel; for a paywalled publisher it's a leak. Pick deliberately — the mistake is picking by accident because you copy-pasted a blocklist.

# ---- Search: allowed ---- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: YandexBot Allow: / # ---- AI training: blocked ---- User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / # ---- AI answer engines: allowed (flip to Disallow if you disagree) ---- User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # ---- Fallback for everything else ---- User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /*?replytocom= Disallow: /*?add-to-cart= Sitemap: https://example.com/wp-sitemap.xml Enter fullscreen mode Exit fullscreen mode 4. The four rules that actually bite A crawler obeys exactly one group. It matches the most specific User-agent line, and then reads only that group.

So the moment you add a Googlebot group, the Disallow: /wp-admin/ under User-agent: * stops applying to Googlebot. This is the single most common way people break their own site while "just adding AI blocking."

If you use specific groups, you have two options: duplicate the shared rules into every group, or leave the shared rules out entirely and keep the specific groups narrow. Don't mix and hope.

Disallow: with nothing after it means allow everything. It is not "block all". Disallow: / is block all.

robots.txt must return 200 at the root. A 404 is read by most crawlers as "no restrictions, go ahead". A 5xx tells crawlers something is broken and they may back off crawling. And never disallow robots.txt itself.

Blocking a URL does not remove it from the index. If a page is linked from somewhere crawlable, Disallow stops the content being read, not the URL being listed. To remove a page you need noindex — and noindex only works if the page is crawlable. Blocking and de-indexing are opposite operations; running both is how you get "why is this still in Google."

Wildcards ( *, $ ) work on the major search engines. Support elsewhere varies, so treat wildcard lines as best-effort cleanup, not as a security boundary.

Read the file back through a parser. Five seconds of Python saves a week of confusion:

import urllib.robotparser from urllib.parse import urljoin BASE = " https://example.com " rp = urllib. robotparser. RobotFileParser () rp. set_url ( urljoin ( BASE, " /robots.txt " )) rp. read () checks = [ ( " Googlebot ", " /blog/post-1/ " ), ( " Googlebot ", " /wp-admin/ " ), ( " GPTBot ", " /blog/post-1/ " ), ( " CCBot ", " / " ), ( " OAI-SearchBot ", " /blog/post-1/ " ), ( " SomeRandomBot ", " /blog/post-1/ " ), ] for ua, path in checks: allowed = rp. can_fetch ( ua, urljoin ( BASE, path )) print ( f " { ua: 15 } { path: 22 } { ' ALLOW ' if allowed else ' BLOCK ' } " ) Enter fullscreen mode Exit fullscreen mode Expected shape of the output with the file above: Googlebot /blog/post-1/ ALLOW, Googlebot /wp-admin/ ALLOW (because the specific group swallowed the * rules — that's the bug this test exists to catch), GPTBot BLOCK, OAI-SearchBot ALLOW.

One caveat: Python's robotparser doesn't implement * and $ wildcards. Use it for group-matching sanity and check wildcard lines by hand.

curl -sI https://example.com/robots.txt | head -5 # expect: HTTP/2 200, content-type: text/plain curl -s https://example.com/robots.txt | head -20 Enter fullscreen mode Exit fullscreen mode 6. The indexing safety checklist Run this every time you touch the file:

That last item is the one people skip. Do it on a real URL from each template, not the homepage.

Once the file is live, the interesting question changes from "did I write it right" to "is anyone honouring it." You can't answer that from the file; you answer it from request data, per user agent, over time.

On WordPress I use AumCrawl for that: it surfaces which AI crawlers are actually hitting the site and lets you allow or block each one individually at the server level — which is the part robots.txt genuinely cannot enforce, since a crawler that ignores the file never reads it in the first place.

So the loop is: declare intent in robots.txt, enforce it at the server, verify the result in the logs. All three, or you're just writing polite suggestions.