AI Crawlers List: Training vs. Search User Agents

Posted on Nov 3 • Originally published at sitecheckly.com

AI crawlers serve different purposes. Blocking GPTBot targets training access, while blocking OAI-SearchBot affects ChatGPT search. The same distinction applies to ClaudeBot and Claude-SearchBot. Google-Extended controls Gemini training, not Google Search or AI Overviews. Choose robots.txt rules by purpose rather than blocking every AI-related user agent.

In the survey, 9.2% of sites blocked training crawlers while still allowing AI search crawlers. That is a useful distinction if you want to limit training access without closing off search discovery.

The November 2026 Sitecheckly survey covered the Tranco top 10,000, using Tranco list 7462X. We could analyze 4,934 of 10,000 domains. The rest were mostly CDN, DNS and API hosts, sites that ask crawlers to stay away in robots.txt, or sites that blocked automated requests. The training-versus-search finding uses the 4,546 sites with a readable robots.txt. See the survey methodology.

Treat your crawler policy as separate decisions:

A training opt-out is not the same as a search opt-out. Start with the outcome you want, then choose the matching user-agent groups.

This AI crawlers list separates training, search and user-requested fetching. Google-Extended is included because it belongs in the policy decision, even though it is a control token rather than a separate fetching crawler.

The allow and block columns describe the whole-site patterns shown below. They are alternatives, not instructions to apply both.

For OAI-SearchBot vs GPTBot, the key difference is search versus training. ChatGPT-User is separate from both: it fetches content in response to a user's request. OpenAI notes that robots.txt rules may not apply to these user-initiated requests, so a ChatGPT-User block is not a guaranteed access restriction. See OpenAI's bot documentation.

For Anthropic, use the distinct tokens for distinct decisions: ClaudeBot is for training, while Claude-SearchBot supports search.

PerplexityBot is Perplexity's search crawler. Do not classify it as a training crawler just because it is associated with an AI service.

Use a user-agent group to name the crawler, then add the path rule. In these examples, the root path covers the whole site under that robots.txt policy. The group and path matching rules come from RFC 9309.

To block AI training access through GPTBot:

User-agent: GPTBot Disallow: / Enter fullscreen mode Exit fullscreen mode To allow AI search access through OAI-SearchBot:

User-agent: OAI-SearchBot Allow: / Enter fullscreen mode Exit fullscreen mode These groups can coexist. Blocking GPTBot does not require blocking OAI-SearchBot.

Each group below expresses an allow-all policy for its named token. Select the groups that match your intended policy. The ChatGPT-User caveat above still applies.

User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Enter fullscreen mode Exit fullscreen mode Block patterns for the listed tokens These are the equivalent disallow-all groups. This is a reference, not a recommended file to paste wholesale. Blocking every listed search crawler would work against a policy intended to preserve AI search access.

User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / Enter fullscreen mode Exit fullscreen mode For a training-blocked, search-allowed policy, select the block groups for GPTBot, ClaudeBot and Google-Extended. Select the allow groups for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Make the user-requested access decision separately.

Do not assume that the last line in the file wins. Under RFC 9309's matching rules:

For example, this group blocks GPTBot from the site except for the more specific public path:

User-agent: GPTBot Disallow: / Allow: /public/ Enter fullscreen mode Exit fullscreen mode An allow rule is permission under the robots.txt policy. It is not a promise that a service will crawl, use or cite the page. Keep access eligibility separate from the goal of earning a citation.

No. To opt out of Gemini training through this control, use:

User-agent: Google-Extended Disallow: / Enter fullscreen mode Exit fullscreen mode Google-Extended is a robots.txt control token. It is not a separate crawler making requests under that name. Blocking it does not block Google Search or AI Overviews; Googlebot is the relevant Search crawler. See Google's crawler documentation.

Do not substitute a Googlebot block when your intended decision is only about Gemini training. Those controls address different purposes.

Start with the published robots.txt, not just the copy in your editor. Compare its user-agent groups with the policy you meant to publish.

Then run the free, no-sign-up Sitecheckly AEO audit on the page URL. Each run audits one page: the URL you enter, not the whole site.

For context beyond crawler access, read the answer engine optimization explainer. Use it to separate the access question from the broader work of making a page useful in search answers.

Is ChatGPT-User the same as OAI-SearchBot?

No. ChatGPT-User fetches content in response to a user's request; OAI-SearchBot supports ChatGPT search. See OpenAI's bot documentation.

Does allowing an AI search crawler guarantee a citation?

No. An allow rule permits crawling under that robots.txt policy; it does not guarantee that a service will crawl, use or cite the page.

What happens if two robots.txt path rules conflict?

The most specific matching path wins. If equivalent allow and disallow rules match, the protocol says the allow rule should be used. See RFC 9309.

Want to see how your own site does? Run a free audit at sitecheckly.com.

Figures in this article come from Sitecheckly's own audit data; see the methodology.