Which AI Crawlers Do the Top 1,000 Websites Block? (robots.txt Study, October 2026)

Posted on Oct 24 • Originally published at seopeck.com

We read the robots.txt files of the 1,000 most popular websites: 20.9% block at least one of 16 AI bots. Per-bot shares, site strategies, common mistakes and open data.

On 12 October 2026 we fetched the robots.txt file of each of the 1,000 most popular websites and checked 16 AI crawlers against it. 209 of the 1,000 sites (20.9%) block at least one AI bot from the whole site; 789 (78.9%) let every one of them in. Training crawlers are blocked most often (CCBot on 15.5% of sites, GPTBot on 13.8%), bots that fetch pages for AI search less often (OAI-SearchBot on 7.0%).

Below are the five main findings, the bot-by-bot table, the strategies sites use, the mistakes we found and what it means for your own robots.txt. The per-site data and the method are open, so you can check any number.

We walked down the Tranco research ranking (list 94GG2) and kept the first 1,000 domains that serve a real website, which took 2,364 domains: the raw list is full of infrastructure domains with no site behind them. For each site we read one robots.txt file with our own research bot, SEOpeckBot, and parsed it under RFC 9309, the robots.txt standard, using the same code as our AI Crawler Checker.

"Blocks a bot" in this study means the bot may not fetch the home page ( / ) while Googlebot still may. A site that only keeps a bot out of some folders is not counted as blocking. 168 sites refused our bot outright and are left out; the full method and its limits are at the end.

Most AI crawler studies split bots into "training" and "search". There is a third kind, and site owners treat each one differently:

Websites blocking at least one AI bot of each kind

Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

Almost every site that blocks AI bots blocks training: only 3 of the 209 block a search or user-triggered bot without blocking any training crawler. 48 sites block all eight training crawlers.

The table shows, for each of the 16 AI bots, the share of the 1,000 websites that block it from the whole site, and the share whose robots.txt names it at all.

Share of the 1,000 websites that block each AI bot site-wide

Google-Extended and Applebot-Extended are control tokens, not separate crawlers. Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

Two things stand out. First, naming a bot is not the same as blocking it: 195 sites name GPTBot, but 74 of them still let it fetch the home page, usually because they keep it out of some folders only. Second, Bytespider, Meta-ExternalAgent and the two Amazon AI bots are blocked by more sites than name them. That comes from "allowlist" files, explained in the next section.

For comparison, the search engines we used as controls are almost never blocked: Googlebot on 2 sites (0.2%) and Bingbot on 7 (0.7%).

Counting bots one by one hides how sites actually decide. So we put each site into one strategy, based on which kinds of AI bot it blocks from the home page.

How the 1,000 websites treat AI bots (one strategy per site)

Read the second row carefully: blocking at least one bot of each kind is not the same as blocking every AI bot. Of those 110 sites, only 20 block all 16.

About 3% of sites, 34 in all, run an allowlist: a User-agent: * group with Disallow: /, plus groups that let named search engines in. Every crawler the file does not name, AI bots included, is shut out. We count these sites as blocking, because that is what the file tells an AI bot. 33 of the 110 sites in the second row work this way, and so do 16 of the 20 sites that block all 16 bots. pinterest.com is one example: none of the 16 AI bots is named, and all of them are blocked through the * group. Other sites list the bots one by one: bbc.com blocks 14 of the 16 by name and cnn.com 15.

The 78 "training only" sites are the clearest group, but the pattern runs wider. Of the 138 sites that block GPTBot, 70 (50.7%) still let OAI-SearchBot fetch the home page. Of the 131 that block ClaudeBot, 58 (44.3%) still allow Claude-SearchBot. Perplexity has no separate training crawler, and 105 sites (10.5%) block PerplexityBot.

forbes.com is one example of the approach: its file blocks GPTBot and ClaudeBot and has groups that allow OAI-SearchBot, Claude-SearchBot and Claude-User. The logic is simple: a training crawler takes content without sending visitors, while an AI search bot can cite the page and link to it. Here is a minimal version of that setup, checked with our parser:

# Block AI training User -agent: GPTBot User -agent: ClaudeBot User -agent: CCBot User -agent: Google-Extended User -agent: Applebot-Extended User -agent: Meta-ExternalAgent User -agent: Bytespider User -agent: Amazonbot Disallow: / # Everyone else, including AI search and user-triggered fetchers User -agent: * Disallow: /admin/ Enter fullscreen mode Exit fullscreen mode Bigger sites block more The share of sites blocking AI bots falls as you go down the ranking. "Rank" here is a site's place among the 1,000 websites in the study, not its raw Tranco rank.

Rank is the place among the 1,000 websites in the study. Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

robots.txt files collect old lines. These are the patterns we counted, described as neutrally as we can: some are deliberate choices we cannot see the reason for.

109 sites still list anthropic-ai and 94 list Claude-Web, names Anthropic no longer documents; its current bots are ClaudeBot, Claude-SearchBot and Claude-User. Other agent names outside our 16 are common too: cohere-ai appears on 115 sites, Diffbot on 106 and YouBot on 94.

Old names do little harm next to current ones, and Anthropic has said ClaudeBot honours rules written for its retired names. Only 7 sites rely on the legacy names alone, with no ClaudeBot line. We call these legacy-only rules rather than broken ones, but a parser that follows RFC 9309 to the letter would not apply them to ClaudeBot. If you want to block Anthropic's crawler, name ClaudeBot.

Blocking GPTBot while allowing OAI-SearchBot is a clear choice. The reverse pairing for user-triggered bots is harder to explain: 22 sites block ChatGPT-User but allow OAI-SearchBot, 7 do the same with Claude-User and Claude-SearchBot, and 7 with Perplexity-User and PerplexityBot. The search bot may index the page, but the assistant cannot open it when a person asks about it. Worth a second look if that is your file.

9 sites have lines that RFC 9309 does not recognise: 5 use a versioned name such as User-agent: GPTBot/1.0, and 4 have Allow or Disallow rules before any User-agent line, so the rules belong to no group. Some crawlers are forgiving about these, but a strict parser, ours included, skips them. Write the bare product token: User-agent: GPTBot.

49 sites (4.9%) answer /robots.txt with an HTML page, usually their home page or an error page sent with a 200 status. Under RFC 9309 that is not a valid robots.txt file, so a crawler finds no rules in it and treats the whole site as open. If you think you have AI rules, check that the URL returns plain text.

The comparison is directional only: samples and methods differ. In 2023, Originality.ai counted 242 of the top 1,000 sites blocking GPTBot within weeks of its launch; we count 138 of 1,000 websites. Cloudflare's 2025 analysis of about 3,900 robots.txt files from the top 10,000 domains found GPTBot was the most disallowed bot; in our data it ranks third, after CCBot and Bytespider. Chris Humphrey's June 2026 study of the Majestic top 10,000 found 22.6% blocking at least one of 10 bots and 4% blocking all ten; we find 20.9% blocking at least one of 16 and 2.0% blocking all 16.

The list is Tranco list 94GG2, generated on 2 October 2026 from 30 days of data (3 September to 2 October 2026). We checked domains in rank order and kept a domain as a website if its home page answered with a 2xx HTML page on the same registrable domain, stopping at 1,000 websites. That took 2,364 domains. The 1,364 skipped domains break down as: no connection 694, robots.txt or home page redirecting to another domain 221, home page not answering 2xx 189, robots.txt refused (401/403) 168, home page disallowed for our bot and not open to Googlebot 37, home page not connecting 23, robots.txt server error 12, home page not HTML 9, timeout 9, rate-limited (429) 2.

Near-duplicates are kept as Tranco lists them (for example bbc.com and bbc.co.uk, or x.com and twitter.com), so the 1,000 websites are not 1,000 separate organisations.

The crawl ran on 12 October 2026 from our own machine, not a server. SEOpeckBot fetched /robots.txt first (up to 5 redirects; if there was no valid file, once more on the www. or bare version of the domain, which found the file for 72 sites), then the home page only if robots.txt allowed it, then /llms.txt only if allowed. 35 sites disallow our bot's home page through their * group but let Googlebot in; we counted them as websites without fetching the page. The bot sent at most 12 requests at once, waited at least 2 seconds between requests to the same host, gave up after 10 seconds without a connection or 20 seconds in total, and read at most 500 KiB of a robots.txt file. Its user agent was:

SEOpeckBot/1.0 (robots.txt study; +https://seopeck.com/page/robots-txt-study) Enter fullscreen mode Exit fullscreen mode The linked page explains the study and how to opt out. At the time of the crawl, no site had opted out or asked to be removed.

Of the 1,000 websites, 890 had a valid robots.txt, 59 had none (404 or 410), 49 served an HTML page and 2 gave another 4xx answer. Under RFC 9309 a missing or invalid file means no restrictions, so all 1,000 are in the denominator. The 168 sites that refused our bot (401 or 403) are not: a refusal says nothing about their rules. Counted as open sites instead, the headline figure would be 17.9% (209 of 1,168) rather than 20.9%.

Each file was parsed with the same code as the AI Crawler Checker, following RFC 9309: a bot obeys the group or groups that name it (case-insensitive), otherwise the * group; within a group the longest matching rule wins, and Allow wins a tie. A bot counts as blocked when / is disallowed for it. AI blocks are counted only where Googlebot may still fetch /; the 2 sites that block Googlebot as well form their own strategy. We classify Amazonbot as a training crawler, as our checker does. Legacy names such as anthropic-ai are counted when a file names them, not evaluated as live crawlers. After the crawl we re-fetched 25 random sites and re-parsed them; every result matched.

Rankings: Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński and Wouter Joosen. 2019. "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." In Proceedings of the 26th Annual Network and Distributed System Security Symposium (NDSS 2019). https://doi.org/10.14722/ndss.2019.23386. List used: Tranco list 94GG2, generated on 2 October 2026, available at https://tranco-list.eu/list/94GG2.

You are welcome to use the data and charts in your own work; please credit "SEOpeck robots.txt study, October 2026" with a link to this page. We plan to repeat the study every quarter and publish what changed.

APA: Akram, N. (2026, October 20). Which AI crawlers do the top 1,000 websites block? (robots.txt study, October 2026). SEOpeck. https://seopeck.com/blog/ai-crawlers-robots-txt-top-1000-websites

Link: SEOpeck robots.txt study, October 2026