Which AI crawlers do 261 top websites block? Study data
We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end. The headline numbers 59 of the 261 sites (22.6%) disallow GPTBot, the
Posted on Oct 15 • Originally published at airanked.ai
We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end.
The sample includes 48 news sites, and 27 of them block GPTBot. That is nearly half of all GPTBot blockers in the study coming from a single category. Publishers have clear reasons to keep training crawlers out, and the numbers show they act on it far more than retailers, software companies or reference sites do.
The gap between 35% and 21% matters. Many sites that shut out training bots still let search bots in, because being fetched for a live answer in ChatGPT, Perplexity or Google AI Overviews can send visitors, while training access does not. OAI-SearchBot being the least blocked agent fits that pattern. Bytespider sits at the other end: it is blocked almost as often as CCBot even though it is far less known, likely because it has a reputation for ignoring rules.
The CSV with every site, every crawler and every rule is open on GitHub: ai-crawler-access-study
The full write up, including method and per category breakdowns, is here: AI crawler access report
If you want to check your own robots.txt against these crawlers, AI Ranked has a free in-browser robots.txt checker.