I checked what 31 big websites tell AI crawlers in robots.txt, and found a bug in my own tool

Can ChatGPT, Claude or Perplexity read your website? The first place to look is a small text file: robots.txt.

I wanted to see what big websites actually say in it, so I picked 40 well-known sites and audited their home pages for 12 AI crawlers. The results split much more cleanly by industry than I expected.

The study also found a bug in my own tool. More on that below, because it's a mistake you could easily make too.

On 5 November 2026 I ran PageSignal, the AI visibility API I built, against the home pages of 40 sites: news, e-commerce, SaaS and developer tools, and reference sites.

I split the crawlers into two groups, because they do different jobs:

The 10 sites that block at least one search or assistant bot were The Guardian, BBC, Reuters, The Verge, TechCrunch, Wired, Amazon, eBay, Reddit and IMDb.

Reuters, Reddit and IMDb are a special case. Their general User-agent: * rules close the home page to any bot without its own rules, so this isn't always AI-specific. Reddit's file ends like this:

User - agent: * Disallow: / Enter fullscreen mode Exit fullscreen mode The pattern: publishers close, software companies open All 6 news sites block at least one bot that fetches pages to answer people. None of the 10 SaaS and developer tool companies do, and 9 of those 10 publish an llms.txt file, while no news site does.

My guess, and it's only a guess, is that this comes down to the business model. A publisher sells its content, so an AI answer can replace the visit. A software company sells a product, so being mentioned in an AI answer works like free marketing.

Neither choice is automatically right. Blocking is often deliberate. The thing to avoid is blocking by accident, for example with a security rule or a template nobody remembers adding.

My first run said Figma was blocking AI search bots. I double-checked before writing that here, and it was wrong.

PageSignal was using Python's built-in urllib.robotparser. It's convenient, but I found two behaviours that don't match the robots.txt standard, RFC 9309:

1. It doesn't understand $, the end-of-URL marker.

from urllib.robotparser import RobotFileParser parser = RobotFileParser () parser. parse ( """ User-agent: OAI-SearchBot Allow: /$ Disallow: / """. splitlines ()) print ( parser. can_fetch ( " OAI-SearchBot ", " https://example.com/ " )) # False, should be True Enter fullscreen mode Exit fullscreen mode Under the standard, the most specific matching rule wins, so Allow: /$ opens the home page. That's how Figma's file is written.

2. It can apply the wrong group when one bot name is contained in another.

parser = RobotFileParser () parser. parse ( """ User-agent: Applebot Allow: / User-agent: Applebot-Extended Disallow: / """. splitlines ()) print ( parser. can_fetch ( " Applebot-Extended ", " https://example.com/ " )) # True, should be False Enter fullscreen mode Exit fullscreen mode The rule for Applebot gets applied to Applebot-Extended. This made my tool miss a real block on The Verge, Reuters and IMDb.

In total, 4 of the 32 sites audited in that first run had wrong crawler results. I replaced the parser with a small RFC 9309 matcher and checked the numbers in this post against a separate implementation. They match.

If you read robots.txt with urllib.robotparser (tested on Python 3.12), it's worth testing these two cases.

Look at yoursite.com/robots.txt and search for the bot names above, then look at the User-agent: * block. Check the pages you care about, not only the home page.

If you'd rather not do it by hand, there's a free live demo with no signup. For automated checks, the API is on RapidAPI and API.market.

Did you decide on purpose what your robots.txt says about AI crawlers, or is it just whatever your template shipped with? I'd love to hear what you found.