Is that really GPTBot? What a week of server logs says about AI crawler claims
I maintain a small CLI called crawlerpeek that checks whether AI crawlers can reach a site. While building it I wanted to know how much of the "bot traffic" in my own logs is what it says it is. This post has the numbers, what they do and do not show, and then how to run the same kind of check on y
Posted on Oct 25 • Originally published at crawlerpeek.com
I maintain a small CLI called crawlerpeek that checks whether AI crawlers can reach a site. While building it I wanted to know how much of the "bot traffic" in my own logs is what it says it is. This post has the numbers, what they do and do not show, and then how to run the same kind of check on your site.
Period: 19 to 25 October 2026. Four new, small sites that I operate. For every request whose user agent contains a bot token, I checked whether the client IP lies inside the IP list that the operator publishes (Google, OpenAI, Anthropic, Perplexity, Bing, Apple, Common Crawl, Ahrefs), fetched the same day. I removed my own server's requests and my own manual tests.
There were 332 requests that did not match. 287 of them came from a single hosting network, and that network claimed eight different bots: Bingbot, CCBot, ChatGPT-User, ClaudeBot, Googlebot, GPTBot, OAI-SearchBot and PerplexityBot. Its reverse DNS did not confirm forward. A further 45 had no reverse DNS at all.
The practical point: a user-agent string is a claim, not an identity. If you decide anything based on it (allow, block, rate limit, count it in analytics), the operator's IP list or reverse DNS is the check that matters. Google and others document reverse-DNS verification, and OpenAI, Anthropic, Perplexity, Bing, Apple and Common Crawl publish JSON lists of ranges.
Separately from spoofing, there is a more common problem: robots.txt says one thing and the server does another. A WAF rule returns 403 to a user agent that robots.txt allows. A redirect sends a bot to a login page. A User-agent: * group blocks a path, and a bot-specific group further down quietly overrides it (a bot with its own group ignores the * group).
crawlerpeek checks both layers for each bot: the robots.txt decision under RFC 9309 matching rules, and what the page returns when fetched with that bot's user agent.
npx crawlerpeek check https://example.com Enter fullscreen mode Exit fullscreen mode Example output for a site with a WAF rule and a restrictive robots.txt:
https://example.com robots.txt: ok (HTTP 200) at https://example.com/robots.txt Bot robots HTTP GPTBot blocked 200 FAIL ClaudeBot allowed 403 (block page) FAIL Google-Extended allowed - Enter fullscreen mode Exit fullscreen mode Here GPTBot is closed by robots.txt even though the server would answer, and ClaudeBot is allowed by robots.txt but gets a block page. Both are things you would want to know before wondering why a product is never cited.
npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked npx crawlerpeek check https://example.com --bots all --json npx crawlerpeek check https://example.com --no-http # robots.txt only npx crawlerpeek see https://example.com --bot ClaudeBot # one bot, full detail Enter fullscreen mode Exit fullscreen mode One limit matters for the topic of this post: crawlerpeek sends the bot's user-agent string from your machine. It is not the real bot and does not come from the operator's IPs. If your site verifies bot IPs, a request that only claims to be Googlebot may be blocked, so a 403 for Googlebot can be correct. A 403 for an unusual bot while a browser gets 200 usually points at a WAF or CDN rule.
Exit code 0 means ok, 1 means the --fail-on condition was met, 2 means a usage or request error. Unknown options are an error, so a typo like --failon cannot pass a build silently.
- uses: crawlerpeek/crawlerpeek/action@v0 with: { url: " https://example.com", fail-on: blocked } Enter fullscreen mode Exit fullscreen mode Or call the CLI yourself after a deploy:
- run: npx crawlerpeek check https://example.com --fail-on blocked Enter fullscreen mode Exit fullscreen mode --fail-on blocked fails when robots.txt closes the path for a bot, or the page answers that bot with 401, 403, 406, 451 or a challenge page. any-non-200 is stricter and also fails on 404, 5xx and redirects that end elsewhere. Run it against the pages you care about, not only the home page.
A robots.txt that cannot be fetched is reported as unknown and does not count as blocked. Under RFC 9309 a 5xx robots.txt means "disallow all" for many crawlers, so treat unknown as something to fix.
crawlerpeek mcp starts an MCP server over stdio with three read-only tools: check_ai_access, see_as_bot and generate_robots_txt. It calls no LLM; the tools only fetch the URL you give them, and results are cached for 10 minutes.
{ "mcpServers": { "crawlerpeek": { "command": "npx", "args": [ "-y", "crawlerpeek", "mcp" ] } } } Enter fullscreen mode Exit fullscreen mode Then you can ask your assistant "can ClaudeBot read /pricing on my staging site?" and it runs the check instead of guessing. It is also listed in the official MCP Registry as io.github.crawlerpeek/crawlerpeek.
To generate a robots.txt that names the AI crawlers explicitly:
npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml Enter fullscreen mode Exit fullscreen mode Disallowed paths are repeated inside each bot's group on purpose, for the reason above.
Source is MIT on GitHub: https://github.com/crawlerpeek/crawlerpeek. Package: https://www.npmjs.com/package/crawlerpeek (0.1.2). I would like to hear about missing bots, wrong user-agent strings and false block-page detections. Issues are open.