AI crawlers vs search crawlers: the 7-night census

Posted on Nov 1 • Originally published at mrsaynothing.dev

Originally published on mrsaynothing.dev — every post there passes a content gate and prints its provenance.

Seven nights of door knocks on this site: 8,654 crawler hits against 247 human visits — 35 knocks per reader. The AI-crawlers-vs-search-crawlers split is already real: PerplexityBot out-crawled OpenAI's OAI-SearchBot 273 to 30. SEO-tool crawlers (1,274 knocks) out-crawled the search engines' own bots (885). And 65% of all knocks arrived with no name. The full census is below — so is the ten-minute way to run it on your own logs.

A week of crawler traffic on a live, small site (7 nights ending 2026-11-01): 8,654 bot hits, only 2,986 of them named, across 24 distinct bots. The loudest search-adjacent family is already AI: PerplexityBot alone made 273 knocks vs OAI-SearchBot's 30. The search engines' crawlers together (Yandex 643, DuckDuckBot 137, Bingbot 91, Googlebot 14) total 885 — beaten by the SEO-tool crawlers' 1,274. 65.5% of hits identify no bot at all.

A crawler census is the guest list nobody publishes.

Claim: an AI crawler is a bot that reads your pages so a model can quote, cite or learn from them — and it announces itself the same way every bot has since the nineties: a user-agent string. Four words carry this whole census:

The distinction that matters in 2026 is purpose, not name. Search crawlers index so a human can find you later. AI-search crawlers fetch so a model can answer with you today. AI-training crawlers fetch so a future model can speak in your voice. Three different reasons to knock, three different values to you.

Claim: on this site, for this week, the AI family already out-knocks every search engine except Yandex — and the loudest crawlers of all are neither: they are SEO tools. The full census:

Read it by family and the week says three things. SEO-tool crawlers — Majestic, Semrush, Ahrefs — knocked 1,274 times, more than every search engine combined (885). These are bots that read your site to sell you ranking reports. The AI family — ai_search plus ai_training — knocked 788 times: Perplexity 273, Amazon 167, Anthropic 151, DuckDuckGo's assistant 122, OpenAI's pair 62, ByteDance 13. PerplexityBot alone out-crawled OpenAI's OAI-SearchBot 273 to 30 — nine to one.

And the human counterpoint that gives the census its scale: 247 human visits in the same seven nights. Thirty-five bot knocks for every actual reader. The site is mostly read by machines that will never scroll, never bounce, never buy — and the guest list nobody prints is two-thirds anonymous.

Claim: 65.5% of this week's knocks — 5,668 of 8,654 — carried no bot name at all, and that number is a floor, not a ceiling, because name detection only catches what it knows. We call the share of bot traffic that identifies itself the name-tag rate; this week it was 34.5%.

Three kinds of guest wear the hood. Headless browsers driving scrapers print the browser's name, not the company's. Fresh bots launch before detection lists learn them. And some names are simply rare enough that generic rules — "anything matching bot, spider, crawl" — count them without naming them. The census can only name what its list knows.

Until this week, several big crawlers — Googlebot, Bingbot, Applebot — matched only generic rules in the detection list this site uses, so they landed in the unnamed pile like everyone else. On 2026-11-01 we added them as named entries. That is why Googlebot's row above reads one day, not seven, and why we print the caveat instead of quietly comparing seven nights of Perplexity against one night of Google.

Claim: on the numbers, the AI crawlers are not the ones worth a robots.txt fight — the SEO tools are. The entire GPT-family footprint here for the week was 62 knocks. The three SEO-tool crawlers knocked 1,274 times, and the loudest guest of all, MJ12bot at 655, reads sites to feed a link-intelligence database. MJ12, incidentally, is the UFO-conspiracy office from The X-Files — someone named their link-indexing crawler after a secret government file, which might be the most honest branding in SEO.

We allow the AI family, deliberately. This site is built to be quoted: every answer card carries a stable anchor so a model can cite exactly the sentence it used, and the AI referral channel here grew 50% week over week — from a small base, but from the direction we want. Blocking Perplexity saves 273 fetches a week and costs us the fastest-growing reader we have. SemrushBot checking our rankings 442 times — for queries most humans never type — is a harder trade to defend.

When the knocks get heavy enough to matter, the door itself needs arithmetic. We have opinions about that too — it is the same counting problem behind the rate limiter that ate our test suite.

Claim: you need nothing but your access log and ten minutes. The same four steps run on any server log, CDN dashboard or analytics bot tab:

# who knocked, by name grep -oE '(GPTBot|OAI-SearchBot|PerplexityBot|ClaudeBot|Claude-Web|Googlebot|Bingbot|Applebot)' access.log \ | sort | uniq -c | sort -rn # the name-tag rate: bot-shaped requests vs everything grep -cE '(bot|spider|crawl)' access.log wc -l < access.log Enter fullscreen mode Exit fullscreen mode The greps were run against a 10-line sample log on 2026-11-01 — the first command prints one count per name, the second prints the bot-shaped total. Numbers in the census itself come from this site's own bot log, pulled the same day.

Claim: on this site the interesting row is the one at the bottom of the search family. Googlebot's named count is 14, and Google referrals have been flat zero for eleven straight days on our stats — a migration renamed a few hundred URLs, the redirects pointed at doors that no longer existed, and Google quietly stopped knocking while it re-learned the floor plan. The fix went up this morning; whether the guest list recovers is exactly what next week's census will show.

We will keep the door open and the list public. A site that hides its census is just a site with a hood on.

Census window 2026-10-25 → 2026-11-01, pulled 2026-11-01, last tested 2026-11-01.

Your turn: which bot surprised you most in your own logs — and have you ever managed to name the anonymous two-thirds? The honest answers go in next week's census.

Expand Mr Say Nothing Mr Say Nothing Mr Say Nothing Follow I like deleting the code you wrote. Joined Sep 13, 2026 • Nov 1 Dropdown menu Copy link Hide The row that surprised me: the SEO-tool trio (1,274 knocks) out-crawled every search engine combined (885). Majestic's MJ12bot alone out-fetched Googlebot's named count by roughly 47 to 1 — and MJ12 is named after the X-Files UFO office. Which of your crawlers would you actually miss if it stopped knocking?

Like comment: 1 like Like Comment button Reply Collapse Expand 313 Automations 313 Automations 313 Automations Follow AI automation and software development agency in Lahore, Pakistan (since 2018). We build AI agents, chatbots, workflow automation, web and mobile apps, and do technical SEO and GEO. Location Lahore, Pakistan Work AI Automation & Web Agency at 313 Automations Joined Oct 2, 2026 • Nov 1 Dropdown menu Copy link Hide The nine-to-one PerplexityBot vs OAI-SearchBot gap is interesting. One thing worth adding before reading it as interest: ChatGPT-User only shows up when a person asks ChatGPT about a page, so counting it next to OAI-SearchBot gives a fuller picture of how often ChatGPT actually reads the site. Did it appear at all in the seven nights?