What 4,700 bot visits taught me about AI crawlers (and a tool to check any site's robots.txt

A few weeks ago I wanted to answer a simple question: how many of the "visitors" on my small side projects are actually people?

The answer turned out to be: not many. Most of the traffic was crawlers, a lot of them from AI companies, and I had no easy way to see which ones, how often, or where they came from. Server logs tell you a User-Agent string and an IP, and the User-Agent can say anything it likes.

So I built BotSpotter. It's a live map of AI crawlers hitting participating sites, open source under MIT. This post is about what the first week of data showed me, because some of it surprised me.

OpenAI, Google, Microsoft, Apple, Perplexity, DuckDuckGo and Common Crawl all publish the IP ranges their crawlers use. BotSpotter checks every hit that claims to be one of those bots against the published ranges, in-process, before the IP is discarded.

After 1,500 such hits: 1,425 verified, 75 impostors. The most popular disguise is not GPTBot, it's Googlebot (40 of the 75), followed by GPTBot (20), Bingbot (12) and PerplexityBot (3). If you rate-limit or whitelist by User-Agent, this is the traffic you're letting through.

Of 4,770 bot visits so far, 2,494 (56%) came from bots nobody has identified — no registry entry, no documented User-Agent, no published ranges. They're fingerprinted by UA family, header order and network, so the same bot keeps the same name ("Onyx Shark #5358"), but we don't know who runs them. The named AI companies are the visible tip; the bulk of machine traffic doesn't announce itself.

The first days after launch showed 500+ bot hits a day. Two weeks later it's ~320. Nothing broke: every search and SEO crawler does a discovery pass on a new domain (Googlebot, Bingbot, Ahrefs, Semrush, Meta all fetched every URL in the sitemap), then drops to maintenance re-checks. If you're measuring bot traffic on a fresh site, the first week is not your baseline.

By hosting network: Google Cloud (2,117 hits), AWS (293), Hetzner (287), OVH (270), Azure (170), Apple (165), Meta (85), DigitalOcean (77). Hetzner and OVH being that high is a tell — a lot of scrapers and unidentified crawlers rent cheap European VMs.

The feature people use most isn't the map. It's botspotter.app/check: type any domain and it fetches the robots.txt and evaluates it against 16 AI crawler tokens using RFC 9309 rules (a named group beats *, longest path match wins, Allow wins a tie). You get a grade per crawler — allowed, kept out of some paths, or blocked — grouped by what the operator says the traffic is for: training, AI search, or agents fetching for a live user.

Try it on a site you know: https://botspotter.app/check?site=nytimes.com

If you own the site, it gives you a paste-ready block for the training crawlers. And there's a second tool on the same page: paste an IP from your logs, pick the bot it claims to be, and it tells you whether the IP is in the operator's published ranges.

One honest caveat the page itself makes: robots.txt is a request, not a wall. The impostors above don't read it.

Code: github.com/bhushanladde02/botspotter

If you run a site, adding it is a GitHub sign-in and one line, and you get a live badge for your footer. More sensors means better numbers for everyone, and right now there are two.

And I'm curious: what's the weirdest bot in your logs?