Which AI bots visit my website? I counted for 30 days

Posted on Oct 22 • Originally published at yappatyih.com

AI crawler detection means finding out which bots that collect pages for AI companies visit your website, how often, and what they read, usually from the name each bot gives in its requests. In the 30 days to 20 October 2026, Cloudflare counted about 4,040 requests to this site from the AI crawlers it recognises. Google sent the most, then Anthropic, OpenAI, Apple and Microsoft.

About half of the successful requests were for training AI models. Only 26 were an AI assistant fetching a page because a person asked it something. The crawlers read my sitemaps more than any post, some "AI crawlers" were scanners hunting for password files, and all that reading did not get me cited: ChatGPT named this site in 0 of 12 answers on 17 October.

Below are the numbers, and how to see the same on your own site.

In the 30 days to 20 October 2026, ten companies sent most of the AI crawler traffic to this site. Cloudflare groups each company's crawlers together, so OpenAI's row covers GPTBot and two more of its bots.

Googlebot is the crawler behind Google Search, and Google's AI answers draw on the same index, which is why Cloudflare lists it here. Anthropic came second, ahead of OpenAI. Perplexity, whose answers did cite this site on 17 October, is not in the top ten at all.

A week earlier, on 13 October, the same 30-day view showed about 3,000 requests: Googlebot 602, ClaudeBot 464, GPTBot 356, Applebot 221. On 20 October it showed Googlebot 889, ClaudeBot 642, GPTBot 373, Applebot 377. The two windows share 23 days, so this is not a clean month on month figure. The direction is still plain: more crawling, most of the growth from Google, Anthropic and Apple.

Cloudflare sorts every AI crawler by purpose, and on this site training came first by a wide margin. Of about 3,540 successful requests in the 30 days to 20 October 2026:

That last row is the one I care about most. Twenty six times in a month, someone asked an AI something and it came to read a page here. Everything else was machines filling their stores in case they need it later.

The most read file on this site is not a post. It is the list of posts. In the 30 days to 20 October 2026:

The four sitemap files took 687 requests between them. The busiest post took 66. Crawlers spend most of their visits finding out what exists, and much less reading it. 271 different paths were requested in all. In bytes, the whole month came to about 74 MB, Googlebot 18.63 MB and GPTBot 11.24 MB of it. On Cloudflare's free plan that cost me nothing.

So a correct, current sitemap matters more than any special file made for AI. In the 30 days to 13 October, the same crawlers fetched my llms.txt zero times and my robots.txt 605 times.

Not every request under an AI crawler's name came from an AI company. Among the ten missing paths requested most often in the 30 days to 20 October 2026, eight were files that hold passwords and keys:

A real training crawler follows links and sitemaps. It has no reason to ask for a.env file that nothing links to. These were scanners using a crawler's name in their user agent, and on the free plan Cloudflare recognises crawlers by that name, so they were counted as AI crawlers. Every one got a 404, because none of those files exist here. Borrowing a name is that easy: in my first check of this site I did it myself, from my own computer, to see what each crawler gets back.

If someone else built your website, ask them to check this today. Open yoursite.com/.env and yoursite.com/.git/config in a browser. Both should give an error page, never a file.

All that reading has not yet turned into AI answers that name this site. The same month, side by side:

A crawler visit means my page is in the pile. Whether an answer quotes it depends on what the page says, who else says it, and how far the engine trusts the site. The number of visits does not predict the number of citations.

I also cannot tell you whether any of those answers sent me a visitor. Cloudflare's Referral Traffic column showed a dash on every path. Its documentation says referral counts are for paid plans, so on the free plan that dash means "not counted", not zero.

There are three ways to check, and the first takes a few minutes if your site already runs on Cloudflare, free plan included.

1. Cloudflare AI Crawl Control. Log in to the Cloudflare dashboard, pick your account and your domain, and open AI Crawl Control.

Two limits: on the free plan a crawler is recognised by the name it gives, so a scanner wearing GPTBot's name counts as GPTBot; and there is nothing older than 30 days.

2. Your server log. If your site is not on Cloudflare, ask your host for the raw access log; on cPanel hosting it is under "Raw Access". Then count the crawler names:

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Googlebot|Bingbot|Applebot|PetalBot|Meta-ExternalAgent|CCBot|Amazonbot|Bytespider" access.log | sort | uniq -c | sort -rn Enter fullscreen mode Exit fullscreen mode A name is only a name, so check the address of any request that matters: Google and OpenAI publish the address ranges their crawlers use. Google Analytics will not show any of this. It leaves known bots out, and most of these crawlers never run its script.

3. Your robots.txt. Logs tell you who came. robots.txt tells you who is allowed. My free AI crawler checker reads your robots.txt for every major AI crawler at once and shows which ones it lets in. To see what a crawler really gets back, request a page with its name, as I did in my first check.

My robots.txt lets every one of these crawlers in, training crawlers included. I write here so that people, search engines and AI answers can find what I think, and an answer cannot quote a page its crawler never read. A month of data has not changed that. It has only shown me that letting them in is the start, not the result. Most big Malaysian websites leave the door open too: when I read the robots.txt of 100 of them, only 13 blocked any AI crawler.

If your site is on Cloudflare, open AI Crawl Control for your domain: it lists every AI crawler it recognises, its company, its purpose and the pages it read, for up to 30 days, on the free plan too. Otherwise, search your server's access log for crawler names such as GPTBot, ClaudeBot and PerplexityBot. To see which bots your robots.txt allows, use an AI crawler checker.

On this site, in the 30 days to 20 October 2026, Google sent the most allowed requests (961), then Anthropic (680), OpenAI (508), Apple (377) and Microsoft (309). By single crawler, Googlebot led with 889 requests, then ClaudeBot 642, Applebot 377 and GPTBot 373. Other sites will differ, but these names are the ones to look for first.

A training crawler, such as GPTBot or ClaudeBot, collects pages to train AI models. An AI search crawler, such as OAI-SearchBot, builds the index an AI searches when it answers. An AI assistant fetch, such as ChatGPT-User or Claude-User, happens when a person asks a question and the assistant reads a page for them. On this site the third kind was 26 requests in 30 days, out of about 3,540.

Yes. Any program can put GPTBot or ClaudeBot in its user agent. On this site, in the 30 days to 20 October 2026, requests under AI crawler names asked for.env,.git and credentials.json files, which no real crawler needs. Check the address against the ranges the company publishes, and keep no secret files on your web server.

Not necessarily. OpenAI's crawlers made 508 requests to this site in the 30 days to 20 October 2026, and ChatGPT cited it in 0 of 12 answers on 17 October. Being crawled puts your page in the pile; being cited depends on what the page says and how far the engine trusts your site.

I write about running a company and building with AI at yappatyih.com.