GPTBot vs OAI-SearchBot vs ChatGPT-User: What to Block (and Keep)

I almost broke my own site's AI discoverability with a copy-paste.

Last week I grabbed one of those "block all AI training bots" robots.txt gists, dropped it into a side project, and felt good about it for about ten minutes. Then reader alexcodebytes showed up in the comments of my Meta crawler piece and, over two days, quietly dismantled everything I'd done — with user agent strings, IP-behavior notes, and census data I hadn't seen.

Thanks to reader alexcodebytes, who did the real digging in the comments of my last post. This article is basically their outline, stress-tested by me.

Here's the part that stung: I'd blocked three bots that do three completely different jobs. And I'm not alone — far from it.

This is OpenAI's training crawler. It scrapes the open web so OpenAI can feed future model-training runs. Blocking it opts your future content out of training datasets. That's it.

What it does NOT do: power citations, answer lookups, or anything you'd see today. Blocking GPTBot changes nothing about whether your site appears in ChatGPT answers right now. It only changes whether your pages get folded into future training data.

Block it if training is what you object to. Keep it if you don't care, or if you want your content in the data the next generation of models learns from. Either way, it's a clean, narrow decision.

This one is completely different. OAI-SearchBot crawls for ChatGPT's search features — the live lookup behind answers with citations, the "sources" links users actually click. It is not a training crawler.

Block it, and your site vanishes from AI-generated answers. Not from training — from being cited. When someone asks ChatGPT a question your content answers perfectly, your page isn't even in the room.

This is where the gist-copy failure happens. The "block all AI" lists bundle OAI-SearchBot right in with GPTBot, usually under a comment like "OpenAI bots". You paste it thinking you've opted out of training. You've actually opted out of being referenced.

This was the piece I didn't know until alexcodebytes dug it up. ChatGPT-User isn't a crawler at all. When a ChatGPT user pastes your URL into a chat and asks "summarize this", ChatGPT-User fetches the page — on demand, from the user's IP address. It runs like a visitor. It's not used for training.

Block this and nothing about training changes. What changes is that every ChatGPT user who pastes one of your links gets a dead fetch. You break "summarize this URL" for your own readers and have no idea why the referral traffic dried up, because from your logs it looks like ordinary visitors failing.

Here's the data that made me rewrite the whole blocklist.

GeoParrot ran a census of the top 5,000 sites' robots.txt files in September 2026. Of the 535 sites that block GPTBot, 238 also block OAI-SearchBot. That's 44.5% — nearly half the sites that opted out of training also, probably without realising it, killed their own ChatGPT citations.

PageLens found the same shape from a different angle: their October 12, 2026 study of 740 top sites showed 15% blocking GPTBot, and of those, 63% also block ChatGPT's search and browsing agents. I haven't run a crawl study myself — these numbers are from GeoParrot's census and PageLens' study, linked below — but two independent sources pointing the same way is hard to argue with.

And there's a second, dumber failure mode. Cloudflare retired its one-click "Block AI Bots" feature in September 2026. One analysis found 118 sites silently lost their GPTBot bans when it went away — their robots.txt said one thing, their edge said nothing. If you set up blocking through a CDN toggle back in the day, go check whether it still does anything. Old rules rot.

The best thing alexcodebytes contributed, honestly, is a habit, not a fact. Write comments in your robots.txt explaining WHY each block exists. Future-you, at 11pm, about to commit a "clean up robots.txt" that nukes your SEO, will thank past-you. Here's the pattern:

# ========================================== # AI BOT POLICY — last reviewed 2026-10-21 # Principle: decide per PURPOSE (training vs answers), not per company. # ========================================== User - agent: GPTBot Disallow: / # WHY: opt out of OpenAI model TRAINING. Does not affect ChatGPT # citations — that's OAI-SearchBot's job (see below). Revisit yearly.

User - agent: OAI - SearchBot Allow: / # WHY: powers ChatGPT search + citations. Blocking kills our visibility # in AI answers. We WANT to be cited.

Do not block. User - agent: ChatGPT - User Allow: / # WHY: on-demand fetcher for user-pasted links, runs from the user's IP, # not used for training. Blocking breaks "summarize this URL" for readers.

User - agent: CCBot Disallow: / # WHY: generic scrape-to-sell dataset bot, no citation value to us. Enter fullscreen mode Exit fullscreen mode Two things about this. First, the comments are the product — the directives are trivially copyable, the reasons are what survive a late-night refactor.

Second: verify your CDN or WAF isn't contradicting the file. One published study found 8-11% of top sites server-refuse bots their robots.txt explicitly allows — edge rules left over from an old "block AI" toggle fighting the robots.txt that replaced them. Check both layers, not just one.

Other bots worth one line each, since you'll meet them: ClaudeBot (Anthropic training — same training-vs-answers decision as GPTBot), Google-Extended (Gemini training; Google says blocking it doesn't affect Search rankings), PerplexityBot (citations and answers, like OAI-SearchBot), meta-externalagent (Meta training/data use), and CCBot (Common Crawl dataset — see my comment above).

Honestly? On most sites I'd run, I'd block GPTBot and ClaudeBot — I'm opting out of training, and that decision is cheap. I'd explicitly allow OAI-SearchBot and PerplexityBot — citations are free discovery, and giving them up to protect against training is the 44.5% mistake. ChatGPT-User stays allowed always; blocking it has zero training upside and real reader downside.

The one case where I'd go stricter: if your content is the product itself — paywalled reporting, a dataset people pay for — then train-blocking plus answer-blocking might be the right call. But make it a conscious call, written in a comment, not a paste accident.

No affiliate links in this one — it's a service post, not a shopping list. If you want the deep data, start with the GeoParrot census and the PageLens study.

When's the last time you actually read your own robots.txt? If this saved you a future 11pm incident, leave a reaction — and go thank alexcodebytes in the comments.