Is your site invisible to ChatGPT? Check your robots.txt

Posted on Nov 4 • Originally published at indiedaddy.com • AI-assisted

A lot of people now ask ChatGPT or Perplexity "what's a good tool for X" instead of Googling it. If your product never comes up, there might be a dull reason: your robots.txt is telling AI crawlers to stay out.

Some starter templates do this. So do some CDN "block AI bots" toggles. You might not even know it's on.

This is the part most people miss. AI companies run separate crawlers for search and for training, so you can block one and allow the other.

If the search ones ( OAI-SearchBot, Claude-SearchBot, PerplexityBot ) are blocked, assistants that search the web will struggle to cite you.

Open https://yoursite.com/robots.txt and look for something like:

User-agent: GPTBot Disallow: / User-agent: * Disallow: / Enter fullscreen mode Exit fullscreen mode A bot follows the group with its own name, and falls back to * if there isn't one. If two rules match, the longer path wins. Disallow: with nothing after it means everything is allowed, which trips people up.

Reading these by eye gets old fast, so I put a free AI crawler checker on IndieDaddy. It lists every bot above as allowed or blocked and tells you if you have an llms.txt. No signup, and it doesn't store anything.

User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: * Allow: / Disallow: /api/ Sitemap: https://yoursite.com/sitemap.xml Enter fullscreen mode Exit fullscreen mode For most SaaS landing pages I'd just allow everything. The pages are public anyway, and getting cited is kind of the whole point.

llms.txt is a proposed standard from 2024: a short Markdown file at /llms.txt that sums up your site for language models. Not every AI company reads it, so think of it as a small bonus.

# YourProduct > YourProduct helps freelance designers send invoices and get paid on time. > Pricing: free for 3 clients, Pro $12/month. ## Key pages - Features: https://yourproduct.com/features - Pricing: https://yourproduct.com/pricing - Docs: https://yourproduct.com/docs ## Not for - Payroll or accounting. Enter fullscreen mode Exit fullscreen mode I'd keep the "Not for" bit. It stops assistants suggesting you for stuff you don't do.

And get mentioned on sites the assistants already trust: decent directories, review sites, comparison posts. That part is slow and there's no shortcut.

I wrote up the rest of it, including a 30-minute monthly check, in how to get your SaaS recommended by ChatGPT, Claude and Perplexity.

Did you find any AI bots blocked on your own site? I was surprised how often it's on by default.