Mit robots.txt und llms.txt die eigene Website fuer KI-Crawler lesbar machen
Wer in ChatGPT und anderen KI-Assistenten genannt werden will, sollte AI-Crawler in der robots.txt erlauben, zentrale Inhalte als einfachen HTML-Text ausliefern und Schema-Markup nutzen. Eine llms.txt-Datei mit einer kurzen Zusammenfassung und Links zu wichtigen Seiten ergänzt das.
Posted on Oct 25 • Originally published at getvizible.online
Short answer: allow the AI crawlers in robots.txt, serve your key facts as plain HTML text, add schema markup, and publish an llms.txt file: a short Markdown summary of who you are with links to your most important pages.
A lot of "why doesn't ChatGPT mention us?" problems start with something boring: the bots that fetch pages for AI answers can't read the site. Here's the checklist.
The important distinction: blocking a search or user agent removes you from that engine's live answers. Blocking a training agent only affects future model versions. A lot of sites added a blanket AI block in 2024–2025 to opt out of training and accidentally took themselves out of ChatGPT search and Perplexity too.
Decide per agent. For most brands that want to be recommended, allowing all of them is the simplest choice.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Disallow: /account/ Sitemap: https://example.com/sitemap.xml Enter fullscreen mode Exit fullscreen mode Also check your CDN or WAF. Bot-protection settings can silently block these agents even when robots.txt allows them. Your server logs will tell you whether they're getting 200s or 403s.
llms.txt is a proposed convention: a Markdown file at /llms.txt that gives language models a short, clean summary of your site. It typically has:
# Example Co > Example Co makes invoicing software for freelancers in the EU. ## Key facts - Pricing: free plan; Pro is €12/month - Founded 2024, based in Lisbon ## Docs - [ Getting started ]( https://example.com/docs/start ): set up in 5 minutes - [ Pricing ]( https://example.com/pricing ): plans and limits Enter fullscreen mode Exit fullscreen mode Some sites also publish llms-full.txt with the full text of their key pages. Support differs by engine, so treat llms.txt as a cheap, helpful extra rather than a guarantee. Crawler access and clear page text matter more.
Does blocking GPTBot remove me from ChatGPT? Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot, and user-triggered fetches use ChatGPT-User, so block or allow those separately.
Is llms.txt required? No. It's a voluntary convention. It's quick to add and gives AI tools a clean summary of your site, but crawler access, clear page text and third-party mentions matter more.
Originally published on the Get AI Vizible blog. Disclosure: I'm the founder of Get AI Vizible, which checks AI crawler access, llms.txt and schema as part of tracking how ChatGPT, Claude, Gemini and Perplexity describe your brand.