Our robots.txt names eleven AI crawlers and then repeats the whole policy, because a crawler only obeys one group

Nakodo's robots.txt is 30 lines of text generated from 34 lines of TypeScript, and more than half of it is the same five rules written twice. That duplication is the only correct way to write the file, and it took reading the specification to believe it.

User-Agent: * Allow: / Disallow: /campaigns Disallow: /settings Disallow: /onboarding Disallow: /billing Disallow: /api/ Disallow: /auth/ User-Agent: GPTBot User-Agent: OAI-SearchBot User-Agent: ChatGPT-User User-Agent: ClaudeBot User-Agent: Claude-SearchBot User-Agent: Claude-User User-Agent: PerplexityBot User-Agent: Perplexity-User User-Agent: Google-Extended User-Agent: Applebot-Extended User-Agent: CCBot Allow: / Disallow: /campaigns Disallow: /settings Disallow: /onboarding Disallow: /billing Disallow: /api/ Disallow: /auth/ Sitemap: https://nakodo.app/sitemap.xml Enter fullscreen mode Exit fullscreen mode Groups are chosen, not merged The rule in RFC 9309 that makes this file look redundant: a crawler finds the group whose User-Agent line matches it most specifically, obeys that group, and ignores every other group in the file, including the wildcard one.

So this, which is what I wrote first, is a mistake:

User -Agent: * Disallow: /campaigns User -Agent: GPTBot Allow: / Enter fullscreen mode Exit fullscreen mode That does not mean "everyone keeps out of /campaigns, and GPTBot is also allowed the rest of the site". It means GPTBot has an empty disallow list and the run of the whole site, including the pages the wildcard group was protecting. Naming a crawler and then writing less than the wildcard group gives it more access, not less.

Which is why the rules are repeated verbatim. In the source there is exactly one copy:

import { PRIVATE_PREFIXES } from " @/lib/public-paths "; // The app behind sign-in has nothing for a crawler; everything else is open. // Links with tracking parameters stay crawlable on purpose: every page sets a // canonical URL, and a crawler has to fetch the page to see it. const DISALLOW = [... PRIVATE_PREFIXES, " /auth/ " ]; export default function robots (): MetadataRoute. Robots { return { rules: [ { userAgent: " * ", allow: " / ", disallow: DISALLOW }, { userAgent: AI_CRAWLERS, allow: " / ", disallow: DISALLOW }, ], sitemap: absoluteUrl ( " /sitemap.xml " ), }; } Enter fullscreen mode Exit fullscreen mode Next's MetadataRoute.Robots takes userAgent as a string or an array of strings.

An array becomes one group with eleven User-Agent lines, which is what RFC 9309 calls a group with several product tokens: they share one set of rules. That is the difference between eleven lines of file and eleven groups of eight lines, and it also means the two policies cannot drift, because they are the same constant.

The wildcard group already allows everything except the app. Every one of these eleven crawlers would be allowed the same by *. So the group adds nothing to the file's behaviour today, and I keep it anyway:

// AI search and assistant crawlers, named so the policy toward them is a // decision on record rather than an accident of the wildcard. A crawler obeys // only the most specific group that names it, so each gets the full policy. const AI_CRAWLERS = [ " GPTBot ", " OAI-SearchBot ", " ChatGPT-User ", " ClaudeBot ", " Claude-SearchBot ", " Claude-User ", " PerplexityBot ", " Perplexity-User ", " Google-Extended ", " Applebot-Extended ", " CCBot ", ]; Enter fullscreen mode Exit fullscreen mode A wildcard that allows AI crawlers is indistinguishable from never having thought about AI crawlers. A named group is a position: somebody wrote these names down, and changing the position is a one-line diff on an array rather than an argument about what the wildcard was supposed to imply.

Nakodo's public pages exist to be read and quoted; a prospective customer asking an assistant about creator outreach and getting our pages quoted back is the outcome we want, so the honest value of the array is "yes, on purpose".

The eleven are not all the same kind of thing, which is the part worth knowing before you copy the list:

That last group is the one I would have got wrong. Disallowing Google-Extended does not stop Googlebot fetching anything, and allowing it does not add a single request to the server. It is a permission, sitting in a file whose other lines are all traffic rules.

The five paths are not typed in robots.ts. They come from the module that decides what needs a session:

// Where robots.txt sends crawlers away from: the signed-in app. The auth pages // are left crawlable so crawlers can read their noindex tag. export const PRIVATE_PREFIXES = [ " /campaigns ", " /settings ", " /onboarding ", " /billing ", " /api/ " ]; Enter fullscreen mode Exit fullscreen mode Two details in there took a while to settle.

/auth/ is appended in robots.ts but the sign-in pages themselves are not blocked. Blocking a page in robots.txt and marking it noindex are mutually exclusive instructions: a crawler that is not allowed to fetch the page can never read the tag that asks it not to index the page, so a blocked-and-noindexed URL can still show up in results as a bare link. The sign-in pages therefore stay crawlable and carry the tag instead:

// Sign-in pages stay out of search results. robots.txt leaves them crawlable so // crawlers can read this. export const metadata = { robots: { index: false, follow: true } }; Enter fullscreen mode Exit fullscreen mode You can check that one from a terminal: curl -s https://nakodo.app/login | grep robots prints name="robots" content="noindex, follow".

And none of the prefixes has a trailing slash. Matching in robots.txt is a literal prefix, not a path segment match, so Disallow: /campaigns also covers a hypothetical /campaigns-public. For this site that is the behaviour I want, and the one URL space where the difference matters, the API, is written with the slash because /api/ needs to block what is under it and there is no page at /api itself.

The real failure mode here is not a crawler getting into the app. It is a public page being treated as private, which is silent: the page works perfectly for anyone signed in, and crawlers get a redirect to the sign-in form. So the same module's test walks the sitemap:

test ( " every page in the sitemap is open to signed-out visitors ", () => { for ( const path of [... STATIC_PAGES. map (( p ) => p. path ),... ENTRIES. map (( e ) => e. path )]) { assert. ok ( isPublicPath ( path ), ` ${ path } would redirect crawlers to /login` ); } for ( const path of [ " /opengraph-image ",... CARD_PATHS. map (( p ) => `/og ${ p } ` )]) { assert. ok ( isPublicPath ( path ), ` ${ path }: social cards must be public for link previews` ); } }); Enter fullscreen mode Exit fullscreen mode The second half of that test is there because a link preview is an unauthenticated fetch of an image route: a social card behind the session check turns every shared link into a picture of a login form, and nothing in the app would ever show you that.

Allowing an assistant to crawl is only half of it. The pages are also published as markdown for exactly these clients, at llms.txt and llms-full.txt, generated from the same arrays that render the HTML, so a quote from an assistant and a quote from a reader are the same sentence.

There is one place where naming yourself a bot changes the response, and it is not the robots file. Prices are rendered per request in the visitor's currency, and anything that looks like a crawler gets the base currency instead of a datacentre's:

// Crawlers get pounds: they're placed wherever their servers are, and search // results should show the base price, not a datacentre's currency. export async function displayCurrency (): Promise < Currency > { if ( BOT. test (( await headers ()). get ( " user-agent " )?? "" )) return BASE_CURRENCY; return currencyForCountry ( await requestCountry ()); } Enter fullscreen mode Exit fullscreen mode Which is checkable in one line:

$ curl -s -A GPTBot https://nakodo.app/pricing | grep -oE '£[0-9]+' | sort -u £0 £39 £119 Enter fullscreen mode Exit fullscreen mode Those are the real plan prices on the pricing page. Fetch the same URL from a browser in the euro area and the same numbers arrive as euros.

The file is boring on purpose and it still needed three decisions: repeat the rules because groups are chosen rather than merged, name the AI clients because a wildcard records no decision, and keep the blocked list in the module that already knows what is private so robots.txt and the sign-in check can never disagree. The one line of the file nobody argues about, the Sitemap: line, is the one that gets fetched most.