Make a Nuxt or Next.js site readable by AI: llms.txt, robots.txt, Markdown

Ask an AI assistant about a page on your site and one of three things happens. It answers from what it learned in training, it answers from a search index it built earlier, or it fetches your page right now. In all three, a crawler read your HTML at some point. What it was allowed to read, and how easily, is the part you can shape. I built four pieces for that into the SEO plugin of Easy CMS, my embedded CMS for Nuxt and Next.js ( the first post in this series introduces it). The ideas work in any server-rendered site:

Each company publishes its own user-agent names, and they change, so don't copy a list from a blog post. Easy CMS keeps them in one exported constant, AI_CRAWLERS, grouped as above and updated in each release. The AI search guide shows the current list. A common choice: stay out of training, stay visible in AI answers.

// app/robots.txt/route.ts (Next.js) import { robotsTxt } from ' @easy-cms/plugin-seo ' import config from ' @/easy-cms.config ' export function GET () { // AI training crawlers stay out; AI search and assistants may still read and cite posts. const text = robotsTxt ({ config, ai: { training: false } }) return new Response ( text, { headers: { ' content-type ': ' text/plain; charset=utf-8 ' } }) } Enter fullscreen mode Exit fullscreen mode The output, shortened:

User-agent: * Allow: /api/cms/media/file/ Disallow: /admin/ Disallow: /api/cms/ User-agent: <one line for each crawler in the training group> … Disallow: / Sitemap: https://example.com/sitemap.xml Enter fullscreen mode Exit fullscreen mode Two choices need explaining. Everything is allowed by default.

If an upgrade started blocking a group, sites would quietly drop out of AI answers. Blocking is the site owner's business decision, so it's opt-in. Each group repeats the base rules.

A crawler follows only the most specific group that names it, without merging in User-agent: *. Add a group for one bot to disallow /search, and that bot may now crawl your admin and API. robotsTxt() adds the admin and API rules to every group it writes, including the ones you add with rules: [{ userAgent: 'SomeBot', disallow: ['/'] }]. If you write robots.txt by hand, copy your base rules into each group.

llms.txt is a proposed standard: a Markdown file at the root that says what the site is and where its main pages are. Here's what Easy CMS generates:

# My Blog > A blog about building websites in Thailand. ## Posts - [ Hello ]( https://example.com/posts/hello.md ): What this blog is about. ## Pages - [ My Blog ]( https://example.com/ ) Enter fullscreen mode Exit fullscreen mode You configure it in the plugin and serve it from a route:

// easy-cms.config.ts (excerpt) seoPlugin ({ collections: [ ' posts ' ], generateURL: ({ doc }) => `/posts/ ${ doc. slug } `, llms: { description: ' A blog about building websites in Thailand. ', limit: 100, // newest pages per collection (the default) markdownURL: ({ doc, collection }) => collection === ' posts ' && doc. slug? `/posts/ ${ doc. slug }.md`: null, }, }) Enter fullscreen mode Exit fullscreen mode // server/routes/llms.txt.ts (Nuxt) import { llmsTxt } from ' @easy-cms/plugin-seo ' export default defineEventHandler ( async ( event ) => { setHeader ( event, ' content-type ', ' text/markdown; charset=utf-8 ' ) return llmsTxt ( await useEasyCMS ()) }) Enter fullscreen mode Exit fullscreen mode Links are made absolute from a { siteURL } you pass, or else from admin.siteURL in the config, or serverURL when admin.siteURL is empty. The choices behind it:

// next.config.ts rewrites: async () => [{ source: ' /posts/:slug.md ', destination: ' /md/posts/:slug ' }], Enter fullscreen mode Exit fullscreen mode // app/md/posts/[slug]/route.ts import { getEasyCMS } from ' @easy-cms/next ' import { docMarkdown } from ' @easy-cms/plugin-seo ' import config from ' @/easy-cms.config ' export const dynamic = ' force-dynamic ' export async function GET ( _request: Request, { params }: { params: Promise < { slug: string } > }) { const { slug } = await params const cms = await getEasyCMS ( config ) // Read as a visitor: published posts only. const { docs } = await cms. find ( ' posts ', { where: { slug: { equals: decodeURIComponent ( slug ) } }, limit: 1, overrideAccess: false, user: null, }) const post = docs [ 0 ] if (! post || post. meta. noindex ) return new Response ( ' Not found ', { status: 404 }) // The address on the site from the config (admin.siteURL), not from the request's Host. const url = ` ${ cms. config. admin. siteURL. replace ( / \/ +$/, '' )} /posts/ ${ post. slug } ` return new Response ( docMarkdown ( cms, { collection: ' posts ', doc: post, url }), { headers: { ' content-type ': ' text/markdown; charset=utf-8 ' }, }) } Enter fullscreen mode Exit fullscreen mode In Nuxt, a server middleware matches /^\/posts\/([^/]+)\.md$/ and does the same. Three details matter: read with overrideAccess: false, user: null, so only what a visitor may see comes out. Respect noindex: a page you hid from search shouldn't reappear as Markdown.

And take the address from the config, not the request: a client can send any Host header, and a cached response would pass that address on to every reader. Why not Accept: text/markdown on the normal URL? One URL with two formats looks cleaner, but many caches key on the URL alone.

Unless every layer between you and the reader honors Vary: Accept, the first Markdown response gets cached and served to the next browser, or the reverse. A separate.md URL can't be mixed up.

Crawlers come back on their own schedule. IndexNow lets you tell Bing, Yandex and the other engines that take part about a changed URL right away. Bing's index also feeds ChatGPT search and Copilot, which is why I put it in this plugin.

seoPlugin ({ // … indexNow: { key: process. env. INDEXNOW_KEY }, // 8–128 letters, digits or dashes, e.g. a UUID }) Enter fullscreen mode Exit fullscreen mode Each part of the behavior has a reason:

llms.txt is a proposal, not a standard every assistant reads. I can't tell you which ones use it today, or whether any of this changes where you rank. What I can say: each file costs one route, and none exposes anything a visitor couldn't already read. [TODO: your story — one line on what you saw (e.g. which crawlers fetched /llms.txt or.md pages) after turning this on for a real site, or delete this line]

Easy CMS is MIT-licensed: github.com/maritonx/easy-cms · docs. If you try it, tell me what got in your way in the feedback issue.