Serving Markdown to AI agents from a Cloudflare Worker, without a second copy of the site

AI agents read web pages the expensive way. They download the HTML, throw away the navigation, scripts and styles, and try to recover the text. Some of them now send Accept: text/markdown and take Markdown if the server offers it.

I wanted our landing page to answer that request without keeping a second Markdown copy of the site in sync by hand. The site is static files behind a Cloudflare Worker, so the Worker converts the HTML on the way out. This post walks through that module: about 160 lines, no dependencies, built on HTMLRewriter.

The rule is narrow on purpose. Convert only a successful GET for an HTML asset, and only when the client actually asked for Markdown:

export async function serveMarkdownForAgents ( request, assets ) { const response = await assets. fetch ( request ); if ( request. method!== ' GET ' ||! wantsMarkdown ( request ) ||! response. ok ||! response. headers. get ( ' content-type ' )?. toLowerCase (). startsWith ( ' text/html ' )) { return response; } //...convert } Enter fullscreen mode Exit fullscreen mode Images, CSS, the sitemap and everything else pass through untouched, even when the request prefers Markdown. Browsers never send text/markdown, so people keep getting HTML.

wantsMarkdown reads the Accept header properly instead of searching it for a substring. It splits the media ranges, looks for text/markdown, and respects the quality value:

function wantsMarkdown ( request ) { return request. headers. get ( ' accept ' )?. split ( ', ' ). some (( part ) => { const [ mediaType,... parameters ] = part. trim (). toLowerCase (). split ( '; ' ); if ( mediaType!== ' text/markdown ' ) return false; const quality = parameters. find (( parameter ) => parameter. trim (). startsWith ( ' q= ' )); if (! quality ) return true; const value = Number ( quality. trim (). slice ( 2 )); return value > 0 && value <= 1; })?? false; } Enter fullscreen mode Exit fullscreen mode The q=0 case matters. Accept: text/html, text/markdown;q=0 means "anything but Markdown", and a naive includes('text/markdown') would send exactly the wrong thing. There is a test for that case, and one confirming that a non-HTML asset is unchanged when Markdown is requested.

HTMLRewriter is a streaming parser. You register handlers for selectors and get callbacks for elements and text chunks. There is no DOM tree, so the converter keeps a little state of its own:

Everything that is page chrome is skipped: head, nav, footer, script, style, noscript, svg, button, template, plus anything marked hidden, aria-hidden="true", role="navigation" or with a skip-link class. When such an element opens, the depth goes up, and onEndTag brings it back down. Text callbacks only write when the depth is zero.

if ( OMIT. has ( tag ) || hidden ) { element. remove (); if (! VOID. has ( tag )) { skippedDepth += 1; element. onEndTag (() => { skippedDepth -= 1; }); } return; } Enter fullscreen mode Exit fullscreen mode The VOID check is easy to miss. Void elements such as <img> or <br> have no end tag, so registering onEndTag on them would leave the counter stuck and silently drop the rest of the page.

The mapping itself is ordinary. Headings become # prefixes. Paragraphs and sections become blank lines. strong and em become ** and *, and pre becomes a fenced block. Two cases need care:

Text nodes are decoded ( &,, numeric entities) and whitespace is collapsed. After the stream ends, one cleanup pass trims spaces around newlines and caps blank lines at one.

The parts of <head> that matter to an agent are kept, just not as HTML. Separate handlers collect the <title>, the meta description, the canonical URL and the text of any application/ld+json blocks. The output starts with front matter:

--- title: "..." description: "..." url: " https://example.com/" --- Enter fullscreen mode Exit fullscreen mode The JSON-LD is appended at the end under a ## Structured data heading. The structured data a search engine sees, such as the FAQ and the product details, also reaches an agent that never parses HTML.

This is the part that breaks caches if you skip it. The body changed, so several headers copied from the HTML asset are now lies:

headers. set ( ' content-type ', ' text/markdown; charset=utf-8 ' ); // add Vary: Accept unless it is already there (or Vary: *) for ( const name of [ ' content-length ', ' content-encoding ', ' content-md5 ', ' etag ', ' content-range ', ' accept-ranges ', ' digest ' ]) headers. delete ( name ); Enter fullscreen mode Exit fullscreen mode Vary: Accept is what stops a shared cache from serving the Markdown version to the next browser that asks for the same URL. Dropping the old ETag avoids the opposite mistake: a conditional request answered with a 304 that points at the HTML body.

Last, the whole conversion is wrapped in try/catch. If anything throws, the original HTML response goes out. An agent getting HTML is a degraded result; a 500 is a broken one.

curl -s -H "Accept: text/markdown" https://talkaside.com/ | head -20 Enter fullscreen mode Exit fullscreen mode The same URL in a browser still returns the normal page.

For a marketing page made of headings, paragraphs, lists and an FAQ, 160 lines of HTMLRewriter were enough, and there is only one source of truth.

I'm building Talkaside, a Windows app for job interviews and calls in another language.

Drafted with AI assistance from this project's code and logs.