What AI crawlers are blind to: the text your JavaScript renders

I run a small property site. Four languages, plain HTML, no framework, no npm dependencies, hosted on Cloudflare Pages. I have spent my working life on copy and search, so I did the usual things: structured data, hreflang, a sitemap, a generator that keeps every page's head in sync from one file.

Then I ran one command against my own listing page:

shell `bash curl -s -A 'GPTBot/1.0' https://example.com/my-listing | \ python3 -c "import sys,re,html; b=sys.stdin.read(); b=b[b.index(' b=re.sub(r'','',b,flags=re.S); \ print(len(re.sub(r'\s+',' ',html.unescape(re.sub(r'<[^>]+>',' ',b))).strip()))"

The answer for my listing page was 23 characters. For my homepage, 17.

Nowhere surprising, once you look. The site is multilingual, and the copy for every language lives in assets/js/i18n.js. On load, app.js picks the language and fills the containers:

var descEl = document. querySelector ( " [data-listing-description] " ); if ( descEl ) descEl. innerHTML = l. description + climateNoteFor ( id ); Enter fullscreen mode Exit fullscreen mode That is a perfectly ordinary pattern. It also means the HTML I ship contains the layout and nothing to read.

Not nothing. The head carried 2,891 characters of JSON-LD: name, price, number of rooms, floor area, address, coordinates, amenities. A machine could reconstruct a tidy product card from that.

What did show it was Cloudflare's AI crawler log. In 24 hours up to 15 September 2026 my domain took 43 requests from AI crawlers: Applebot 20, Googlebot 14, Meta-ExternalAgent 8, GPTBot 1. They fetched pages, the sitemap, images, the CSS, both JS files, and the favicon I had published an hour earlier.

Worth noting, since it is the fashionable question: llms.txt sits in my site root, and across 23 distinct crawled paths in that window it was requested exactly zero times. Cloudflare's own readiness checklist did not mention the file either. On that day its items were robots.txt, a sitemap, content signals, markdown versions of pages, and WebMCP.

I already had a generator for the head. It takes one data file and writes it into every page between markers:

<!-- SEO:START (generated by build/generate-seo.js — do not hand-edit between markers) -->... <!-- SEO:END --> Enter fullscreen mode Exit fullscreen mode The previous post is about it. The same idea works for the body. I put markers inside the containers the script already fills:

<div class= "listing-description" data-listing-description > <!-- BODY:listing-description --><!-- /BODY:listing-description --> </div> Enter fullscreen mode Exit fullscreen mode and taught the generator to fill them from i18n.js and listings-data.js — the same files the browser uses. That detail is the whole point: the static text cannot drift from what the script renders, because there is one source and one build step. app.js still redraws those containers on load, so the language switcher behaves exactly as before.

Total work: one script and a pass that inserted markers. No framework, no SSR runtime, no rewrite.

A generator earns its keep twice. The first time when it saves you from editing twenty files by hand. The second time when a new requirement lands and the data is already in one place, so the fix is a function rather than a project.

Run the command above on your own site. It takes ten seconds. If your stack renders on the server you will see a number in the thousands and can stop reading. If you see two digits, you now know something your analytics will never tell you.

The site: https://girne-apartment-sale.com/?utm_source=devto&utm_medium=article&utm_campaign=article04

Standalone, genericized version of the generator, if you want the code rather than the description: https://github.com/sergei-swh/seo-generator