AI crawlers don't run your JavaScript
This note has an interactive demo. Try it in the original If your page builds its content with JavaScript in the browser, the crawlers behind ChatGPT, Claude and Perplexity see an empty shell. They download the HTML, they may even download your bundles, but they don't execute them. Googlebot does r
Posted on Nov 5 • Originally published at richi.codes
This note has an interactive demo. Try it in the original
And these aren't fringe bots. In the month Vercel measured, GPTBot made 569 million fetches and Claude's crawlers 370 million. Together with AppleBot and PerplexityBot that's close to 1.3 billion requests, a little over 28% of Googlebot's volume.
The demo does the same thing in the browser with DOMParser. MDN describes it plainly: when you parse with text/html, <script> elements are marked as non-executable and event handlers don't run, so the resulting document is "essentially inert". That's a decent stand-in for a crawler that only reads HTML:
function readAsBot ( html ) { const doc = new DOMParser (). parseFromString ( html, ' text/html ' ); // scripts are never run const types = [... doc. querySelectorAll ( ' script[type="application/ld+json"] ' )]. map (( s ) => { try { return JSON. parse ( s. textContent )[ ' @type ' ]; } catch { return '? '; } }); doc. querySelectorAll ( ' script, style, template ' ). forEach (( n ) => n. remove ()); return { title: doc. title. trim (), description: doc. querySelector ( ' meta[name="description"] ' )?. content || '', h1: doc. querySelector ( ' h1 ' )?. textContent. trim () || '', jsonLd: types, links: doc. querySelectorAll ( ' a[href] ' ). length, text: ( doc. body?. textContent || '' ). replace ( / \s +/g, ' ' ). trim () }; } Enter fullscreen mode Exit fullscreen mode Two details matter. The JSON-LD has to be read before the <script> elements are removed, because structured data lives inside a script tag even though it's data rather than code. And the browser pane gets its numbers from a nested sandboxed iframe that runs the same HTML for real and reports document.body.innerText back with postMessage.