How to Read JavaScript-Rendered Pages from an AI Agent (When Extract Comes Back Empty)

Posted on Oct 16 • Originally published at agentsearchhq.com

Extract fetches the HTML a server sends and turns it into clean markdown. On a server-rendered page, that HTML already holds the article, so the markdown is complete.

Render loads the URL in headless Chromium, lets the scripts run, and returns what the browser ended up with: markdown, text, html, title, links and an optional screenshot. That is the tool for client-rendered pages. It also costs the same $0.005 per call, which is what makes "try Extract, then Render" a sensible default instead of rendering everything.

1. A length threshold. The simplest check: if the markdown is shorter than a few hundred characters, treat it as suspicious. Pick the number yourself and tune it for your sources; it is a local heuristic, not a service field. Some pages really are short (a contact page, a redirect notice), so a short result is a trigger for one Render call, not proof.

3. An empty root container. The classic SPA shell is a single <div id="root"></div> or <div id="app"></div> with nothing inside. That div produces no text at all in markdown, so in practice it looks like a title with no body, or a body that is only navigation labels and a footer. Treat "title present, body essentially empty" as the markdown signature of an empty root div.

Two things to avoid: do not retry Extract on a thin result (it fetches the same HTML again and you pay again), and do not treat a website error as "thin." Errors are a separate branch, covered below.

The function below uses the same call_service wrapper as the deep research agent guide: pay_fetch is your x402 v2 client (stubbed there), and call_service returns either a portal failure or the delivered data. Request and response shapes match that post: both tools take url, formats and max_chars, and return url (final URL after redirects), title, markdown and error (null on success).

Branch on code and retryable, never on the message text. The full list lives in Error handling for AI agent web tools.

Render honors robots.txt. If a site disallows its user agent, Render refuses the URL before fetching anything. Called directly, that is HTTP 422 with ROBOTS_DISALLOWED:

{ "error": { "code": "ROBOTS_DISALLOWED", "message": "robots.txt disallows this URL for AgentSearchRender.", "retryable": false, "request_id": "…" }} Enter fullscreen mode Exit fullscreen mode Through the Pocket portal it surfaces as HTTP 400 UPSTREAM_REJECTED, and there is no charge. The right agent behavior is to accept the answer: the site has asked bots not to load that page, so move on to another source. Do not retry, rotate tools, or try to get around the block.

Render is a fallback, not a default. Skip it when:

$0.005 for a page Extract can read, $0.01 (2 calls) when Render is needed, and nothing when robots.txt refuses Render.

If some content rendered, you get HTTP 200 with TARGET_DEADLINE and partial markdown; keep it and mark it incomplete. If nothing rendered, you get HTTP 422 TARGET_DEADLINE, which is retryable and not charged.

No. It does not sign in, and it respects robots.txt. Pick another source.

Drop read_js_page into any agent tool that already calls Extract. Start with Web Extract, fall back to Web Render, and find URLs with Web Search. More guides are on the AgentSearch blog.

Originally published on the AgentSearch blog.