Same page, two outputs: Markdown vs structured JSON for AI agents

I work on WebExtract, a pay-per-call API that turns a URL into data. It has two main endpoints: GET /v1/markdown returns clean Markdown, and GET /v1/extract returns the same Markdown plus fixed-schema fields ( product, tables, metadata ). This post runs one product page and one table page through both and shows what an agent actually gets back. Every snippet below is copied from saved API responses in the examples/compare folder, not written by hand.

One note on where the files came from: they were captured through the Base Sepolia sandbox routes ( /sandbox/v1/..., paid with test USDC) on 2026-10-25 (UTC). The sandbox runs the same extraction code as the live /v1/... routes.

The page is https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html, a practice site built for scraping. The page itself says so: "Prices and ratings here were randomly assigned and have no real meaning."

/v1/markdown returned 1567 characters, 261 words. The end of it looks like this:

## Product Information | UPC | a897fe39b1053632 | | --- | --- | | Product Type | Books | | Price (excl. tax) | £51.77 | | Price (incl. tax) | £51.77 | | Tax | £0.00 | | Availability | In stock (22 available) | | Number of reviews | 0 | Enter fullscreen mode Exit fullscreen mode This is good input for a model that only needs to read the page. It is short, and the price, stock and table are all there. But look at the table. The HTML table has no header row, and a Markdown table needs one, so the first data row ( UPC ) ends up in the header position. A model will probably read it correctly. A program that splits on pipes and treats row one as column names will not.

/v1/extract returned the same Markdown (the two strings are identical) plus extra fields. Here is a jq selection of the product object:

jq '.product | {name, price, currency, availability, rating, source}' out/product.extract.json Enter fullscreen mode Exit fullscreen mode { "name": "A Light in the Attic", "price": 51.77, "currency": "GBP", "availability": "InStock", "rating": { "value": 3, "count": null }, "source": "heuristic" } Enter fullscreen mode Exit fullscreen mode price is a number, currency is a code, and availability is a fixed value. Your code can compare or store these without any regex.

The important field is source. This page has no JSON-LD, OpenGraph product tags or microdata, so the price came from page heuristics. The response says so instead of hiding it:

"evidence": "first [class*=price] element; same amount in table row \" Price (excl. tax) \" " Enter fullscreen mode Exit fullscreen mode [ "heuristic_price_low_confidence", "currency_inferred_from_symbol" ] Enter fullscreen mode Exit fullscreen mode That is product.price_detail.prices.evidence and product.price_detail.warnings. The evidence tells you where the number came from (and that the product table shows the same amount). The warnings tell you the price came from heuristics and the currency was inferred from the £ symbol. If your agent is about to act on a price, check price_detail.warnings first. We keep these warnings on purpose, even when the number turns out to be right, because heuristics can be wrong on other pages.

[ "col_1", "col_2" ] [[ "UPC", "a897fe39b1053632" ],[ "Product Type", "Books" ],[ "Price (excl. tax)", "£51.77" ]] { "row_count": 7, "column_count": 2, "consistent_width": true } Enter fullscreen mode Exit fullscreen mode Those are tables.headers, the first three rows, and checks. Since the page has no header row, the headers are placeholders ( col_1, col_2 ) and UPC stays a data row. The checks object lets you confirm the shape before using it: 7 rows, 2 columns, every row the same width.

The second page is https://webscraper.io/test-sites/tables, a practice page with two standard HTML tables. /v1/markdown returned 508 characters, 124 words, and here the Markdown tables look fine, because the page has real <thead> rows.

The difference is what /v1/extract adds. Here is tables, verbatim:

{ "index": 0, "caption": null, "headers": [ "#", "First Name", "Last Name", "Username" ], "rows": [ [ "1", "Mark", "Otto", "@mdo" ], [ "2", "Jacob", "Thornton", "@fat" ], [ "3", "Larry", "the Bird", "@twitter" ] ], "checks": { "row_count": 3, "column_count": 4, "consistent_width": true } } Enter fullscreen mode Exit fullscreen mode In the Markdown the two tables sit one after the other with nothing between them. In the JSON they are tables and tables, each with its own headers and checks, so you don't have to work out where one ends and the next starts.

This is not a product page, and product is simply null. Nothing is guessed. The response also explains why:

{ "reason": "no_product_markup", "detail": "no JSON-LD Product/ProductGroup, product:price meta, schema.org microdata, or price+title+cart heuristic match" } Enter fullscreen mode Exit fullscreen mode That is the product_unresolved field. A null with a reason is easier to handle than an empty-looking object.

Both pages also get a metadata object. A selection for the table page:

{ "title": "Table Playground | Web Scraper Test Sites", "language": "en", "canonical_url": "https://webscraper.io/test-sites/tables", "type": "website" } Enter fullscreen mode Exit fullscreen mode The full object also has description, authors, published_at, modified_at, site_name and image. Fields the page doesn't provide are null or empty, not filled in.

Some limits to know about: product comes from JSON-LD, OpenGraph, microdata or, as a last resort, page heuristics. product.rating is filled only when the page has JSON-LD or star-rating markup. The image URL is kept as the page wrote it (../../media/... here), so you may need to resolve it against final_url.

On REST, the free tier is opt-in: 20 free calls per day for each caller IP address, no signup. Add the header X-Billing: free:

curl -H "X-Billing: free" "https://api.webextract.dev/v1/extract?url=https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" Enter fullscreen mode Exit fullscreen mode or the query parameter?free=1 if your client can't set headers:

curl -G "https://api.webextract.dev/v1/markdown" --data-urlencode "free=1" \ --data-urlencode "url=https://webscraper.io/test-sites/tables" Enter fullscreen mode Exit fullscreen mode The free allowance is counted per caller IP address. If your outgoing IP changes between requests (a proxy, or a cloud host behind NAT), the calls are counted separately for each address, so X-Free-Remaining can go up and down. Failed REST fetches are not charged. Without the free opt-in, a key or a payment, the endpoint answers HTTP 402 with x402 payment requirements. You can then pay per call in USDC or use prepaid credits.

The folder also has four one-file clients: compare_rest.py (both endpoints side by side), langchain_tool.py, mcp_client.py (against https://api.webextract.dev/mcp, tools page_markdown and extract_url ) and x402_client.py. The x402 client only runs against the Base Sepolia sandbox with test USDC and refuses mainnet. The LangChain tools were tested by calling them directly, not inside a model-driven agent.

If you run your own pages through both endpoints and the JSON gets something wrong, I'd like to hear about it. Wrong prices and broken tables are the bugs we most want to find.

Disclosure: I'm on the WebExtract team, and WebExtract is our own product. This post was drafted with AI assistance. The JSON and Markdown excerpts are copied from saved API responses in the repository, and I checked them against those files before publishing.