How to give Claude Code a web scraping tool with Crawl4AI (MCP)

Posted on Oct 25 • Originally published at dedupedai.com

Coding agents are great at research until they need to read a real web page. Crawl4AI is an open-source crawler and scraper that turns websites into clean, LLM-ready Markdown, and it ships an MCP server so Claude Code can scrape, extract and search on its own.

This guide covers both ways to use it: the hosted MCP server (one command in Claude Code) and the free open-source library you run yourself.

Sign up at crawl4ai.com and verify your email. Per the README, the free tier is 1,000 credits every month, 10 requests a minute and 2 at a time, with no card. Paid plans start at $10 a month (120 a minute, 10 at a time). Hosted requests are billed against your key, so keep CRAWL4AI_KEY out of shared scripts.

export CRAWL4AI_KEY = "your-key-here" Enter fullscreen mode Exit fullscreen mode Add the MCP server claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \ --header "Authorization: Bearer $CRAWL4AI_KEY " Enter fullscreen mode Exit fullscreen mode The README links setup for Codex, Cursor and OpenCode too.

Start Claude Code and ask for a page in plain language:

Use crawl4ai to scrape https://docs.crawl4ai.com and summarize the installation options as a table. Enter fullscreen mode Exit fullscreen mode The same key gives you /scrape (page to Markdown), /extract (page to JSON, no LLM key of your own needed), /search (web search), /serp (Google's results page as data, in beta) and /answer (a direct answer with sources, experimental). JS-heavy pages and bot walls are handled for you on the hosted side.

curl -s https://api.crawl4ai.com/scrape \ -H "Authorization: Bearer $CRAWL4AI_KEY " \ -H "Content-Type: application/json" \ -d '{"url": "https://news.ycombinator.com"}' | jq -r.markdown Enter fullscreen mode Exit fullscreen mode Option 2: run it yourself for free The hosted API is optional. The open-source crawler runs free on your machine.

pip install -U crawl4ai crawl4ai-setup # installs and sets up the browser crawl4ai-doctor # checks the installation Enter fullscreen mode Exit fullscreen mode If the browser setup fails, install Chromium by hand with python -m playwright install --with-deps chromium.

import asyncio from crawl4ai import AsyncWebCrawler async def main (): async with AsyncWebCrawler () as crawler: result = await crawler. arun ( url = " https://news.ycombinator.com " ) print ( result. markdown ) asyncio. run ( main ()) Enter fullscreen mode Exit fullscreen mode Save result.markdown per page and you have clean input for a RAG index. For tighter output, the "fit Markdown" filters ( PruningContentFilterLXML, BM25ContentFilter, LLMContentFilter ) strip menus, footers and boilerplate.

crwl https://news.ycombinator.com -o markdown crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 Enter fullscreen mode Exit fullscreen mode Self-hosted MCP with Docker If you want MCP without the hosted API, run your own server. It needs a token; without one it only answers inside its container.

export CRAWL4AI_API_TOKEN = " $( openssl rand -hex 32 ) " docker run -d -p 11235:11235 --name crawl4ai --shm-size = 1g \ -e CRAWL4AI_API_TOKEN = " $CRAWL4AI_API_TOKEN " \ unclecode/crawl4ai:latest Enter fullscreen mode Exit fullscreen mode Allow about 10 seconds for it to start, then test:

curl -s http://localhost:11235/md \ -H "Authorization: Bearer $CRAWL4AI_API_TOKEN " \ -H "Content-Type: application/json" \ -d '{"url": "https://news.ycombinator.com"}' | jq -r.markdown Enter fullscreen mode Exit fullscreen mode The server exposes /md, /html, /crawl, /screenshot, /pdf and /execute_js, plus a dashboard at http://localhost:11235/dashboard and a playground at /playground. The README says the self-hosted server also supports MCP for Claude Code and other agents; the exact connection steps are in the self-hosting guide.

Whichever you choose, respect each site's terms and robots rules when crawling.

Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com