Turn any web page into clean markdown for ChatGPT or Claude

It is open source under AGPL-3.0 (the SDKs are MIT) and also runs as a hosted service. The examples below use the hosted API, which needs a key from firecrawl.dev. The README links a self-hosting guide if you would rather run it yourself.

Sign up at firecrawl.dev to get your API key. It starts with fc-. Put it in an environment variable rather than in a file you commit:

export FIRECRAWL_API_KEY = "fc-YOUR_API_KEY" Enter fullscreen mode Exit fullscreen mode Step 2: scrape a page to markdown Python

pip install firecrawl Enter fullscreen mode Exit fullscreen mode import os from firecrawl import Firecrawl app = Firecrawl ( api_key = os. environ [ " FIRECRAWL_API_KEY " ]) doc = app. scrape ( " https://firecrawl.dev ", formats = [ " markdown " ]) print ( doc. markdown ) Enter fullscreen mode Exit fullscreen mode Node.js

npm install firecrawl Enter fullscreen mode Exit fullscreen mode import { Firecrawl } from ' firecrawl '; const app = new Firecrawl ({ apiKey: process. env. FIRECRAWL_API_KEY }); const doc = await app. scrape ( ' https://firecrawl.dev ', { formats: [ ' markdown ' ] }); Enter fullscreen mode Exit fullscreen mode curl

curl -X POST 'https://api.firecrawl.dev/v2/scrape' \ -H "Authorization: Bearer $FIRECRAWL_API_KEY " \ -H 'Content-Type: application/json' \ -d '{ "url": "firecrawl.dev" }' Enter fullscreen mode Exit fullscreen mode CLI, with only the main content of the page:

firecrawl https://firecrawl.dev --only-main-content Enter fullscreen mode Exit fullscreen mode The output is plain markdown: headings, lists and text, without the page chrome. That is what you paste into a prompt or split into chunks for retrieval.

The same client covers the jobs that usually come next:

# Find URLs on a site, ordered by relevance to a word result = app. map ( " https://firecrawl.dev ", search = " pricing " ) # Scrape several URLs at once job = app. batch_scrape ([ " https://firecrawl.dev ", " https://docs.firecrawl.dev ", ], formats = [ " markdown " ]) # Crawl a docs site (the SDK waits for the job to finish) docs = app. crawl ( " https://docs.firecrawl.dev ", limit = 50 ) for d in docs. data: print ( d. metadata. source_url, d. markdown [: 100 ]) Enter fullscreen mode Exit fullscreen mode Crawling a whole docs site this way is the fastest route to a knowledge base for a docs chatbot.

If you want your coding agent to read current docs instead of guessing from training data, install Firecrawl as a skill and restart the agent:

npx -y firecrawl-cli@latest init --all --browser Enter fullscreen mode Exit fullscreen mode Or add it as an MCP server in any MCP client:

{ "mcpServers": { "firecrawl-mcp": { "command": "npx", "args": [ "-y", "firecrawl-mcp" ], "env": { "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY" } } } } Enter fullscreen mode Exit fullscreen mode That config holds your key, so keep it out of shared or committed config files.

Compare Firecrawl with other options on best AI repos for web scraping for LLMs, or see every Firecrawl use case, including giving your coding agent live web access.

Want the setup commands for 50 AI repos people actually ship with? Get the free AI Repos That Actually Ship pack: https://dedupedai.com/ship-pack/?utm_source=devto&utm_medium=article&utm_campaign=shippack