A web crawler for AI agents with no LLM in the crawl loop

When an AI agent needs an answer that lives a few clicks deep on a website, there are two common ways to get it.

I wanted the accuracy of the second option at a price closer to the first. So I built jevcrawl, an open-source (Apache-2.0) MCP server and CLI where a small scoring model drives the crawl and no LLM sits inside the crawl loop.

You give it a question and a start URL. It fetches as few pages as it can and returns only the passages that answer the question, copied verbatim with their source URLs.

uvx jevcrawl crawl https://docs.python.org/3/ \ --goal "What does the walrus operator:= do and which Python version added it?" Enter fullscreen mode Exit fullscreen mode On that example it fetched 6 pages and cut about 80,000 tokens of page text down to about 1,900 tokens of answers. That took 11 scoring calls, cost about $0.007, and finished in 8 seconds.

The decisions come from Jev, a small model from TypeSafe that only answers yes/no questions with a probability. For each page, jevcrawl sends one request:

Plain code does the rest. It keeps the high-scoring passages, orders the links by priority, and asks once more whether the kept passages already answer the goal, so it can stop early.

Because the model only returns numbers, it cannot invent text, and a page cannot give it instructions. At worst, a prompt-injection attempt biases a score.

I used WebWalkerQA (Wu et al., Alibaba Tongyi Lab, ACL 2025), a benchmark of questions buried several clicks deep in real websites. I re-verified 50 tasks against the live sites and did not look at them during development. Every system used the same answering LLM and the same judge.

I also compared against Crawl4AI's best-first deep crawl, a keyword-scored crawler with no LLM. On 20 other tasks, measuring only whether the crawler fetched the answer page, jevcrawl reached it on 10 of 20 and Crawl4AI on 5 of 20. The median time was 29 s against 43 s.

These are small samples (50 and 20 tasks). I trust the cost gap more than the accuracy gap, and the Crawl4AI test measures reaching the page, not final answers. The benchmark code is in the repo's bench/ folder, so you can run it or try your own tasks.

claude mcp add jevcrawl -e TYPESAFE_API_KEY = <your key> -- uvx jevcrawl Enter fullscreen mode Exit fullscreen mode It also works in Claude Desktop and Cursor through a standard mcpServers entry; the README has the JSON.

The repo is at github.com/RayenMlayeh/jevcrawl. There are several good-first-issues open: robots.txt support, a Dockerfile, Windows and macOS CI, and more.

I would like to know which sites it does badly on. Tell me in the comments, or open an issue with the URL and the question.