I built a token killer for AI agents. Here's what it didn't kill.

When an agent reads a web page, it pays for the whole page.

On GitHub's page for psf/requests, that's 123,176 tokens. Only 519 of them are the README. 🤯 The rest is navigation menus, sign-in banners, file lists and scripts.

leantext is an open-source tool (Apache-2.0) that turns what an agent reads into lean Markdown. It strips page chrome, keeps the content, checks the result against the source, and reports tokens before and after. It handles HTML pages, Word, Excel and PowerPoint files, and PDFs, including scanned ones via OCR.

I benchmarked it on well-known pages, then checked line by line what it threw away. Both results are below, including where it still loses data. 👇

Across 7 well-known pages and PDFs: 611,978 → 112,181 tokens. Counts from tiktoken, run on a MacBook on 26 Oct 2026, leantext 0.2.0.

GitHub: psf/requests: 123,176 → 519 (99.6% saved) 🔥 PyPI: requests: 88,741 → 519 (99.4% saved) 🔥 Kubernetes docs: Overview: 129,405 → 8,271 (94% saved) Python docs: Data Structures: 33,914 → 8,423 (75% saved) Python docs: os: 221,301 → 79,578 (64% saved) PDF: Bitcoin whitepaper: 5,273 → 5,033 (5% saved) PDF: Attention Is All You Need: 10,168 → 9,838 (3% saved)

Total: 611,978 → 112,181 tokens. 82% saved. 🏆

The spread is the point. Pages that wrap a short README in a big site template save 99%. Docs pages are mostly code and tables, which leantext keeps, so they save 64–94%. Text PDFs are already lean; a scanned invoice dropped from 2,318 tokens as a page image to 52 as OCR text (98%).

The run also tried 7 more pages. PyPI's numpy and fastapi pages and Wikipedia served bot checks, and GPT-3's PDF, npm and MDN were flagged as suspicious. leantext refused those 6 instead of passing off a partial page. react.dev served plain text, so there was nothing to strip. None of the 7 is in the table.

On 3 of 5 web pages, leantext dropped nothing but page chrome, and it kept every code block on all of them. ✅

Token savings mean nothing if the answer gets thrown away, so I checked. The check takes each original page, pulls out every line of 40+ characters, and looks for it in the lean output. Then I read every missing line by hand.

Kubernetes: Overview: 100% of lines found. Nothing missing. PyPI: requests: 31% found. Only chrome: file hashes, provenance notes, the security disclaimer, the trademark footer. Every README line is there. GitHub: psf/requests: 41% found. Only chrome: Copilot and Enterprise menus, "You signed in with another tab" banners, the star button, the file list. Every README line is there. Python docs: Data Structures: 97% found. 1 footnote, plus the copyright footer. Python docs: os: 98.7% found. About 20 short API descriptions, plus the footer.

The low raw scores on PyPI and GitHub are the checker counting the whole page. leantext drops that material on purpose. The README is short, so a few dozen lines of chrome swing the percentage. For an agent, both pages are complete.

On Python's API reference, leantext loses short definitions, and its own checks don't catch it. Both Python pages passed strict mode.

The pattern: Sphinx wraps each API entry in a

That matters most for docs, because "what does X do?" is exactly what agents ask. Two fixes are next for leantext:

The code-block count is also a weaker check than it looks. On the os page it went from 16 to 21, because leantext turns some inline code into fenced blocks. "Code in is at most code out" passes even when prose is missing.

Install with uv (or pip with the same argument):

uv tool install "leantext[tokens,office] @ git+ https://github.com/shouvik12/leantext "

leantext https://pypi.org/project/requests/ --strict --report > requests.md

The report prints tokens before and after on stderr. With --strict, a page that looks wrong exits with an error instead of handing you a partial result, so your agent can fall back to the original.

Run it on the pages your agent reads most, and tell me what it drops. ⭐ Issues and PRs welcome: github.com/shouvik12/leantext