I Gave My AI Agents Their Own Documentation Crawler, and Pulled 60 Pages of Clean Markdown in 49 Seconds

My AI agents were flying blind. They could write code, but when they needed to reference library docs, they either hallucinated APIs or stalled waiting for me to paste documentation. I needed a way for them to fetch and convert web docs into clean Markdown on demand. So I built a lightweight crawler that runs inside the agent's tool loop. It pulled 60 pages of Python docs and converted them to Markdown in 49 seconds. Here's how it works.

Agents are only as good as their context. When an agent needs to use a library like requests or pydantic, it often relies on stale training data. The result: wrong function signatures, deprecated parameters, and broken code. I wanted my agents to fetch current documentation, convert it to Markdown (which LLMs handle well), and use it as context. But existing crawlers are heavy, require external services, or output messy HTML.

The agent calls it as a tool: crawl_docs(url, max_pages=60). The crawler returns a list of Markdown documents. The agent then uses those documents for RAG-style retrieval.

pip install requests beautifulsoup4 html2text Enter fullscreen mode Exit fullscreen mode The Crawler import requests from bs4 import BeautifulSoup from urllib.parse import urljoin, urlparse from concurrent.futures import ThreadPoolExecutor, as_completed import html2text import time from collections import deque class DocsCrawler: def __init__ ( self, base_url, max_pages = 60, max_workers = 8, delay = 0.1 ): self. base_url = base_url self. base_domain = urlparse ( base_url ). netloc self. max_pages = max_pages self. max_workers = max_workers self. delay = delay self. visited = set () self. queue = deque ([ base_url ]) self. results = [] self. session = requests. Session () self. session. headers. update ({ ' User-Agent ': ' DocsCrawler/1.0 (+https://example.com/bot) ' }) def is_valid ( self, url ): parsed = urlparse ( url ) return ( parsed. netloc == self. base_domain and parsed. scheme in ( ' http ', ' https ' ) and not parsed. path. endswith (( '.pdf ', '.zip ', '.png ', '.jpg ' )) ) def fetch ( self, url ): try: time. sleep ( self. delay ) resp = self. session. get ( url, timeout = 10 ) resp. raise_for_status () return resp. text except Exception as e: print ( f " Failed { url }: { e } " ) return None def extract_links ( self, html, current_url ): soup = BeautifulSoup ( html, ' html.parser ' ) links = set () for a in soup. find_all ( ' a ', href = True ): href = a [ ' href ' ] full_url = urljoin ( current_url, href ) if self. is_valid ( full_url ): links. add ( full_url ) return links def html_to_markdown ( self, html ): h = html2text. HTML2Text () h. ignore_links = False h. ignore_images = True h. body_width = 0 return h. handle ( html ) def crawl ( self ): with ThreadPoolExecutor ( max_workers = self. max_workers ) as executor: futures = {} while self. queue and len ( self. visited ) < self. max_pages: # Fill the executor with pending URLs while self. queue and len ( futures ) < self. max_workers: url = self. queue. popleft () if url in self. visited: continue self. visited. add ( url ) futures [ executor. submit ( self. fetch, url )] = url # Process completed futures for future in as_completed ( futures ): url = futures. pop ( future ) html = future. result () if not html: continue md = self. html_to_markdown ( html ) self. results. append ({ ' url ': url, ' markdown ': md }) for link in self. extract_links ( html, url ): if link not in self. visited: self. queue. append ( link ) if len ( self. visited ) >= self. max_pages: break return self. results Enter fullscreen mode Exit fullscreen mode Usage if __name__ == ' __main__ ': start = time. time () crawler = DocsCrawler ( ' https://docs.python.org/3/library/ ', max_pages = 60 ) docs = crawler. crawl () elapsed = time. time () - start print ( f " Crawled { len ( docs ) } pages in { elapsed:.

2 f } seconds " ) # Save to files for i, doc in enumerate ( docs ): with open ( f ' doc_ { i }.md ', ' w ' ) as f: f. write ( f " # Source: { doc [ ' url ' ] } \n\n { doc [ ' markdown ' ] } " ) Enter fullscreen mode Exit fullscreen mode Integrating with an Agent The agent loop calls the crawler as a tool. Here's a simplified agent loop with explicit termination:

def agent_loop ( task, max_iterations = 10 ): context = [] for i in range ( max_iterations ): action = llm_decide ( task, context ) if action [ ' type ' ] == ' crawl_docs ': docs = crawler. crawl () context. extend ( docs ) elif action [ ' type ' ] == ' answer ': return action [ ' content ' ] else: context. append ( f " Unknown action: { action } " ) return " Max iterations reached " Enter fullscreen mode Exit fullscreen mode Security note: Never execute arbitrary code from crawled pages. The crawler only extracts text. If you use eval or exec on Markdown content, you open yourself to code injection. Always sanitize and treat crawled content as untrusted input.

The full code is on GitHub. Happy crawling!