CrawlGuard: Kostenloses Tool sperrt aggressive KI-Crawler vom eigenen Server aus
Ein Entwickler hat CrawlGuard vorgestellt, ein kostenloses Tool gegen Bots wie Bytespider, GPTBot und ClaudeBot. Viele dieser KI-Scraper ignorieren übliche Crawl-Delays und erzeugen Meanwhile keinen Gegenwert in Form von Traffic.
Over the last few months, I noticed a huge spike in aggressive web crawlers like Bytespider, GPTBot, and ClaudeBot hitting web servers. Unlike Google or Bing, many of these AI training scrapers ignore standard crawl delays and scrape content without sending any referral traffic back.
Manually looking up user-agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard.
👉 Try CrawlGuard Free at crawlguard.dev
I’d love to hear your feedback on the UI, and if there are any specific malicious or rogue crawlers you'd like me to add to the registry!