I built CrawlGuard: A free tool to block aggressive AI crawlers and protect server bandwidth
Hey everyone! 馃憢 Over the last few months, I noticed a huge spike in aggressive web crawlers like Bytespider , GPTBot , and ClaudeBot hitting web servers. Unlike Google or Bing, many of these AI training scrapers ignore standard crawl delays and scrape content without sending any referral traffic b
Over the last few months, I noticed a huge spike in aggressive web crawlers like Bytespider, GPTBot, and ClaudeBot hitting web servers. Unlike Google or Bing, many of these AI training scrapers ignore standard crawl delays and scrape content without sending any referral traffic back.
Manually looking up user-agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard.
馃憠 Try CrawlGuard Free at crawlguard.dev
I鈥檇 love to hear your feedback on the UI, and if there are any specific malicious or rogue crawlers you'd like me to add to the registry!