xZeroProtect: Smarter Crawler Rules for SEO and AI Visibility
xZeroProtect is the firewall layer bundled with the Webrium framework , built to keep scanners, exploit tools, and malicious visitors away from your application before they ever reach your code. This update refines how it treats automated traffic that isn't malicious at all: search engines, SEO aud
Posted on Oct 15 • Originally published at webrium.dev
xZeroProtect is the firewall layer bundled with the Webrium framework, built to keep scanners, exploit tools, and malicious visitors away from your application before they ever reach your code. This update refines how it treats automated traffic that isn't malicious at all: search engines, SEO audit tools, and the new generation of AI crawlers that power tools like ChatGPT, Claude, and Perplexity.
Earlier versions blocked these two well-known SEO crawlers outright, treating them the same as aggressive scanning tools. In practice, Semrush and Ahrefs are legitimate services that site owners and marketers rely on every day for backlink audits and competitive research. They're not attack tools, so they no longer sit on the hard-blocked list. Their traffic still passes through the normal rate limiter like any other unverified bot, so a heavy crawl is throttled rather than either blocked outright or let through without limits.
xZeroProtect's trusted-crawler list previously covered search engines (Googlebot, Bingbot, Yandex, Baidu...) and social preview bots (Twitterbot, Slackbot, Discordbot...), verifying each one so a spoofed User-Agent can't sneak through. It had no concept of the crawlers that now matter for how AI assistants discover and cite content on the web.
This update adds recognition for OpenAI's GPTBot and ChatGPT-User, Anthropic's ClaudeBot and anthropic-ai, PerplexityBot, Common Crawl's CCBot, Amazonbot, Google-Extended, and Applebot-Extended. Where a vendor runs on infrastructure that supports it, like Google-Extended and Applebot-Extended, verification reuses the same double-DNS check already trusted for Googlebot and Applebot. The rest are recognized by User-Agent, the same standard already applied to social crawlers that don't publish a verifiable DNS suffix.
Crawler matching works on a first-match basis, checking entries in order. Because "google-extended" and "applebot-extended" both contain the shorter, more generic "google" and "applebot" strings, they were being silently matched against the older, generic entries before ever reaching their own dedicated rule — any site that added them exactly as documented would have seen no effect at all. Both entries were reordered ahead of their generic counterparts so the match actually resolves as intended.
A firewall that blocks indiscriminately isn't more secure, just less useful. The goal of xZeroProtect has always been to stop real threats — scanners, exploit kits, credential stuffing — without getting in the way of the traffic a site actually wants. SEO tools help site owners understand and improve their own sites. AI crawlers increasingly decide whether a page gets cited in an AI-generated answer at all, which is quickly becoming as important as a search ranking. Treating both as hostile by default was overly cautious at the cost of visibility.
At the same time, trust is still extended carefully. Crawlers that can be verified through DNS still are. Crawlers that can't be, like Semrush, Ahrefs, or most AI bots, are let through without a blanket pass that would let an attacker spoof their way around rate limiting and payload scanning. This update widens what's allowed through the door without weakening the lock on it.