How to Stop Bad Bots and AI Scrapers

Automated scrapers, AI training crawlers, and credential stuffing bots account for over 40% of all internet traffic.

Allowing bots to scrape your web application without restrictions inflates hosting bills, degrades performance for real users, and drains database connection pools.

In this tutorial, we are going to use Aegis —an open-source, self-hosted Web Application Firewall (WAF) and reverse proxy—to configure automated bot defense, manage AI scrapers, and deploy client-side cryptographic browser challenges.

Aegis evaluates incoming client fingerprints, request headers, and traffic behavioral patterns in real time.

Manage how automated crawlers access your site:

For requests that exhibit suspicious behavior or exceed baseline request velocities:

Test the endpoint with an automated CLI client:

curl -i -A "GPTBot/1.0" http://localhost:8080/ Enter fullscreen mode Exit fullscreen mode Response:

HTTP / 1.1 403 Forbidden Content-Type: text/html; charset=utf-8 X-Aegis-Action: bot-blocked Access denied: Automated AI scraper blocked by security policy. Enter fullscreen mode Exit fullscreen mode The request is rejected at the ingress proxy and never reaches your origin web server.

The Community Edition is free to self-host: