How to Audit AI Crawler Traffic and Identify Spoofed Bots in Logs

Posted on Oct 31 • Originally published at py-forge.com

Unmanaged web scraping by artificial intelligence agents can consume server bandwidth, increase egress costs, and skew operational analytics. While legitimate crawlers identify themselves via their request headers, malicious scrapers often spoof those same headers to bypass rules or scrape protected assets under the guise of well-known automated systems.

Analyzing access logs manually across multiple servers is inefficient. You need a systematic way to parse server logs, match known crawler signatures, estimate hosting expenses tied to crawler bandwidth, and verify whether the IP addresses sending those headers actually belong to the claimed crawler networks.

The script aibot_ledger_crawler_log_reporter.py addresses this issue directly. It inspects web server access logs, identifies automated AI crawlers, performs Forward-Confirmed Reverse DNS lookups to catch spoofed clients, checks paths against your robots exclusion file, and generates server-level mitigation templates.

The script relies entirely on standard modules included with Python, such as urllib.robotparser, socket, gzip, and json. You do not need to install any third-party libraries using pip.

To get started, save the source code into a file named aibot_ledger_crawler_log_reporter.py.

You can view the command-line help interface by running:

python aibot_ledger_crawler_log_reporter.py --help Enter fullscreen mode Exit fullscreen mode Because the core processing logic is encapsulated in the AIBotLedger class, you can also import and execute the ledger directly within your own administrative scripts:

from aibot_ledger_crawler_log_reporter import AIBotLedger # Initialize the ledger with custom bandwidth pricing and DNS verification ledger = AIBotLedger ( cost_per_gb = 0.09, verify_dns = True ) # Ingest standard or compressed log files ledger. ingest_file ( " /var/log/nginx/access.log " ) ledger. ingest_file ( " /var/log/nginx/access.log.1.gz " ) # Print the formatted terminal summary print ( ledger. generate_terminal_summary ()) Enter fullscreen mode Exit fullscreen mode How It Works The tool is organized around log parsing, pattern matching, DNS validation, and rules checking.

The ingest_file function checks the file extension. If the target file ends in.gz, it transparently reads the stream via Python's built-in gzip module.

Log entries are processed by parse_log_line, which detects both standard Combined log entries (typical for Nginx and Apache) and structured JSON log lines (typical for Caddy):

def parse_log_line ( line ): """ Parses standard Combined format or Caddy JSON logs. """ line = line. strip () if not line: return None # Try Caddy JSON log format if line. startswith ( " { " ) and line. endswith ( " } " ): try: data = json. loads ( line ) req = data. get ( " request ", {}) headers = req. get ( " headers ", {}) ua_list = headers. get ( " User-Agent ", [ "" ]) ua = ua_list [ 0 ] if isinstance ( ua_list, list ) and ua_list else str ( ua_list ) ip = req. get ( " remote_ip " ) or req. get ( " client_ip " ) or "" if ": " in ip and not ip. startswith ( " [ " ): # Handle host:port ip = ip. rsplit ( ": ", 1 )[ 0 ] return { " ip ": ip, " time ": data. get ( " ts ", "" ), " method ": req. get ( " method ", " GET " ), " path ": req. get ( " uri ", " / " ), " status ": int ( data. get ( " status ", 200 )), " bytes ": int ( data. get ( " size ", 0 ) or 0 ), " ua ": ua, } except Exception: pass # Try Combined / Nginx / Apache format m = COMBINED_LOG_REGEX. match ( line ) if m: d = m. groupdict () b = d [ " bytes " ] bytes_val = int ( b ) if b. isdigit () else 0 return { " ip ": d [ " ip " ], " time ": d [ " time " ], " method ": d [ " method " ], " path ": d [ " path " ], " status ": int ( d [ " status " ]), " bytes ": bytes_val, " ua ": d [ " ua " ], } return None Enter fullscreen mode Exit fullscreen mode 2. Signature Detection The script maintains an internal collection of bot definitions named BOT_PROFILES.

It matches user-agent values against regular expressions corresponding to entities such as GPTBot, ClaudeBot, Bytespider, CCBot, Google-Extended, PerplexityBot, Cohere-AI, and Diffbot. Each entry maps the signature to a classification category (such as Live Browsing, Dataset Mining, or Aggressive Scraper) and assigns legitimate root domains used for verification.

Anyone can configure an HTTP client to send User-Agent: GPTBot. To prove authenticity, the script uses Forward-Confirmed Reverse DNS via verify_fcrdns:

def verify_fcrdns ( ip, expected_domains, timeout = 2.0 ): """ Forward-Confirmed Reverse DNS (FCrDNS) check. Returns: ' VERIFIED ', ' SPOOFED ', ' UNVERIFIED ', or ' SKIPPED '. """ if not expected_domains: return " SKIPPED " if ip in _DNS_CACHE: return _DNS_CACHE [ ip ] # Don't resolve private or special addresses if ip. startswith (( " 127.

", " 10. ", " 192.168. ", " 172.16.

", " 172.17. ", " 172.18. ", " 172.19.

", " 172.20. ", " 172.21. ", " 172.22.

", " 172.23. ", " 172.24. ", " 172.25.

", " 172.26. ", " 172.27. ", " 172.28.

", " 172.29. ", " 172.30. ", " 172.31.

" )): result = " SKIPPED " _DNS_CACHE [ ip ] = result return result orig_timeout = socket. getdefaulttimeout () try: socket. setdefaulttimeout ( timeout ) try: hostname, _, _ = socket. gethostbyaddr ( ip ) except ( socket. herror, socket. gaierror, socket. timeout, OSError ): result = " SPOOFED " _DNS_CACHE [ ip ] = result return result hostname_lower = hostname. lower () domain_matched = any ( hostname_lower. endswith ( domain. lower ()) for domain in expected_domains ) if not domain_matched: result = " SPOOFED " _DNS_CACHE [ ip ] = result return result try: forward_ips = socket. gethostbyname_ex ( hostname )[ 2 ] if ip in forward_ips: result = " VERIFIED " else: result = " SPOOFED " except ( socket. gaierror, socket. timeout, OSError ): result = " SPOOFED " finally: socket. setdefaulttimeout ( orig_timeout ) _DNS_CACHE [ ip ] = result return result Enter fullscreen mode Exit fullscreen mode This verification process consists of three distinct checks:

If any check fails, the record is flagged as SPOOFED and recorded for auditing.

By accumulating byte counts associated with verified and unverified crawlers, AIBotLedger multiplies total gigabytes transferred by the cost_per_gb parameter to provide an estimate of hosting bandwidth charges.

It also includes standalone helper functions—such as generate_nginx_snippets and generate_robots_snippet —to output firewall maps, rate-limiting directives, and exclusion lines targeted at the specific crawler names encountered in your logs.

While practical for audits, the current code has several built-in constraints:

Does running this script require elevated root privileges? No. The script only reads log files from disk and performs standard outbound DNS resolution queries. As long as your operating system user has read permissions for the target log files, root or administrative access is not required.

Can the script process logs stored in.gz archives alongside plain text logs? Yes. The ingest_file method inspects the file name. If it detects a.gz suffix, it uncompresses and reads the file stream via Python's gzip module without requiring prior extraction on disk.

Why does an IP show up as spoofed even if the user agent looks valid? A request is marked as spoofed when the client supplies a known AI crawler user agent, but its IP address either has no reverse DNS entry, resolves to a domain outside the provider's authorized domain list, or fails forward verification when matching the hostname back to the originating IP address.

Full version and the rest of the toolkit: AIBot Ledger crawler log reporter on PyForge. Product page: https://py-forge.com/products/aibot-ledger-crawler-log-reporter