Spotting AI Crawlers in Your Server Logs With a Short Python Script

Before you optimize for AI search, check whether AI systems can reach your site at all. Server logs answer that without any paid tool. This post walks through a small script that counts requests from well-known AI-related user agents and flags blocked responses.

Providers publish their crawler names. Common examples include GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended (a control token rather than a crawler), and Bingbot, which feeds several assistants. Names change, so keep the list in a config file and review it quarterly.

Read your access log line by line, extract the user agent, status code, and path, and aggregate counts. A few lines of Python with a regular expression for the combined log format is enough:

User-agent strings can be spoofed. For important decisions, verify IP ranges published by the provider, or use your CDN's verified-bot features. Do not block or allow based on the string alone.

You may want search-oriented crawlers to read your public pages while keeping training crawlers out, or the opposite. Both are legitimate. Write the policy down, then express it in robots.txt and in CDN rules so they agree.

Group requests by path prefix. If crawlers hit your blog but never your pricing or services pages, check internal links and sitemap coverage. Pages that are never fetched cannot be cited.

Save the weekly counts in a sheet. Changes after a CDN update or a robots.txt edit show up immediately, which makes debugging much faster than guessing.

Schedule the script as a weekly job, send the table to the marketing channel, and agree on who responds when blocked requests spike. Ten minutes a week of attention is usually enough to catch an accidental firewall rule before it costs a quarter of visibility.

For context on why crawl access matters as AI assistants break one question into many, read this explainer on query fan-out: notes on query fan-out and AI search crawling.

Published by PandaWM, an overseas marketing team based in Hong Kong.