Which AI crawler is actually loading your server, and which one to rate-limit first

A site slows down, someone opens the access log, sees a wall of bot requests, and blocks them. Two days later the site is slow again and the list of bots has changed. The step that got skipped is the one that makes the answer stable: pricing what each crawler actually costs before you change anything.

This is the sequence I'd run on a WordPress box, and it works the same on any LAMP-style stack. It ends with a ranked list and one change to make, instead of a firewall full of guesses.

cat /proc/loadavg top -b -n1 -o %CPU | head -15 Enter fullscreen mode Exit fullscreen mode If the top CPU consumers are php-fpm and mysqld, HTTP requests caused this and the access log holds the evidence. If you see heavy kswapd, kcompactd, or %wa near 100, memory or disk is the problem — rate-limiting a crawler buys you a few days and nothing else. Look at the PHP-FPM pool as well. All children busy with a non-zero listen queue means requests are queueing, so continue. Idle children while load is high means the time is going somewhere else: cron, a CLI import, a backup job.

The default combined format gives you bytes and user agent but no duration, and duration is what separates a crawler making 200 cheap requests from one making 200 expensive ones.

log_format main_ext ' $remote_addr - $remote_user [ $time_local ] " $request " ' ' $status $body_bytes_sent " $http_referer " " $http_user_agent " ' 'rt= $request_time '; access_log /var/log/nginx/access.log main_ext; Enter fullscreen mode Exit fullscreen mode nginx -t && systemctl reload nginx Enter fullscreen mode Exit fullscreen mode Apache: add %D to a custom LogFormat; it's the request duration in microseconds. Existing lines keep the old format, so give it an hour before reading.

Both nginx and Apache combined quote the request, the referer and the user agent. That makes " a reliable field separator, and the UA the sixth field:

awk -F '"' '{print $6}' /var/log/nginx/access.log \ | sort | uniq -c | sort -rn | head -20 Enter fullscreen mode Exit fullscreen mode You'll get the usual mix: uptime monitors, search engines, your own scripts, and whatever AI crawlers your content attracted. Note the candidates, then move to cost.

Hit counts flatter crawlers that are noisy but cheap. Multiply by time instead:

awk -F '"' ' { split($3, s, " ") # s=status, s=body bytes ua = $6 rt = $NF; sub(/^rt=/, "", rt) hits[ua]++ bytes[ua] += s secs[ua] += rt } END { for (u in hits) printf "%10.1f %8d %13d %s\n", secs[u], hits[u], bytes[u], u }' /var/log/nginx/access.log | sort -rn | head -20 Enter fullscreen mode Exit fullscreen mode Columns: seconds of request handling, hits, bytes served, user agent. A crawler at 8,000 hits and 40 seconds is background noise. One at 900 hits and 600 seconds is your problem, and it's probably hitting something expensive.

If the file is too large to parse in one pass, sample it:

shuf -n 500000 access.log > sample.log Enter fullscreen mode Exit fullscreen mode Ratios survive sampling. Absolute counts don't, so don't quote them anywhere.

awk -F '"' '$6 ~ /BotName/ {split($2, r, " "); print r}' access.log \ | sort | uniq -c | sort -rn | head -20 Enter fullscreen mode Exit fullscreen mode Replace BotName with a token from the previous step. What you're looking for:

If the expensive paths are the same across several bots, fix the paths first. It's a smaller change and it applies to every visitor.

A user agent is a claim, and a throttle is cheap to test. Short-circuit one UA temporarily:

# temporary, inside the server block if ( $http_user_agent ~ * "BotName") { return 403; } Enter fullscreen mode Exit fullscreen mode Reload, wait 20–30 minutes, and compare load average, PHP-FPM queue depth and database CPU against the same window before the change. If nothing moved, that bot wasn't the cause and you can drop the rule. Then check whether the UA is genuine rather than a scraper wearing it:

dig +short -x 203.0.113.10 # reverse lookup dig +short crawl.example.com # forward lookup, should return the same IP Enter fullscreen mode Exit fullscreen mode Forward-confirmed reverse DNS is what separates the real crawler from someone running curl with a copied header. It matters most right before you write a permanent rule.

map $http_user_agent $bot_key { default ""; ~*GPTBot "gptbot"; ~*ClaudeBot "claudebot"; } limit_req_zone $bot_key zone=ai_bots:10m rate=30r/m; server { limit_req zone=ai_bots burst=20 nodelay; limit_req_status 429; #... } Enter fullscreen mode Exit fullscreen mode Requests with an empty key aren't counted, so ordinary visitors are untouched. Use the tokens you actually saw in your own log, not the two above. If your build exposes $limit_req_status, add it to the log format and you can confirm the limits are firing instead of assuming.

robots.txt sits third for a reason: it's advisory. Some crawlers honour it, some don't, and the only way to know which happened is to read the log a day later. Write the file, then verify it, rather than reporting it as done.

Run the cost query again over a window after the change. You want the same shape of answer as before: which UA is first, and how many seconds it now costs. If the top entry has changed, you fixed the last problem and this one was already there waiting.

Wrap the whole sequence in a small script and keep it. One command that prints seconds, hits and bytes per user agent, sorted, is worth more than a firewall rule nobody can explain six months from now.

If you'd rather not maintain the parser: AumCrawl is a free plugin that records which AI crawlers are hitting a WordPress site and lets you allow or block them individually from the admin. It sits next to the log work above rather than replacing it — the log is still what tells you what each bot costs.