Your traffic spike might be one AI newsletter bot: find it, verify crawlers, rate-limit the rest

Sammy Straus posted a side project to Hacker News, got about a hundred upvotes, and watched the site get slower for days. The culprit was not GPTBot or ClaudeBot. Nearly half of all traffic came from one client, a service that turns website changes into an AI newsletter, hitting the site many times a minute.

I expect a lot of small sites to hit the same thing, and his write-up is on the Hacker News front page today.

The fix they landed on is the right one: find out who is actually calling you, block the abusers, and rate limit the rest. Here are three tested pieces: a log analyzer that also catches fake "GPTBot" traffic, a robots.txt that slows crawlers without hiding you from AI search, and a Next.js middleware that returns 429s to anyone over budget.

This script reads a standard nginx or Apache access log, sorts every request into a category, and verifies the big crawlers against the IP lists OpenAI, Anthropic and Google publish. It uses only the Python standard library.

#!/usr/bin/env python3 """ Who is really hitting your site? Classify an access log by bot type, check that " GPTBot ", " ClaudeBot " and " Googlebot " come from their published IP ranges, and flag single clients that hammer you. Usage: python3 botlog.py access.log (nginx/Apache " combined " format) """ import collections, ipaddress, json, re, sys, urllib. request LINE = re. compile ( r ' (?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] " (?P<method>\S+) (?P<path>\S+)[^ " ]* " ' r ' (?P<status>\d{3}) (?P<bytes>\d+|-) " [^ " ]* " " (?P<ua>[^ " ]*) "' ) # user-agent token -> (category, published IP list or None) BOTS = { " GPTBot ": ( " AI training ", " https://openai.com/gptbot.json " ), " ClaudeBot ": ( " AI training ", " https://claude.com/crawling/bots.json " ), " CCBot ": ( " AI training ", None ), " Bytespider ": ( " AI training ", None ), " OAI-SearchBot ": ( " AI search ", " https://openai.com/searchbot.json " ), " Claude-SearchBot ": ( " AI search ", " https://claude.com/crawling/bots.json " ), " PerplexityBot ": ( " AI search ", None ), " ChatGPT-User ": ( " AI user fetch ", " https://openai.com/chatgpt-user.json " ), " Claude-User ": ( " AI user fetch ", " https://claude.com/crawling/bots.json " ), " Googlebot ": ( " Search engine ", " https://developers.google.com/static/search/apis/ipranges/googlebot.json " ), " bingbot ": ( " Search engine ", None ), } SCANNER_PATHS = re. compile ( r " wp-(admin|login|content|includes)|phpinfo|\.env|\.git/|xmlrpc|/cgi-bin/ ", re.

I ) _ranges = {} def ranges ( url ): if url not in _ranges: try: with urllib. request. urlopen ( urllib. request. Request ( url, headers = { " User-Agent ": " botlog " }), timeout = 15 ) as r: data = json. load ( r ) _ranges [ url ] = [ ipaddress. ip_network ( p. get ( " ipv4Prefix " ) or p. get ( " ipv6Prefix " )) for p in data [ " prefixes " ]] except Exception as e: print ( f " warning: could not load { url }: { e } ", file = sys. stderr ) _ranges [ url ] = None return _ranges [ url ] def classify ( ip, ua, path ): if SCANNER_PATHS. search ( path ): return " Scanner ", None for token, ( cat, url ) in BOTS. items (): if token. lower () in ua. lower (): if url and ( nets: = ranges ( url )) is not None: addr = ipaddress. ip_address ( ip ) if not any ( addr in n for n in nets ): return " Spoofed bot ", token return cat, token if re. search ( r " bot|crawler|spider|scrapy|python-requests|curl|go-http|headless ", ua, re. I ): return " Other bot ", None return " Human or unknown ", None reqs, size = collections.

Counter (), collections. Counter () per_client, per_minute, spoofed = collections. Counter (), collections.

Counter (), collections. Counter () total = 0 for raw in open ( sys. argv [ 1 ], errors = " replace " ): m = LINE. match ( raw ) if not m: continue total += 1 cat, token = classify ( m [ " ip " ], m [ " ua " ], m [ " path " ]) b = 0 if m [ " bytes " ] == " - " else int ( m [ " bytes " ]) reqs [ cat ] += 1 size [ cat ] += b client = f ' { m [ " ip " ] } { m [ " ua " ][: 60 ] } ' per_client [ client ] += 1 per_minute [( client, m [ " ts " ][: 17 ])] += 1 # dd/Mon/yyyy:HH:MM if cat == " Spoofed bot ": spoofed [( token, m [ " ip " ])] += 1 print ( f " { total } requests \n " ) print ( f " { ' category ': 18 } { ' requests ': > 9 } { ' share ': > 7 } { ' MB ': > 8 } " ) for cat, n in reqs. most_common (): print ( f " { cat: 18 } { n: 9 } { n / total: 7.1 % } { size [ cat ] / 1e6: 8.1 f } " ) print ( " \n Top clients " ) for client, n in per_client. most_common ( 5 ): peak = max ( v for ( c, _ ), v in per_minute. items () if c == client ) print ( f " { n: 7 } req peak { peak: 4 } /min { client } " ) if spoofed: print ( " \n Claims to be a known bot but is outside its published IP ranges " ) for ( token, ip ), n in spoofed. most_common ( 10 ): print ( f " { n: 7 } req { token: 12 } { ip } " ) Enter fullscreen mode Exit fullscreen mode I generated a test log of 6,330 requests that mirrors the pattern in the post: real GPTBot, ClaudeBot and Googlebot traffic from addresses inside their published ranges, a fake GPTBot from an address outside them, a vulnerability scanner probing /wp-content/phpinfo.php, a change-monitoring service polling an API endpoint, and normal browsers. The output:

6330 requests category requests share MB Human or unknown 4000 63.2% 59.7 AI training 1600 25.3% 51.5 Spoofed bot 300 4.7% 9.8 Search engine 250 3.9% 8.1 Scanner 180 2.8% 0.0 Top clients 2600 req peak 13/min 3.91.20.77 SiteWatch/2.3 (+https://example-monitor.io; AI change digest 900 req peak 6/min 20.171.206.15 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatibl 700 req peak 6/min 216.73.216.40 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatibl 300 req peak 3/min 45.83.12.9 Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gpt 250 req peak 6/min 66.249.66.1 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) Ap Claims to be a known bot but is outside its published IP ranges 300 req GPTBot 45.83.12.9 Enter fullscreen mode Exit fullscreen mode

Two things to notice. The heaviest client, with 2,600 requests, lands in "Human or unknown" because its user agent never says "bot". Only the top clients list exposes it. And 300 requests claim to be GPTBot but come from an address that is not in OpenAI's published list, so they are not OpenAI at all.

Each AI company runs separate bots for training, for search and for fetching a page when a user asks. They are independent, and blocking the wrong one costs you visibility:

Here is the version that keeps search bots and keeps crawlers out of your API:

User-agent: ClaudeBot Crawl-delay: 5 Disallow: /api/ User-agent: GPTBot Disallow: /api/ User-agent: * Disallow: /api/ Enter fullscreen mode Exit fullscreen mode A crawler obeys only the most specific group that names it, so ClaudeBot and GPTBot ignore the * group. That is why each named group repeats Disallow: /api/. Crawl-delay is a hint, not a guarantee: Google does not support it, and OpenAI does not document it, so GPTBot is slowed by the limiter in step 3. Anything that ignores robots.txt needs the next step.

A token bucket per client, with different budgets for search engines, AI search, AI training crawlers and unknown bots:

// lib/bot-limit.ts: a token bucket per client, tuned per kind of visitor. export type Rule = { perMinute: number; burst: number }; export const RULES: Record < string, Rule > = { search: { perMinute: 120, burst: 60 }, // Googlebot, bingbot: never starve these aiSearch: { perMinute: 60, burst: 30 }, // OAI-SearchBot, Claude-SearchBot, PerplexityBot aiTraining: { perMinute: 20, burst: 10 }, // GPTBot, ClaudeBot, CCBot, Bytespider unknownBot: { perMinute: 10, burst: 5 }, // anything else that says bot, curl, python... default: { perMinute: 120, burst: 60 }, // browsers }; export function kindOf ( ua: string ): keyof typeof RULES { const u = ua. toLowerCase (); if ( /googlebot|bingbot/. test ( u )) return " search "; if ( /oai-searchbot|claude-searchbot|perplexitybot/. test ( u )) return " aiSearch "; if ( /gptbot|claudebot|ccbot|bytespider/. test ( u )) return " aiTraining "; if ( /bot|crawler|spider|scrapy|python-requests|curl|go-http|headless/. test ( u )) return " unknownBot "; return " default "; } const buckets = new Map < string, { tokens: number; at: number } > (); /** Returns 0 if allowed, or the seconds to wait before retrying. */ export function check ( key: string, rule: Rule, now = Date. now ()): number { const b = buckets. get ( key )?? { tokens: rule. burst, at: now }; const refill = (( now - b. at ) / 60 _000 ) * rule. perMinute; b. tokens = Math. min ( rule. burst, b. tokens + refill ); b. at = now; if ( b. tokens >= 1 ) { b. tokens -= 1; buckets. set ( key, b ); return 0; } buckets. set ( key, b ); return Math. ceil ((( 1 - b. tokens ) / rule. perMinute ) * 60 ); } Enter fullscreen mode Exit fullscreen mode And the middleware that uses it, which also drops scanner probes before they reach your app:

// middleware.ts (Next.js): rate limit by visitor kind, drop scanner probes early. import { NextResponse, type NextRequest } from " next/server "; import { RULES, check, kindOf } from " @/lib/bot-limit "; const SCANNER = /wp- ( admin|login|content|includes ) |phpinfo| \. env| \. git \/ |xmlrpc| \/ cgi-bin \/ /i; export function middleware ( req: NextRequest ) { const { pathname } = req. nextUrl; if ( SCANNER. test ( pathname )) return new NextResponse ( null, { status: 404 }); const ua = req. headers. get ( " user-agent " )?? ""; const ip = req. headers. get ( " x-forwarded-for " )?. split ( ", " )[ 0 ]. trim ()?? " unknown "; const kind = kindOf ( ua ); const wait = check ( ` ${ kind }: ${ ip } `, RULES [ kind ]); if ( wait > 0 ) { return new NextResponse ( " Too many requests ", { status: 429, headers: { " Retry-After ": String ( wait ) }, }); } return NextResponse. next (); } // Skip static files so images and JS never count against the limit. export const config = { matcher: [ " /((?!_next/static|_next/image|favicon.ico|.* \\.(?:png|jpg|webp|svg|css|js)$).*) " ] }; Enter fullscreen mode Exit fullscreen mode I built this into a fresh Next.js 15 app, ran next start, and sent 14 requests in a row as each visitor:

# Scanner probe curl -o /dev/null -w "%{http_code}" localhost:3000/wp-content/phpinfo.php # 404 # 14 requests as GPTBot 200 200 200 200 200 200 200 200 200 200 429 429 429 429 # HTTP/1.1 429 Too Many Requests # retry-after: 3 # 14 requests as Googlebot 200 200 200 200 200 200 200 200 200 200 200 200 200 200 Enter fullscreen mode Exit fullscreen mode

GPTBot gets its burst of 10, then a 429 with a Retry-After header that well-behaved crawlers honor. Googlebot never notices the limiter.

Keep search and AI search bots allowed, because being cited in AI answers is now a real source of visitors. Slow the training crawlers. Find the anonymous heavy hitters in your logs, block them by IP or behavior, and put every unknown bot on a small budget. Run the log script weekly, because the worst client next month will not be the one you blocked this month.

Axionry's own site allows AI crawlers on purpose, so this is the toolkit I keep ready for the day one of them, or something pretending to be one, gets greedy. Logging and rate limits like these belong in the SaaS MVP tech stack from day one. If bandwidth from bots is showing up on your bill, it is also one of the first things I check in a cloud cost review.

Sources: I posted to HN, and then my website was flooded with AI bot traffic, Hacker News discussion, OpenAI crawlers and IP ranges, Anthropic on ClaudeBot, Claude-SearchBot and Claude-User, Googlebot IP ranges. Scripts tested on 14 October 2026 against a generated log and a Next.js 15.5 production build.