AI bot traffic mistakes: most sites’ AI bot traffic is unremarkable, but a few aren’t. Here’s what real infrastructure data shows about diagnosing it correctly.
Table of Contents
One crawler generated 3.75 million requests to add-to-cart URLs on a single site in 24 hours, roughly one request every 23 milliseconds, according to infrastructure data Kinsta published this month. That’s the kind of number that makes a site owner want to block every bot they see in the logs.
However, according to the same research, that volume does not reflect what most sites experience: across more than 5,000 WordPress sites Kinsta measured, AI bots accounted for a median of just 1.57% of bandwidth, compared with 17.8% at the 90th percentile and 90.3% at the 99th. The gap between the median site and the extreme cases is the whole story here. Reacting to a scary-looking number without checking which situation applies to your server leads site owners to make avoidable mistakes.
Mistake 1: treating the percentage alone as the diagnosis
A crawler generating 20% of your requests against cached blog posts puts far less load on your server than a much smaller share of requests repeatedly hitting search results, filtered product pages, or a cart endpoint. Kinsta’s data shows why averages mislead here: mean AI-bot requests per site ranged from 667 to 928 a day across their measurements, while the median sat at just 33 to 67, and 20% to 27% of sites received no AI-bot requests at all on a given day.
A small number of heavily crawled sites drags the average up far past what a typical site experiences. If you’ve seen a headline claiming bots make up half of all web traffic, that statistic describes the internet in aggregate, not your specific site, and it shouldn’t be the basis for a decision about your own server.
Before changing any setting, three questions are worth answering with your own log data rather than an industry average: how much AI traffic is reaching your site, which specific pages or endpoints it’s requesting, and whether any of it affects page load times or server resources for human visitors. A site with light, well-behaved crawler traffic against static content may not need to do anything.
Mistake 2: blocking every bot because the traffic is automated
“AI bot” describes multiple distinct categories of traffic. OpenAI runs GPTBot for content that may train its models and a separate OAI-SearchBot specifically to make sites discoverable in ChatGPT search; blocking the second one can remove you from ChatGPT search results entirely, a different outcome from blocking the first. Anthropic splits the same way, with ClaudeBot for training and Claude-SearchBot and Claude-User handling search and live retrieval. Perplexity runs PerplexityBot for its index and a separate Perplexity-User bot for requests made live in response to someone’s question. Google gives publishers a distinct Google-Extended control specifically for Gemini training data, and states plainly that changing it has no effect on Search rankings or inclusion.
Lumping all of that into a single block-or-allow decision discards information you could otherwise use. A business that would rather not spend server resources on model training but still wants to show up when someone asks ChatGPT or Perplexity a question about their industry needs to treat those as separate settings. There is a business reason to care about the distinction too: Kinsta’s own consumer research found 44.7% of people say they usually or always visit a company’s website after getting an AI recommendation naming it. Blocking every AI system indiscriminately forecloses that channel along with whatever training traffic you were trying to avoid.
Mistake 3: treating a legitimate crawler spike like a security incident
Bots can be part of a genuine security problem, such as brute-force login attempts, credential stuffing, or DDoS traffic. A verified AI crawler that is simply making too many requests is a different kind of problem requiring a different fix. Kinsta’s CTO Daniel Pataki put it plainly in discussing the company’s bot-traffic research: overreacting to a crawler spike concerns him more than underreacting, because in most cases it isn’t a security issue, just inefficient crawling.
That said, if bot traffic is degrading the site right now by tying up PHP workers and slowing pages for visitors, stabilizing the site takes priority over getting the classification perfectly right first. Kinsta’s guidance is to temporarily block the offending traffic when there’s an active problem, then investigate properly once things are stable, rather than turning an emergency measure into a permanent policy without ever finding out what caused the spike.
Mistake 4: watching request counts and ignoring where they land
A request for a cached blog post and a request for an uncached product search or cart page both register as one request in a raw traffic count, but they cost your server wildly different amounts of work. A cached page can often be served without WordPress doing much at all; a dynamic request typically needs a PHP worker, database queries, and page generation before anything gets returned, and checkout-adjacent pages add more on top of that. This skew toward the expensive part of your site is a much better predictor of infrastructure pressure than the raw request count on its own, and it explains why that 3.75-million-request add-to-cart incident turned from an annoyance into something requiring an emergency rule.
What this looks like without an enterprise infrastructure dashboard
Kinsta’s customers get a Bot Protection panel in MyKinsta that classifies traffic by type and shows exactly which paths and user agents are behind it, which makes diagnosing this fairly quick. Most small businesses on shared or unmanaged VPS hosting don’t have that view built in, which doesn’t mean the diagnosis above stops applying, just that you have to build it from raw materials.
Your access logs (available through your hosting control panel or via SSH on a VPS) show user agents and request paths directly; grep for known bot identifiers like GPTBot, ClaudeBot, or PerplexityBot and check what URLs they’re hitting before deciding anything. A free Cloudflare account in front of your site adds bot-classification and rate-limiting rules without needing enterprise tooling. And your robots.txt file remains the first, cheapest lever for separating a training crawler like GPTBot from a search-and-retrieval bot like OAI-SearchBot, rather than blocking both with one blanket rule.
If you’re on a shared plan and can’t tell whether bot traffic is the cause of a slowdown versus something else on a shared server, that’s a reasonable question to bring to your host directly rather than guess at server-side settings you don’t have full access to change. AllCloudHost’s hosting plans include access to raw request logs precisely so a customer can do the same page-versus-endpoint check Kinsta’s research describes, rather than being stuck choosing between doing nothing and blocking everything.
Avoiding AI bot traffic mistakes: start with your own numbers, not someone else’s
The through-line across all four mistakes is the same: a percentage, a request count, or another site’s horror story isn’t a decision on its own. Kinsta’s data makes a strong case that most sites’ AI bot traffic is unremarkable, and that the sites facing genuine problems are a distinct minority defined by pattern, not headline percentage. Check your own logs, identify which specific bots and paths are involved, and match the response to what you find rather than to the scariest number you’ve read this week.
Further reading: Google Search Central: overview of Google’s crawlers.