Summary of Crawler Security Findings
- Single Largest “AI Crawler” Analysis: A website owner discovered that their top AI crawler by request volume was actually a credential scanner spoofing Common Crawl’s user agent.
- Request Volume and Targets: The scanner made roughly 1,500 requests over 24 hours (6.7 MB of bandwidth total), of which about 1,028 specifically targeted over 100 credential-hunting file paths (e.g., SSH private keys,
.envfiles, Firebase config, AWS credentials) rather than any real page on the site. - Agent-Specific Wordlists: Scanners now target new AI-coding-agent configuration paths like
/.mcp.jsonand/.continue/config.json. - Spoofing Statistics: Research shows 5.7% of requests presenting a known AI crawler or scraper user agent were spoofed.
The Traffic Shift in Modern Server Operations
Non-human traffic is projected to reach up to 1,000 times human traffic within five years, meaning humans will soon be a statistical rounding error on the internet.
This credential-scanning discovery landed alongside a massive claim from Cloudflare’s CFO regarding the future of the web. Cloudflare has already confirmed that fewer than half of all HTML page requests today originate from actual people. Most coverage of that prediction focuses on bandwidth costs and search indexing, which are real concerns for anyone paying for hosting by the resource. What gets less attention is the security half of that same shift: as automated traffic becomes the overwhelming majority of what hits your server, hostile traffic has more cover to blend in, because “a bot with an unfamiliar user agent” stopped being a useful signal on its own a long time ago.
AI-Agent Target File Lists in Credential Scanning
Credential scanners now target newly deployed AI-coding-agent configuration paths like /.mcp.json and /.continue/config.json alongside traditional sensitive files.
SSH keys and .env files being targets isn’t new; credential scanning has gone after those for years, because a leaked .env file routinely contains database passwords, API keys, and third-party service credentials in plain text. What’s newer, and worth paying closer attention to if you run any kind of automated or AI-agent tooling on your server, is the addition of agent-specific configuration files to these wordlists:
/.mcp.json/.continue/config.json
These aren’t legacy web-app config files. They’re configuration for AI coding agents and automation tools, and they routinely hold API keys and access tokens with real reach into whatever services that agent is authorized to touch.
A scanner updating its wordlist to include AI-agent config paths is built by someone tracking what developers are deploying in 2026, not running a static list from three years ago. If you or anyone on your team has set up an AI coding assistant, an automation agent, or an MCP-based tool on a server you manage, and its config file sits anywhere web-accessible, that’s now a specifically targeted path, not an incidental one.
Common Crawl User Agent Spoofing and Bot Traffic
Malicious credential scanners spoof trusted entities like Common Crawl because its reputable status bypasses simple user-agent filters and avoids suspicion during log reviews.
Common Crawl is a legitimate, widely respected web-archiving project, and plenty of site owners either allow its crawler deliberately or simply don’t bother blocking it, since it’s not typically associated with abuse. That reputation is exactly why a malicious scanner would spoof its user agent: a request that claims to be Common Crawl is far less likely to get blocked by a simple user-agent filter or raise suspicion in a quick log skim than a generic “unknown bot.” Separate research cited alongside this finding found that 5.7% of requests presenting a known AI crawler or scraper user agent were spoofed, meaning the traffic wasn’t genuinely from the company whose name it was borrowing. That’s a real fraction of total traffic concentrated in exactly the kind of requests a site owner is least likely to scrutinize.
Automated Credential Scanning on Small Business Sites
Small business websites and new domains are constantly targeted by automated credential scanners because indiscriminate wordlist probing incurs virtually zero cost for attackers.
A common reason small business owners don’t bother checking crawler logs is the assumption that automated scanning is aimed at large, high-value targets, not a modest local business site or a small online store. That assumption doesn’t hold up against how this kind of scanning operates in practice. Credential scanners of this type aren’t manually targeting specific companies; they’re running the same wordlist against enormous numbers of domains indiscriminately, because the cost of checking one more site for an exposed .env file is close to zero. Site size and traffic volume have nothing to do with whether a scanner tries these paths against you. What matters is only whether the file happens to be reachable. The same logic applies to timing: a new site launched last week is scanned on roughly the same schedule as one that has been live for a decade, since these wordlist scans run continuously against broad ranges of domains rather than waiting for sites to become established.
How to Audit Web Server Logs for Credential Scanners
System administrators can identify credential scanners by filtering access logs for high-volume user agents requesting non-existent sensitive file paths instead of actual pages.
The practical takeaway from the original finding isn’t “block Common Crawl,” which would be an overcorrection that costs you legitimate archival indexing for no real security benefit. It’s that volume alone tells you almost nothing about whether traffic claiming to be a known crawler is safe, and the only way to find out is to look at what the requests are asking for.
A useful version of this check doesn’t require specialized tooling. Pull your server’s access logs (most hosting control panels expose these, or they’re available directly via SSH on a VPS) and filter for requests from your highest-volume “crawler” user agents. Then look at the paths being requested. Legitimate crawler traffic requests your actual pages, your sitemap, your images. A credential scanner requests paths that don’t correspond to anything in your site’s structure, such as:
.env.git/configwp-config.php.bak.aws/credentials/.mcp.json
A crawler that never once requests a real page on your site while making a thousand-plus requests to file paths that don’t project files is not indexing your content. It’s fingerprinting your server for exposed secrets.
Server Security Architecture Best Practices for AI Files
Securing web applications requires moving sensitive assets entirely outside the document root and verifying deployment configurations on your hosting infrastructure.
The deeper fix isn’t log-watching after the fact, it’s making sure the files these scans are hunting for were never reachable over HTTP in the first place. .env files, SSH keys, and any AI-agent configuration should live outside your web server’s document root entirely, not just be excluded by a .htaccess rule that a misconfiguration could later undo. On a VPS where you control the full server stack, that’s a configuration choice worth verifying directly rather than assuming your framework or deployment process handled it by default, especially if you’ve added AI agent tooling to an older server. On shared hosting, the equivalent check is simpler: confirm that none of these file types sit inside a publicly served folder at all, since a shared environment gives you less direct control over server-level blocking rules.
The original finding is a reminder about a rounding error, not a rounding error itself. One credential scanner making 1,500 requests and coming up empty is a near-miss, not an incident. The value in the story is what it reveals about where scanning wordlists are heading (specifically, toward AI-agent config files that didn’t exist as a target category two years ago) and the fact that volume-based bot detection alone won’t catch a scanner willing to borrow a trusted name. Checking what your top “crawlers” are actually requesting is a cheap way to find out whether your server has this specific gap.