Cloudflare blocks AI crawlers by default on every one of the roughly 3.8 million domains it’s onboarded since July 1, 2025, unless the site owner specifically turns that setting off. Most site owners never touch it either way, which means a meaningful share of newer websites are invisible to the exact crawlers that feed AI answer engines like ChatGPT and Perplexity, without anyone on the team ever deciding that on purpose. Common Crawl, whose archive is a foundational training source for a large share of major language models, recently published an 18-page manual explaining how to check your own site’s status, and a follow-up from Search Engine Journal automated most of it into a free browser tool. Together they surface a problem that checking robots.txt alone won’t catch.<\/p>\n
The instinct most site owners have is to open robots.txt, look for a line blocking GPTBot or CCBot, and consider the question answered. That check misses two entire categories of blocking. A CDN or security service can reject an AI crawler at the network edge before the request ever reaches the point where robots.txt would even apply, which is exactly what Cloudflare’s default setting does. And a web application firewall can challenge or reject a crawler based on request patterns rather than a published rule anywhere, meaning there’s no file a site owner could read to discover it. Common Crawl’s own research found that of the roughly 492,000 sites that explicitly name its crawler (CCBot) in robots.txt, about 95% are blocking it on purpose, which means for the sites that do mention it, robots.txt says what it means. The problem is the far larger set of sites where blocking is happening somewhere robots.txt never mentions at all.<\/p>\n
Common Crawl isn’t a niche archive. Its July 2026 crawl covers roughly 2.14 billion pages, and by one estimate, roughly 64% of large language models released between 2019 and 2023 were trained at least partly on Common Crawl data, with GPT-3 reportedly around 60% Common Crawl by training weight. That scale is exactly why a default block that quietly excludes a site deserves more attention than it usually gets: it isn’t just one crawler among many, it’s one of the archives multiple major model builders draw from directly or use to bootstrap their own crawls. At the same time, roughly 75% of top US and UK publishers now block AI training crawlers outright, a deliberate industry-wide pushback that’s a very different situation from a site being blocked by a default setting nobody chose.<\/p>\n
Common Crawl opened up a browser-based API in 2025 that lets anyone query its crawl history for a specific domain, which is what SEJ’s Suganthan Mohanadasan built the free Common Crawl Visibility Checker around, automating the manual’s 18 pages of instructions into a tool that needs no signup. It surfaces four things a site owner couldn’t easily see otherwise: how many pages of a domain got captured per monthly crawl over the last year, a month-by-month history of stored robots.txt versions (useful for spotting exactly when a block started, not just that one exists now), whether a detected block matches a known template like Cloudflare’s managed default or a Squarespace default rather than a deliberate choice, and a live probe that requests the homepage as CCBot specifically versus a normal browser, catching edge-level challenges that a static robots.txt read would never show.<\/p>\n
That crawl-frequency data isn’t just diagnostic, either. CCBot’s captures follow harmonic centrality, meaning better-linked domains get crawled more often and more thoroughly. A site with a thin backlink profile and low crawl frequency is going to be underrepresented in AI training data even with zero blocking configured anywhere, a different and harder problem than a misconfigured firewall rule, and one a visibility check alone won’t fix.<\/p>\n
The abstract description of four diagnostic panels is easier to trust with a concrete example attached. Testing BBC.com through the tool shows the site blocking 13 of 14 tracked AI crawlers consistently across every month reviewed, a clear, deliberate, industry-scale block rather than an accident, and exactly the kind of result the checker is built to distinguish from an unintentional one. The tool’s own creator ran it against his personal site and found a more ordinary, more common result: 42 of 97 sitemap pages had been captured in the most recent monthly crawl, and one specific article had been captured only once across the entire 12-month history, still stored with an obsolete tracking parameter from an email service he’d since switched away from. Neither of those findings shows up by reading robots.txt, since nothing in that file was blocking anything. The gap was in what actually got crawled and how often, not in a rule anywhere. For scale, Wikipedia’s crawl volume is large enough that the tool shows an estimate rather than an exact count, roughly 3.7 million pages, useful context for what “well-crawled” looks like at the extreme end versus a small business site’s realistic numbers.<\/p>\n
Not all AI crawlers get blocked at the same rate, and the gap between ordinary sites and major publishers is large enough to matter when deciding how seriously to treat this. Across the whole web, GPTBot’s block rate sits around 7.9%, but among publishers specifically that jumps to roughly 50.5%, a five-times difference that reflects how much more defensively news and content publishers have positioned themselves against AI training crawlers compared to typical small business or e-commerce sites. GPTBot is the single most-blocked crawler in robots.txt DISALLOW rules generally (about 5.52% of such rules as of Q1 2026), narrowly ahead of CCBot (5.08%) and ClaudeBot (4.88%). Among the top 1,000 websites specifically, GPTBot blocking has climbed from about 5% in early 2023 to roughly 25% today, a five-fold increase in three years that tracks the broader shift toward publishers treating AI training access as something to actively manage rather than ignore.<\/p>\n
None of those numbers say whether blocking is the right call for any specific site; a publisher protecting subscription content has different incentives than a hosting company’s blog trying to get cited in an AI answer engine’s response to a customer’s question. But they do put a single site’s own settings in useful context: a small business site that’s accidentally blocking every AI crawler via an unreviewed Cloudflare default isn’t following a deliberate publisher strategy, it’s an unintentional side effect of a setting nobody looked at, which is exactly the gap this whole check is meant to close.<\/p>\n
Once a block is confirmed, the fix depends on where it’s happening, and the order matters. Check CDN or WAF settings before touching robots.txt at all; editing a robots.txt rule does nothing if a Cloudflare or similar edge setting is already rejecting the crawler upstream of that file. After confirming the edge isn’t the problem, explicitly allow CCBot (and any other AI crawlers worth allowing, like GPTBot, ClaudeBot, or PerplexityBot) in robots.txt rather than assuming the absence of a disallow rule is enough, and make sure a sitemap reference is included so the crawler can find pages efficiently rather than relying on link discovery alone. One easy-to-miss technical point: CCBot doesn’t execute JavaScript, so a site that renders its main content client-side (a common pattern on newer, JS-framework-built sites and some page builder themes) can look empty to the crawler even with every permission set correctly, a completely different problem from a blocking rule, and one that server-side rendering or pre-rendering is the only real fix for. None of this guarantees inclusion in any specific AI model’s training data either; being crawled by Common Crawl doesn’t put a site in a model, since builders apply their own filtering on top of the raw archive. It just removes the crawler-access problem, which is the one part of this within a site owner’s direct control.<\/p>\n
For any site that switched CDN or security providers, or that was set up new, sometime after mid-2025, it’s worth spending fifteen minutes confirming AI crawler access rather than assuming it’s fine. Start with the basics: read robots.txt directly and look for disallow rules naming GPTBot, CCBot, ClaudeBot, or PerplexityBot. Then check whatever CDN or WAF service sits in front of the site (Cloudflare’s dashboard has a specific AI Bots setting under Security worth checking even if nothing in robots.txt looks like a block) for a default-on AI-blocking toggle. If there’s any uncertainty after that, a tool that live-probes the homepage as an AI crawler, rather than just reading a static file, is the only way to catch a WAF-level block that wouldn’t show up any other way. None of this requires guessing. It requires checking a setting that, for a meaningful share of sites built in the last year, was quietly turned on by default and never revisited.<\/p>\n","protected":false},"excerpt":{"rendered":"
Cloudflare blocks AI crawlers by default on every one of the roughly 3.8 million domains it’s onboarded since July 1, 2025, unless\u2026<\/p>\n","protected":false},"author":2,"featured_media":898,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0,"rank_math_title":"AI Crawler Visibility Check: What Robots.txt Won't Show You","rank_math_description":"Cloudflare blocks AI crawlers by default on 3.8 million domains created since mid-2025. Here's how to actually check if your site is one of them.","rank_math_focus_keyword":"AI crawler visibility check, Common Crawl, is my site blocked from AI training","rank_math_canonical_url":"","rank_math_robots":[],"footnotes":""},"categories":[8],"tags":[],"class_list":["post-899","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo-marketing"],"_links":{"self":[{"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/posts\/899","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/comments?post=899"}],"version-history":[{"count":1,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/posts\/899\/revisions"}],"predecessor-version":[{"id":907,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/posts\/899\/revisions\/907"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/media\/898"}],"wp:attachment":[{"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/media?parent=899"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/categories?post=899"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/allcloudhost.net\/blogs\/wp-json\/wp\/v2\/tags?post=899"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}