In short: Cloudflare’s Disallow AI Training setting asks AI training crawlers to stay away while Googlebot and other search crawlers keep indexing your site. This guide explains what Disallow AI Training does and what its limits are.
Table of Contents
On September 15, 2026, Cloudflare rolled out a setting called Disallow AI Training. This feature solves a dilemma site owners faced: the choice used to be all or nothing, letting every crawler in or blocking everyone out, including the search engines a site needs. The new setting, covered by Web Designer Depot, splits that choice apart. A site can keep welcoming Googlebot, Bingbot, and Applebot for search indexing while telling AI-training-only crawlers, the bots run by OpenAI, Anthropic, Meta, and Amazon specifically to gather training data, to stay out entirely.
The mechanism is the same robots.txt file that’s governed crawler behavior since the 1990s, just managed through Cloudflare’s dashboard instead of hand-edited line by line. Cloudflare’s own announcement, posted to its blog, describes the underlying system as Bot Preference Sync: flip the setting, and Cloudflare writes the appropriate disallow directives into the site’s robots.txt automatically, distinguishing between crawlers it considers purely training-focused and crawlers it labels “mixed-use,” meaning they power both search results and, potentially, AI training data collection at once.
How mixed-use crawlers are classified
Google, Apple, and Microsoft’s crawlers don’t fit neatly into “search” or “AI training” as separate buckets, because the same crawl can feed both a traditional search index and an AI system. Blocking that crawler outright to stop the AI-training use also kills search visibility, which is a much bigger cost for most site owners than the AI training concern on its own. Cloudflare’s answer is a signal, not a block: mixed-use crawlers keep crawling, but the site’s preference not to have that content used for training gets communicated through the same robots.txt mechanism. Google, Apple, and Microsoft have each committed to honoring that signal for their respective crawlers.
That arrangement differs from existing guides on blocking AI crawlers, which usually rely on a flat list of user-agent strings (GPTBot, ClaudeBot, Amazonbot, and so on) to individually disallow by hand. Cloudflare’s version automates that list and adds the search-preserving nuance manual robots.txt guides omit, since they predate this specific mixed-use distinction.
Built-in limitations of the honor system
Robots.txt has always worked on an honor system. It is a published preference rather than a technical barrier; nothing stops a crawler from ignoring it entirely, and enforcement depends entirely on the crawler operator choosing to comply. Cloudflare’s own messaging is direct about this limitation: a company that respects the training disallow signal today isn’t contractually bound to keep doing so, and a crawler with no reputation to protect has no incentive to honor it in the first place. For a site concerned about aggressive, non-compliant scraping beyond the major AI labs who have publicly committed to respecting these preferences, Cloudflare’s separate bot-blocking and rate-limiting tools handle the technical enforcement.
Disallow AI Training serves as a preference declaration layered on top rather than a replacement.
Blocking Google’s AI-training crawler signal does not remove existing content from Google’s AI Overviews or AI Mode results. Those features draw on Google’s search index directly, which the mixed-use crawler still builds by design. Disallowing training data collection and opting out of AI-generated search summaries are two different things, and this setting only controls the first one.
Impact on search rankings
Disallowing AI training through this mechanism doesn’t touch Googlebot’s, Bingbot’s, or Applebot’s ability to crawl and index pages for search results. The setting is specifically designed to separate whether a crawler can put a page in search results from whether that crawl can be used to train a model, and the search half of that stays untouched. Site owners who were avoiding any AI-related setting out of fear it would quietly tank their search visibility can treat that specific worry as addressed here.
How to Migrate Existing Settings to Disallow AI Training
Cloudflare previously offered a blunter “Block AI Bots” toggle. According to Cloudflare’s announcement, that toggle is being deprecated and customers are migrated based on their legacy settings, so anyone who had it switched on before September 15 shouldn’t need to reconfigure anything. The announcement doesn’t say how long the migration takes, so anyone who wants the new mixed-use nuance immediately, rather than waiting, can check the relevant Cloudflare dashboard setting directly instead of assuming the old toggle already covers it.
Why business websites should use this tool
Site owners running a business site, a blog, or a small e-commerce store face a straightforward practical calculus: there is close to no downside to enabling Disallow AI Training, since search visibility is preserved, and the upside is a preference against having original content, product descriptions, blog posts, and pricing pages used to train a competitor’s AI model or a general-purpose one without any benefit flowing back to the site that wrote it. The exception worth considering is a site that specifically wants AI-driven discovery, being cited by name in ChatGPT or Perplexity answers, treating any AI crawler access as free exposure rather than a cost.
That is a strategic choice worth making deliberately rather than defaulting into either position without thinking about which one serves a particular site’s goals.
For sites running ads, Cloudflare’s recommendation is a layered setup: let search engines in, block AI training everywhere, and additionally block AI agents specifically on pages where ads are detected, since an AI agent browsing on a user’s behalf and never rendering the ad defeats the economic model that page depends on. That is a narrower concern than most small business sites need to worry about, but it illustrates how granular bot management is becoming as AI agents increasingly browse the web on behalf of human users rather than just crawling it for indexing or training.
Verification steps for site administrators
Confirming dashboard configurations requires direct validation. The robots.txt file lives at a predictable, public address on every site: yourdomain.com/robots.txt, viewable in any browser or via a quick curl yourdomain.com/robots.txt from a terminal. After enabling Disallow AI Training, that file should show explicit disallow directives for the training-only crawlers (GPTBot, ClaudeBot, Amazonbot, and similar user-agent strings), while Googlebot, Bingbot, and Applebot should have no matching disallow rule blocking them from the site generally.
If the file still looks like it did before enabling the setting, the migration window may not have completed yet, or the setting may need to be applied at the account level rather than per-site depending on how a particular Cloudflare plan is structured. Checking the published file directly is more reliable than trusting a dashboard toggle’s label alone.
A second reason to check directly: AI crawler traffic has grown enough on some sites over 2026 that their server logs now show more bandwidth consumed by training and agent crawlers than by human visitors, a cost that shows up as server load and bandwidth regardless of whether that traffic converts into a customer. Confirming the block took effect in the published robots.txt results in a reduction in wasted server resources once the training-only crawlers stop showing up in the access logs.
Related reading: OpenAI Says ChatGPT’s Own Fetch Bot Doesn’t Have to Follow Your Robots.txt and Your Robots.txt Looks Fine. Your Site Might Still Be Invisible to AI Anyway.