Cloudflare announced Bot Preference Sync on August 21, 2026, a feature that automatically rewrites a site’s robots.txt file to match whatever AI crawler policy is already configured in the Cloudflare dashboard, and its September 15 update to that post confirms it is available to every customer from the free tier through Enterprise. The problem it solves is a real, common one: site owners were setting AI crawler preferences in Cloudflare’s dashboard, and separately maintaining a robots.txt file that frequently said something different, or nothing at all, about the exact same crawlers.
Table of Contents
The Actual Problem Bot Preference Sync Fixes
robots.txt and a CDN-level bot policy are two genuinely separate systems that happen to control overlapping ground. robots.txt is a voluntary, publicly-readable file that well-behaved crawlers check and (mostly) respect; a CDN-level bot policy is an active technical control that Cloudflare’s infrastructure actually enforces at the network level. Before Bot Preference Sync, keeping both in agreement required manually editing robots.txt to match dashboard settings, and it’s exactly the kind of maintenance task that quietly drifts out of sync over time as one gets updated and the other doesn’t. “Say it once” is literally Cloudflare’s own framing for the feature: set the actual policy once in the dashboard, and let it propagate to the publicly-visible file automatically.
The Three-Category System Underneath It
Bot Preference Sync builds on a bot taxonomy Cloudflare introduced on July 1, 2026, replacing an older, simpler allow-or-block framing with three behavioral categories: Search (crawling to index content and answer questions about it later, the traditional search-engine use case), Agent (automation acting in real time on a specific person’s behalf, an AI assistant fetching a page because a user asked it to), and Training (crawlers gathering material specifically to train or fine-tune a model).
That distinction matters because a site owner’s actual preferences often genuinely differ across these categories, wanting to remain indexable by Search crawlers while declining to have content used for Training, for instance, isn’t a contradiction, it’s a normal, coherent policy that the old binary allow/block framing couldn’t represent cleanly.
The Default That’s Worth Knowing About
Cloudflare also changed its defaults. Its August announcement made disallowing AI training the default for publishing sites that rely on ads, and from September 15 new domains get one of two presets depending on whether the site earns money from advertising: non-ad sites allow Search, Training and Agent crawlers, while ad-supported sites allow Search, disallow AI training, and block agents on pages with ads. If your revenue depends on ad impressions and you haven’t reviewed your Cloudflare bot settings recently, it’s worth checking what’s actually configured rather than assuming the older defaults still apply.
What Changed on September 15
On September 15 Cloudflare added a “Disallow AI Training” setting. Bot Preference Sync publishes the no-training preference in robots.txt, mixed-use crawlers that Cloudflare labels accountable (Googlebot, Bingbot and Applebot among them, whose operators commit that disallowing training won’t affect traditional search results) stay allowed for search, and every other training crawler is blocked. Cloudflare is also phasing out its older “Block AI Bots” toggle in favor of the granular Search, Training and Agent controls, and replacing Managed Robots.txt with Bot Preference Sync, so if you rely on either, check your settings.
What to Actually Do With This
If you’re already on Cloudflare, turning on Bot Preference Sync removes one specific, easy-to-neglect maintenance task, keeping robots.txt honest about what your actual bot policy is, without requiring you to remember to update a static text file every time the underlying policy changes. Beyond just enabling the feature, this is a reasonable prompt to actually review your current settings across all three categories (Search, Agent, Training) deliberately, rather than assuming whatever was configured previously, possibly by default rather than by an intentional choice, still reflects what you actually want given how much this space has shifted in a short period.
A Quick Check of Your Own Setup
Whether or not you use Cloudflare, it is worth looking at what your robots.txt actually says. Open yourdomain.com/robots.txt in a browser and read it as a crawler would. Look for rules that name AI crawlers, check that nothing blocks Googlebot or Bingbot by accident, and confirm the sitemap line points to a sitemap that still exists. If you use Cloudflare, compare that file with what the dashboard says your policy is. They should tell the same story, and if they don’t, that is the drift Bot Preference Sync is designed to remove.
Deciding What to Allow
- A publisher funded by ads often wants Search allowed, Training disallowed and agents limited, which is close to the preset Cloudflare now applies to ad-supported sites.
- A store or service business usually benefits from being found in search and being described accurately by assistants, so allowing Search and Agent access is often reasonable, with Training a separate choice.
- A site with proprietary or paid content may want all three restricted, and to rely on enforcement rather than on robots.txt alone.
There is no universally right answer. The useful step is to make the choice deliberately for each category instead of inheriting a default.
What robots.txt Can’t Do
robots.txt is a request, not a lock. Well-behaved crawlers honor it and badly behaved ones ignore it. Cloudflare’s pitch is that the published preference is backed by enforcement at the network edge: it says it will identify who is crawling, classify why they are crawling, and block the crawlers that ignore the preference. If your content is valuable enough that ignoring a request would cost you, that enforcement is the part that matters, and it is worth confirming it is switched on rather than assuming the text file is doing the work.
Why This Connects to a Bigger Shift
The existence of three distinct categories, rather than a single “AI bots: yes or no” toggle, reflects something real about how differently these traffic types actually behave and what site owners actually want from each. A search-indexing crawler that might eventually drive a visitor to your site is a fundamentally different proposition than a training crawler harvesting content with no direct traffic benefit ever coming back, and treating them identically was always a blunt instrument. Cloudflare formalizing that distinction, and now making it easy to keep consistent across both the enforced policy and the publicly-declared one, is a genuine improvement in how site owners can actually express nuanced preferences about AI crawler access.
Related reading: OpenAI Says ChatGPT’s Own Fetch Bot Doesn’t Have to Follow Your Robots.txt and Your Robots.txt Looks Fine. Your Site Might Still Be Invisible to AI Anyway.