Buried in OpenAI’s crawler documentation is a distinction that changes what a robots.txt file is genuinely good for. ChatGPT-User is the bot that fetches a specific page the moment a live user asks ChatGPT about it or pastes a link into a conversation. According to OpenAI, because that visit is triggered by a real person asking for that exact page in real time rather than a bot crawling the web on its own schedule, robots.txt rules “may not apply” to it. A file most publishers have treated for three decades as a binding instruction to any automated visitor turns out, in OpenAI’s own framing, to be closer to a suggestion this particular bot isn’t obligated to check.<\/p>\n
That’s not a theoretical reading of a footnote. TollBit’s 2026 State of the Bots report measured it directly: across European sites that explicitly disallowed ChatGPT-User in their robots.txt file, ChatGPT-User still reached disallowed URLs on close to half of them anyway, more than any other AI crawler TollBit tracked. Roughly 15% of European sites overall experienced some access to pages they’d blocked, and ChatGPT-User accounted for most of those breaches. If your site added a Disallow rule for OpenAI’s bots and assumed that settled the question, the data says it might not have.<\/p>\n
Part of what makes this confusing is that OpenAI runs more than one bot, and they don’t do the same thing. GPTBot is the training crawler, the one that scrapes broadly to build future model data and is the bot most “block AI training” guides are written about. OAI-SearchBot is the one that determines whether your pages show up as sources in ChatGPT Search results; block it and you lose that visibility entirely, the same way blocking Googlebot costs you organic rankings. ChatGPT-User is neither of those. It’s a fetch-on-demand agent that retrieves one specific page because a user asked about it in the moment, closer to a browser loading a page a person typed in than a crawler indexing your whole site.<\/p>\n
That distinction matters because a site can block ChatGPT-User in robots.txt, technically ask OpenAI not to fetch on a user’s behalf, and still show up fine in ChatGPT Search results, because OAI-SearchBot is a separate agent with separate rules. Conversely, blocking OAI-SearchBot costs you search visibility whether or not ChatGPT-User respects your rules at all. Site owners who set one blanket “block OpenAI” rule assuming it covers both cases are usually only affecting one of them, and per OpenAI’s own stated position, not necessarily the one they thought.<\/p>\n
OpenAI’s reasoning rests on a real distinction in how bots have historically worked: a robots.txt file was built to manage automated, unattended crawling, not to stop a browser from loading a page a human being explicitly requested. If ChatGPT-User is functioning more like a proxy for a person’s own browser than like a crawler indexing your site at scale, there’s a coherent argument that the traditional robots.txt contract doesn’t cleanly apply.<\/p>\n
The problem is that “may not apply” isn’t the same as “we ignore it deliberately,” and OpenAI has also said it respects the signal where it can, which leaves publishers in an unclear middle ground. A site owner reading a Disallow rule as an enforceable boundary and a bot vendor reading the same rule as one input among several aren’t operating from the same assumption, and TollBit’s numbers suggest that gap shows up in actual traffic on close to half the sites that tested it.<\/p>\n
The interesting part isn’t that one company has an aggressive interpretation of robots.txt. It’s that the major AI labs don’t agree with each other. Perplexity takes essentially the same position as OpenAI for its user-triggered bot, Perplexity-User: because a person asked for the page, the company has said its fetch agent generally doesn’t check the file. Anthropic, by contrast, has stated that all three of its bots, including its own user-triggered fetch agent, respect robots.txt without the same carve-out.<\/p>\n
That means the exact same Disallow rule on your site produces three different outcomes depending on which company’s bot reads it: reliably honored by Anthropic’s crawlers, generally not checked by Perplexity’s user-triggered bot, and inconsistently honored by OpenAI’s, closer to half the time based on the European site data. A robots.txt file that felt like a single, universal instruction three years ago now behaves more like a set of separate requests, each interpreted under a different company’s internal policy, with no visible confirmation of which policy applied to a given visit.<\/p>\n
Cloudflare has announced that starting September 15, 2026, it will begin enforcing AI crawler controls at the network layer by default for newly onboarded domains, blocking Training and Agent crawler categories automatically on ad-supported pages unless a site owner opts out. That’s a meaningfully different enforcement model: instead of relying on a bot to voluntarily check a text file sitting on your server, the network itself refuses the connection before the bot ever gets a response to interpret. For any AllCloudHost customer running a site behind Cloudflare, or considering it, this shifts the practical question from “did I write my robots.txt correctly” to “which network-layer controls does my host or CDN enforce, and did I check what the new default does to legitimate AI traffic I might want to keep.”<\/p>\n
That second half matters because not every business wants to block ChatGPT-User. If your customers are asking ChatGPT to look up your pricing page, your hours, or your product specs, a blanket AI-crawler block done at the network edge cuts that off just as effectively as a training bot you never wanted indexing your content in the first place. The fetch-on-demand bots are frequently the ones doing something a business would want to allow, while the bulk training crawlers are the ones most site owners are trying to stop in the first place.<\/p>\n
A robots.txt file that reflects this distinction doesn’t need to be complicated, but it does need to name the bots separately instead of grouping every “OpenAI” or “AI” crawler under one blanket rule. A site that wants to keep fetch-on-demand access and search visibility while blocking bulk training might use a table like this to plan the rules before writing them:<\/p>\n
| Bot<\/th>\n | Purpose<\/th>\n | Typical rule for a business that wants AI visibility<\/th>\n<\/tr>\n<\/thead>\n |
|---|---|---|
| GPTBot<\/td>\n | Bulk training crawl<\/td>\n | Disallow<\/td>\n<\/tr>\n |
| OAI-SearchBot<\/td>\n | ChatGPT Search visibility<\/td>\n | Allow<\/td>\n<\/tr>\n |
| ChatGPT-User<\/td>\n | Fetch-on-demand (per TollBit, inconsistently honored)<\/td>\n | Allow, but verify in logs<\/td>\n<\/tr>\n |
| PerplexityBot<\/td>\n | Perplexity’s own indexing crawler<\/td>\n | Allow, if search visibility matters there<\/td>\n<\/tr>\n |
| Perplexity-User<\/td>\n | Fetch-on-demand (also user-triggered, similar carve-out to OpenAI’s)<\/td>\n | Allow, but verify in logs<\/td>\n<\/tr>\n |
| ClaudeBot \/ anthropic-ai<\/td>\n | Training and search<\/td>\n | Per Anthropic’s stated policy, reliably honors robots.txt either way<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n The “verify in logs” column matters more than it looks. A Disallow rule for a bot that doesn’t reliably check it isn’t harmless, it’s a rule that gives a false sense of control. For the two rows marked that way, the real access-control decision is happening in your server logs and, going forward, at the network layer, not in the text file.<\/p>\n Three practical steps make more sense than a blanket rule either way. First, check server logs for `ChatGPT-User`, `OAI-SearchBot`, and `PerplexityBot` user-agent strings before assuming your current robots.txt is doing what you think; log evidence beats assumptions about compliance. Second, treat GPTBot (training), OAI-SearchBot (search visibility), and ChatGPT-User (fetch-on-demand) as three separate access-control decisions instead of one, since blocking all three the same way probably isn’t what most businesses actually want in every case. Third, if you’re on a CDN or host with network-layer bot controls, confirm what the default posture is before September 15 rather than discovering it after a new domain’s traffic changes; a control enforced at the network is far more reliable than one that depends on a third party voluntarily reading a text file, but it’s also easier to get wrong in the other direction by blocking traffic you wanted.<\/p>\n Robots.txt isn’t obsolete. It’s still respected by traditional search crawlers and by some AI vendors’ bots, and it remains worth maintaining correctly. What changed is the assumption that a Disallow line is self-enforcing across every automated visitor that might read it. For AI fetch bots specifically, whether that line does anything now depends on which company wrote the bot, and per OpenAI’s own documentation and TollBit’s measured data, the answer for ChatGPT-User is: not necessarily.<\/p>\n |