Two files, both plain text, both sitting at your site’s root, both apparently about controlling what crawls your site. In practice, they do almost nothing alike, and neither one is actually enforceable.<\/p>\n
robots.txt: a request, not a lock<\/strong><\/p>\n robots.txt has been around since the mid-1990s and is now a formal standard (RFC 9309). It tells crawlers, by user-agent, which parts of a site they’re being asked to skip. The catch, and it’s a big one: it’s voluntary. A well-behaved crawler (Googlebot, Anthropic’s crawler, OpenAI’s) reads it and complies. A crawler that doesn’t care simply ignores it, and robots.txt has no mechanism to stop that. It’s a sign on the door, not a lock on it.<\/p>\n llms.txt: even less than that<\/strong><\/p>\n llms.txt is newer and does less than robots.txt, not more. It’s a plain Markdown file meant to work as a table of contents for AI tools, a pointer to a site’s most important pages, so an AI system doesn’t have to crawl and parse everything to understand what a site is about.<\/p>\n Two things matter here that are easy to miss:<\/p>\n – Adoption is still low. Roughly 9-10% of sites have one at all.<\/p>\n – No major AI lab has committed to treating llms.txt as an access signal. It’s a discovery aid, at best. Not a rate limiter, and not a gate. A crawler that respects it does so voluntarily; nothing about the file format forces compliance.<\/p>\n Put plainly: llms.txt controls nothing. robots.txt controls crawler behavior, but only for crawlers that choose to follow the rules. Neither one can block traffic or throttle a request. That job belongs to bot-protection tools running at the infrastructure level, not to a text file.<\/p>\n Why this actually matters for hosting bills, not just principle<\/strong><\/p>\n This isn’t a purely academic distinction. AI crawler traffic has grown fast enough that it now shows up as a real infrastructure cost, not a rounding error. A separate but related Kinsta report found AI-driven bot traffic up 300% in a single year, with AI bots now responsible for roughly 1 in 31 web visits on one major crawler-tracking network, up from 1 in 200 at the start of last year. A site owner who assumes robots.txt or llms.txt is “handling” that traffic is assuming a text file is doing a job it was never built to do.<\/p>\n What actually works<\/strong><\/p>\n