llms.txt vs robots.txt: the two files sound similar but do different jobs, and neither can block an AI crawler that ignores it. Here’s what each controls.
Table of Contents
Two files, both plain text, both sitting at your site’s root, both apparently about controlling what crawls your site. In practice, they do almost nothing alike, and neither one is actually enforceable.
robots.txt: a request, not a lock
robots.txt has been around since the mid-1990s and is now a formal standard (RFC 9309). It tells crawlers, by user-agent, which parts of a site they’re being asked to skip. The catch, and it’s a big one: it’s voluntary. A well-behaved crawler (Googlebot, Anthropic’s crawler, OpenAI’s) reads it and complies. A crawler that doesn’t care simply ignores it, and robots.txt has no mechanism to stop that. It’s a sign on the door, not a lock on it.
llms.txt: even less than that
llms.txt is newer and does less than robots.txt, not more. It’s a plain Markdown file meant to work as a table of contents for AI tools, a pointer to a site’s most important pages, so an AI system doesn’t have to crawl and parse everything to understand what a site is about.
Two things matter here that are easy to miss:
- Adoption is still low. Roughly 9-10% of sites have one at all.
- No major AI lab has committed to treating llms.txt as an access signal. It’s a discovery aid, at best. Not a rate limiter, and not a gate. A crawler that respects it does so voluntarily; nothing about the file format forces compliance.
Put plainly: llms.txt controls nothing. robots.txt controls crawler behavior, but only for crawlers that choose to follow the rules. Neither one can block traffic or throttle a request. That job belongs to bot-protection tools running at the infrastructure level, not to a text file.
llms.txt vs robots.txt: why this actually matters for hosting bills, not just principle
This isn’t a purely academic distinction. AI crawler traffic has grown fast enough that it now shows up as a real infrastructure cost, not a rounding error. A separate but related Kinsta report found AI-driven bot traffic up 300% in a single year, with AI bots now responsible for roughly 1 in 31 web visits on one major crawler-tracking network, up from 1 in 200 at the start of last year. A site owner who assumes robots.txt or llms.txt is “handling” that traffic is assuming a text file is doing a job it was never built to do.
A Quick Audit of Your llms.txt vs robots.txt Setup
You can find out where you stand in an afternoon:
- Open your robots.txt and check that it says what you intend for each crawler.
- Look at your access logs and note which crawlers visit most often.
- Decide which automated visitors you welcome, tolerate, or want to stop.
- Ask your host what bot protection and rate limiting are available.
- Watch your bandwidth and server load for a month and compare them to before.
Deciding Which Crawlers to Allow
There is no single right answer to the llms.txt vs robots.txt question of what to allow. Some site owners welcome AI crawlers because being cited helps them; others want to limit them because of the load or because they dislike their content being used. Whatever you decide, put it in robots.txt for the crawlers that respect it and use server-level controls for the ones that do not. Review the decision now and then, since new crawlers appear and old ones change their behavior.
What actually works
If uncontrolled crawling is inflating your bandwidth or server load, the fix sits at the hosting/infrastructure layer — rate limiting, bot-detection rules, and actually blocking AI crawlers server-side — not at the file-convention layer. AllCloudHost’s Hepsia control panel gives you visibility into what’s actually hitting your account, which is the first real step: you can’t decide whether to block something you can’t see.
The honest summary: robots.txt is a request most legitimate crawlers respect. llms.txt is a hint that almost nothing currently acts on. Both are worth having. Neither is a substitute for real bot management.
Further reading: RFC 9309: Robots Exclusion Protocol.