Two files at the root of your site claim to control how AI systems use your content. One is a thirty-year-old convention that every major AI company honours and that will quietly remove you from AI search results if you configure it wrong. The other is a 2024 proposal that, as far as anyone can demonstrate, almost nothing actually reads. Most advice about them treats the pair as equivalent, which is how sites end up blocking the traffic they were trying to attract.
The short version
robots.txt tells crawlers what they may fetch. It is honoured by every major AI company, it is the file that actually changes what happens, and its effects are easy to get backwards.
llms.txt is a proposed markdown index of your best content, intended to help models find what matters without wading through your navigation. It is cheap to publish and genuinely useful to people pasting your docs into an assistant — but no major provider has confirmed their crawlers read it, so do not expect visibility gains.
Two files, two different jobs
They are not alternatives to each other, and they do not operate at the same layer. One is about access; the other is about presentation.
| robots.txt | llms.txt | |
|---|---|---|
| Status | Standard (RFC 9309, 2022) | Proposal (Sept 2024) |
| Purpose | Restrict what gets fetched | Highlight what matters |
| Format | Directive lines | Markdown |
| Honoured by | All major AI crawlers | No confirmed consumer |
| Effect if wrong | Lost visibility | None measurable |
| Worth your time | Yes, get it right | Cheap, low expectations |
The asymmetry is the point. A mistake in robots.txt can remove you from ChatGPT's search results or Google's index. A mistake in llms.txt does approximately nothing. Spend your attention accordingly.
How robots.txt actually works
The Robots Exclusion Protocol was informal for twenty-eight years before being written up as RFC 9309 in 2022. A crawler requests /robots.txt before fetching anything else, and applies the rules in the group matching its user agent.
Three matching rules cause most of the confusion:
- A crawler obeys exactly one group. The most specific matching user agent wins. If
GPTBothas its own group, it ignores theUser-agent: *group entirely — including anyDisallowrules you assumed applied to everyone. - The longest matching rule wins, not the first. A specific
Allowoverrides a broaderDisallow, which is how you carve exceptions out of a blocked directory. - Matching is on the path, and it is case-sensitive.
/Admin/and/admin/are different rules.
⚠️ Disallow is not noindex
These solve different problems and people reach for the wrong one constantly. Disallow stops a page being fetched. It does not stop the URL appearing in results — search engines can list a URL they have never crawled, based on links pointing at it, which is where those bare "no information is available for this page" entries come from.
Worse, the two conflict. A noindex tag lives in the page's HTML, so a crawler has to fetch the page to see it. Block the page in robots.txt and the crawler never reads the tag, and the URL stays listed indefinitely. To remove a page: allow the crawl, serve noindex.
The AI crawlers, and what each one does
The names below cover the significant operators. Two of them are not crawlers at all, which matters more than it sounds.
| Token | Operator | What it does |
|---|---|---|
GPTBot | OpenAI | Crawls for training data |
OAI-SearchBot | OpenAI | Builds the ChatGPT search index |
ChatGPT-User | OpenAI | Fetches a page a user just asked about |
ClaudeBot | Anthropic | Crawls for training data |
Claude-SearchBot | Anthropic | Builds the search index |
Claude-User | Anthropic | Fetches on user request |
Google-Extended | Not a crawler — a training opt-out token | |
Applebot-Extended | Apple | Not a crawler — a training opt-out token |
PerplexityBot | Perplexity | Builds the search index |
CCBot | Common Crawl | Public archive used by many trainers |
Bytespider | ByteDance | Crawls for training data |
meta-externalagent | Meta | Crawls for training data |
Amazonbot | Amazon | Crawls for Alexa and assistant answers |
The -Extended tokens are the unusual case. Neither Google-Extended nor Applebot-Extended ever requests a page. Google's normal Googlebot and Apple's normal Applebot do the crawling; the Extended token is a signal about what may be done with the content after it has been fetched. You are not blocking a visitor. You are declining a use.
💡 Crawler names change — verify before you ship
This list reflects the position as of writing. Operators add tokens, rename them and split one crawler into three with some regularity — OpenAI and Anthropic have both done exactly that as their products grew. A blocked name that no longer exists silently does nothing.
Check the operator's own documentation before deploying, and read your server logs for the user agents actually hitting you. The logs are the ground truth; every published list, including this one, is a snapshot.
Training, indexing and answering are three different things
This is the distinction that determines whether your configuration does what you intended. AI crawlers arrive for three unrelated reasons, and lumping them together is how sites disappear from AI answers while trying to protect their content from training.
Category one is the one most people mean when they say "block AI". Categories two and three are how you get found — a citation in an AI answer is a referral, and the third category is a person who has already asked about you specifically.
🚨 The blanket block that costs you traffic
Advice to "block AI crawlers" often circulates as a list of every AI user agent, Disallow: / for all of them. Applied wholesale, that keeps you out of training data — and simultaneously removes you from ChatGPT's search results, Perplexity's citations, and the ability of any assistant to fetch your page when a user explicitly asks about it.
You have blocked the training you objected to and the discovery you wanted, in one file. If the goal is "not in the training set, still in the answers", you must block by category — which requires knowing which token is which.
The Google-Extended misconception
The most persistent piece of wrong advice in this area: that adding Google-Extended to robots.txt removes you from AI Overviews. It does not.
AI Overviews are a feature of Google Search. They are generated from Google's search index, which is built by Googlebot. Google-Extended governs a separate question — whether your content helps improve Gemini models and ground Vertex AI — and has no bearing on Search features.
Which leaves an uncomfortable position for anyone who genuinely wants out of AI Overviews:
- Block Googlebot — and leave Google Search entirely.
- Use
nosnippet— and lose your normal search snippets too. - Use
max-snippet:[n]— cap the text that can be shown, snippets included. - Use
data-nosnippet— exclude specific elements, the one genuinely surgical option.
There is no control that removes you from AI Overviews while leaving normal search results untouched. That is a deliberate product decision on Google's part, not an oversight, and no robots.txt directive routes around it. data-nosnippet on your most extractable passages is the closest available compromise.
Four configurations that do what they say
Pick the intent that matches yours rather than combining fragments from several.
1. Maximum AI visibility. You want citations and referrals, and you are not troubled by training use. For most content sites and tool sites this is the correct default:
2. Out of training, still in the answers. The configuration most publishers actually want, and the one that requires the token distinction:
3. Protect the paywall, publish the rest. Directory-level control, which is usually more proportionate than a site-wide decision:
4. Out of AI entirely. Honest about its cost — you are giving up AI referral traffic, not just training use:
robots.txt is a request, not a control
Every rule above works only because the crawler chose to read the file and honour it. Nothing enforces it. The protocol has no authentication, no verification and no penalty — a scraper that ignores robots.txt suffers no technical consequence whatsoever.
The major operators comply, for reasons that have little to do with the protocol: being caught ignoring robots.txt is a legal exposure and a reputational one, and the resulting news cycle is not worth the extra pages. That incentive holds for companies with lawyers and brands to protect. It does not extend to the long tail of scrapers reselling datasets.
⚠️ User agents are self-reported
A user agent string is a claim, not an identity. Anything can send User-Agent: Googlebot, and plenty does — blocking or allowing on the string alone is trivially defeated.
Real verification is a reverse DNS lookup on the requesting IP, checking it resolves into the operator's documented domain, then a forward lookup confirming that hostname resolves back to the same IP. Major operators publish their IP ranges for exactly this purpose. Anything less and you are trusting a string the client made up.
If you need enforcement rather than a request, it has to happen where requests are actually refused:
- CDN or WAF rules. Cloudflare, Fastly and the rest offer AI-crawler blocking as a toggle, applied before the request reaches you. Cloudflare has moved toward blocking AI crawlers by default for new domains — worth checking, because it means your content may already be blocked in ways your robots.txt does not reflect.
- Server-level blocking by verified IP rather than user agent string.
- Rate limiting, which catches aggressive scrapers regardless of what they call themselves.
- Authentication. The only genuine control. Content behind a login is not crawled, by anyone, and every other measure on this list is a speed bump by comparison.
What llms.txt actually is
Proposed by Jeremy Howard of Answer.AI in September 2024, /llms.txt addresses a real problem: an HTML page is mostly not content. Navigation, scripts, cookie banners and footers surround the part a model needs, and a context window spent on your header is context not spent on your documentation.
The proposal is a markdown file at your root — an H1 with the project name, a blockquote summary, then H2 sections containing annotated links. Optionally a companion /llms-full.txt carrying the full text rather than links.
The ## Optional section is the one piece of real semantics in the format: it marks links a model may skip when context is tight. Everything else is a conventional markdown link list, which is why generating the file is straightforward — if you already publish structured content, it is close to a sitemap with descriptions attached. Our Markdown to HTML converter is useful for checking the file renders as valid markdown before you publish it.
Does anything actually read it?
Here is where most coverage of llms.txt becomes unhelpfully optimistic. The honest answer, on the public evidence: no major AI provider has confirmed that its crawlers consume llms.txt.
Google has been explicit. John Mueller compared it to the keywords meta tag — a file site owners write about themselves, which is precisely the sort of self-declared signal search engines learned decades ago to distrust. Google's crawlers use the actual page. No other major operator has documented reading the file either.
⚠️ The circular evidence problem
The case for llms.txt is often supported by pointing out that Anthropic, Vercel, Cursor and other developer-tools companies publish one. They do. But publishing a file is evidence that someone thought it worth publishing — not evidence that any crawler reads it.
The reason those particular sites publish llms.txt is visible in who their users are: developers who paste documentation directly into an assistant's context. That is a real use, and it is a human-and-agent convenience, not a crawler feature. It says nothing about training pipelines or search indexes.
So the defensible position is narrow but not dismissive:
- It has a real user — people and agents fetching a clean index of your docs on purpose, which is common for technical documentation and rare for most other sites.
- It costs very little — one generated file, no risk, no maintenance beyond keeping links current.
- It has no demonstrated SEO or AI-visibility effect. Publish it for the first reason. Anyone selling it on the third is ahead of the evidence.
- A stale one is worse than none — a curated index that points at pages you moved is actively misleading, so generate it from the same source as your sitemap rather than maintaining it by hand.
What to actually do
In order of how much the decision matters:
| Step | Effort | Impact |
|---|---|---|
| Decide training vs indexing separately | Low | High |
| Check you are not blocking search crawlers by accident | Low | High |
| Confirm your CDN is not blocking on your behalf | Low | High |
| Verify against your own server logs | Medium | Medium |
| Move enforcement to the WAF if you need it | Medium | Medium |
| Publish llms.txt | Low | Low |
The third row catches people out. A CDN that blocks AI crawlers by default will happily override the permissive robots.txt you carefully wrote — your file says come in, the edge says no, and nothing in your repository reveals the conflict. Check the dashboard, not the file.
This site takes the first option deliberately. Every tool runs in the browser, there is nothing sensitive to protect from a crawler, and being cited in an AI answer is how someone with a file problem finds a tool that solves it. The robots.txt is correspondingly boring — Allow: /, a sitemap reference, and a few genuinely uninteresting directories excluded. When your content is the product and the product is free, blocking the readers is not a strategy.
Working on your site's markdown and structured files?
Convert, format and validate markdown, JSON and CSV entirely in your browser — nothing is uploaded to a server.
Browse all tools →Summary
- robots.txt is the file that matters. It is honoured, it is enforceable in practice by the operators that count, and it is easy to misconfigure.
- Training, indexing and user-triggered fetching are three separate things. Blocking all of them to stop the first costs you the other two.
- Google-Extended does not touch AI Overviews. It governs Gemini training; AI Overviews come from Googlebot and the Search index.
- There is no clean AI Overviews opt-out.
data-nosnippeton specific passages is the closest available. - Disallow is not noindex — block the crawl and the crawler never sees your noindex tag.
- User agents are self-reported. Verify by reverse DNS, or you are trusting a string.
- Real enforcement lives at the CDN, WAF or login, not in a text file.
- llms.txt is cheap and unproven. Publish it for the humans pasting your docs, not for a ranking effect nobody has demonstrated.
Frequently Asked Questions
What is llms.txt?
A proposed markdown file at the root of your site — /llms.txt — that gives language models a curated index of your most useful pages, so they do not have to parse navigation, scripts and boilerplate to find your actual content. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is a convention, not a standard, and no major AI provider has confirmed that its crawlers read it.
Does blocking Google-Extended remove my site from AI Overviews?
No, and this is the single most common misunderstanding. Google-Extended controls whether your content is used to improve Gemini models and Vertex AI grounding. AI Overviews are a feature of Google Search, crawled by Googlebot, so the only way to opt out of them is to restrict Googlebot or use snippet controls like nosnippet and max-snippet — which also cuts your normal search snippets.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls the open web to gather content for training. ChatGPT-User fetches a specific page because a user asked ChatGPT about it right then. OAI-SearchBot builds the search index behind ChatGPT's browsing. Blocking GPTBot keeps your content out of training while leaving you visible in answers; blocking OAI-SearchBot removes you from ChatGPT search results entirely.
Does robots.txt actually stop AI scrapers?
Only the ones that choose to obey it. robots.txt is a request, not an access control — it depends entirely on the crawler reading the file and honouring it. The major AI companies do honour it, because being caught ignoring it is a legal and reputational problem. Scrapers that do not care will ignore it completely, and stopping those requires blocking at the server, CDN or WAF layer by user agent and verified IP.
Should I add an llms.txt file to my site?
Add one if you publish documentation or reference material that people paste into an AI assistant, because the file is genuinely useful to those humans and agents. Do not add one expecting search or AI-visibility gains — there is no evidence any major crawler consumes it. It costs very little to publish, so the honest position is low cost, real but narrow benefit, and no measurable SEO effect.