Robots.txt for LLMs: The 2026 State of Play
Robots.txt is no longer a single-purpose file. It is the policy surface where a site declares its position on at least nine distinct categories of automated visitor: classic search crawlers, AI training crawlers, AI retrieval crawlers, AI user-agents fetching on behalf of a human, advertising bots, security scanners, research crawlers, archive bots, and impersonators. The same text file now has to express different rules for visitors with different commercial implications, and the wrong directive can quietly cost a property its citation surface across ChatGPT, Claude, Perplexity and AI Overview. This piece is a working map of the 2026 state of robots.txt for LLM-era operators.
What Changed Between 2023 and 2026
Three years ago, the robots.txt file on most commercial sites had two stanzas: a generic `User-agent: *` block, and occasionally a Googlebot-specific section. The file rarely changed. Today, the same file commonly carries between six and twelve user-agent stanzas, each with distinct allow and disallow lists. The shift was forced by three vendor-side changes.
OpenAI introduced GPTBot in August 2023 with a documented respect for robots.txt. Anthropic followed with ClaudeBot. Perplexity, after early controversy, formalised PerplexityBot and Perplexity-User as separate identifiers. Google published Google-Extended as a separate token to let publishers opt out of AI training without leaving Search. Common Crawl, the upstream corpus that feeds many smaller models, also added an identifier (CCBot) that more publishers began controlling.
The compounding effect is that a 2023-era robots.txt is functionally underspecified in 2026. A site that blocked nothing in 2023 is now opted in to training inclusion across at least eight crawlers without ever making an explicit decision. A site that blocked everything is invisible to retrieval surfaces that drive a growing share of branded discovery.
The Policy Matrix
Operators need to make four distinct policy decisions, not one. Each decision maps to a different set of user-agents.
Decision 1: Should the property allow training-corpus inclusion? User-agents: GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot, Applebot-Extended, Meta-ExternalAgent. Blocking these removes the property from future model training. The effect is gradual (current models already trained on past snapshots are unaffected). The decision is irreversible at the corpus level but reversible at the property level.
Decision 2: Should the property allow real-time retrieval and citation? User-agents: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended (partial). Blocking these removes the property from AI Search products. The effect is immediate. A site cannot be cited by an engine it has disallowed.
Decision 3: How should classic search be treated? User-agents: Googlebot, Bingbot, DuckDuckBot. The default in 2026 remains permissive for most commercial properties. Exceptions exist for sites in legal or compliance categories with explicit reasons to limit indexing.
Decision 4: Which non-AI bot traffic should be filtered? User-agents: SemrushBot, AhrefsBot, MJ12bot, dotbot. Mostly a bandwidth question. Some legal portfolios additionally block these for competitive intelligence reasons. The robots.txt file is not the right place for security blocks (those belong at the WAF), but it is the polite-traffic filter.
Observed Anti-Patterns in 2026 Robots.txt Files
Three patterns recur in the audits we run on commercial properties, and each one is costing the property something.
Pattern A: The blanket disallow. A robots.txt that disallows every named AI bot under the assumption that this protects IP. In practice it removes the property from citation surfaces that would otherwise drive branded discovery. For a 25,000-page lender we audited in 2026, this configuration combined with a WAF that 403-blocked unknown bots resulted in an 8% ChatGPT mention rate against a category leader at much higher rates. The site had effectively opted out of the citation layer without ever being asked to.
Pattern B: The copy-paste from a popular gist. Files pulled from public lists that were last updated in 2023, missing OAI-SearchBot, Applebot-Extended, and Meta-ExternalAgent. The site is opted in to all three by default because their tokens are absent from the file. This is the most common pattern we see, and the easiest to remediate.
Pattern C: Disallow rules that contradict the sitemap. A robots.txt that disallows `/blog/archive/` while the XML sitemap still lists those URLs. Bots that respect robots.txt skip the URLs. Bots that ignore it (impersonators) still fetch them. The contradiction is a noise source that distorts every diagnostic that follows.
A Reference Robots.txt for a Commercial Property in 2026
Annotated reference (allow training, allow retrieval, block scrapers)
# Classic search: permissive User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # AI training corpora: opt in for content surfaces User-agent: GPTBot Allow: / Disallow: /account/ Disallow: /checkout/ User-agent: ClaudeBot Allow: / Disallow: /account/ User-agent: Google-Extended Allow: / # AI retrieval (live citation surface): allow User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # SEO scrapers: block (bandwidth and competitive) User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / # Default User-agent: * Disallow: /account/ Disallow: /checkout/ Disallow: /search? Sitemap: https://example.com/sitemap.xml
This template assumes a permissive AI posture appropriate for editorial and commercial content properties. Sites with proprietary research, paywall content, or compliance-driven restrictions need a tighter version. The structure (named-agent stanzas first, wildcard last) matters: most parsers apply the most specific match.
The Verification Layer
Publishing the file is not enough. Three checks need to run after every change.
The first check is robots.txt parsability against Google’s tester and against an open-source parser like google/robotstxt (the same library Googlebot uses, available on GitHub). A directive that looks valid to a human can be silently ignored by a parser because of a misplaced colon or a wildcard in the wrong position. The lender audit referenced earlier surfaced a directive that the Google parser accepted but the Bing parser interpreted differently, which had been costing 11% of crawled URLs.
The second check is log verification. Pull a 14-day window of access logs after the file changes and confirm each named user-agent is requesting only the directories the file permits. A bot that continues to fetch a disallowed directory either does not respect robots.txt (in which case it should be blocked at the WAF) or is being impersonated (in which case the source IP should be verified).
The third check is citation observability. Run a fixed prompt panel against the engines whose user-agents the file allows, before and after the change, to confirm the policy lift translates into citation share. The methodology for this panel is in our AI visibility audit, and the page-level patterns that move citation rates given a permissive robots.txt are in how LLMs decide which sources to cite.
Common Operator Questions
Does blocking AI bots improve organic SEO? No. Googlebot and the AI bots are independent crawlers. Blocking GPTBot or ClaudeBot has no effect on Google’s index. The conflation is the most common misconception we encounter on intake calls.
Does Crawl-delay matter for AI bots? Most AI bot vendors do not honour Crawl-delay. For rate-limiting, the WAF is the correct enforcement point. For polite-traffic shaping, the most reliable lever is the Sitemap declaration and the freshness signal on individual URLs, not the Crawl-delay directive.
Should robots.txt be different in staging? Yes. Staging environments should carry a full `Disallow: /` for `User-agent: *` and additionally for every named bot in the production file. The most painful version of this failure is a production-grade file accidentally deployed to a publicly accessible staging host, which can leak unannounced content into AI training corpora before the brand has approved publication.
Five Actions to Take Before Next Quarter
- Audit the current robots.txt against the 2026 user-agent list. Confirm explicit positions on GPTBot, ClaudeBot, OAI-SearchBot, Perplexity-User, Google-Extended, CCBot, Applebot-Extended, and Meta-ExternalAgent. Any missing entry is an implicit allow.
- Reconcile robots.txt against the XML sitemap. Any URL in the sitemap that the file disallows is a contradiction worth fixing.
- Run the file through the Google robots.txt tester and an open-source parser. Parser disagreements are the diagnostic.
- Pull a 14-day post-change log window. Verify per-bot compliance and source-IP authenticity.
- Document the policy decision behind each directive. A file changed without a written rationale is one that will get overwritten by the next person who touches it.
Frequently Asked Questions
What happens if I block GPTBot but allow OAI-SearchBot?
The site is excluded from OpenAI’s training corpus but remains eligible for citation in ChatGPT Search. This is the most common configuration for commercial properties that want retrieval visibility without contributing to training. The two user-agents are independent.
Will robots.txt prevent my content from being used in AI answers?
Partially. Vendors that respect the file (OpenAI, Anthropic, Google, Perplexity for their disclosed bots) will exclude the property. Vendors that ignore it, and content already in past training snapshots, are unaffected. Robots.txt is a polite-traffic instrument, not a legal enforcement mechanism.
Does the order of user-agent stanzas matter?
Yes for some parsers. Google’s parser applies the most specific match, so order is less critical, but legacy parsers used by smaller crawlers may follow first-match semantics. Listing named agents before the wildcard stanza is safer.
Can I use robots.txt to control AI Overview specifically?
Indirectly. Google-Extended controls AI training inclusion for Gemini and is the closest direct lever. Google’s AI Overview inherits the Search index, so blocking Googlebot would also remove the property from AI Overview, but at the cost of organic Search visibility. Most operators allow Googlebot and use other levers for AI Overview optimisation.
Should noarchive or nosnippet meta tags be used alongside robots.txt?
For page-level control over Google’s display behaviour, yes. The meta robots directives operate at the page level and are independent of robots.txt directory rules. The two layers serve different purposes and are not interchangeable.
Get the current robots.txt of your property audited against the 2026 user-agent map, with a per-stanza rationale and a 14-day log verification pass.