Robots.txt for AI Crawlers: Search, Training, and User Fetches

The robots.txt file tells web crawlers what they can and can't access on your website. Traditionally, this meant Googlebot and Bingbot. But now there's a new generation of AI crawlers, and many websites are blocking them without realizing it.

Blocking a search crawler can prevent your pages from being indexed for that provider's current search answers. Training crawlers and user-requested fetchers serve different purposes. This guide shows how to configure each category independently.

What the Data Shows About AI Crawler Blocking

Before configuring your robots.txt, it helps to understand the landscape. AI agent traffic grew over 6,900% year-over-year in 2025, making this decision increasingly consequential.

A 2025 BuzzStream study of top news publishers found that 79% block AI training bots via robots.txt, while 71% also block AI retrieval bots. The most-blocked crawlers: ClaudeBot (69% of sites), PerplexityBot (67%), and GPTBot (62%).

Cloudflare's Q1 2026 data reveals that 89.4% of AI crawler traffic is training or mixed-purpose, while only 8% is search-related and just 2.2% responds to actual user queries in real time. This distinction matters for your blocking strategy.

The Three-Tier Crawler Framework (2026)

As of 2026, major AI companies no longer use a single crawler. They've split into three tiers, and your robots.txt strategy needs to account for each:

Key insight: Blocking a training bot does not block the provider's separate search crawler. OpenAI documents that a site can block GPTBot while allowing OAI-SearchBot for ChatGPT search. Anthropic likewise documents independent controls for ClaudeBot, Claude-SearchBot, and Claude-User.

Complete AI Crawler Reference Table

Here's the complete list of AI crawlers as of April 2026, updated to include Apple, Meta, DuckDuckGo, and Common Crawl agents that launched in 2024-2025. ClaudeBot has approximately doubled its crawl rate between Q3 2025 and Q1 2026, suggesting Anthropic is scaling retrieval infrastructure significantly (TechnologyChecker.io, Q1 2026). Meta-ExternalAgent launched in July 2024 for Llama training (Meta Developers, 2024); Applebot-Extended launched June 2024 as an AI-training opt-out control for Apple Intelligence (Apple Support, 2024).

Sources: Search Engine Journal (Dec 2025), ALM Corp (2026), Anthropic documentation, Apple Support (119829, 120320), Meta Developers (externalagent crawler docs), DuckDuckGo Help Pages (duckassistbot), Common Crawl (commoncrawl.org/ccbot). Note: anthropic-ai and Claude-Web are deprecated. Use ClaudeBot, Claude-SearchBot, and Claude-User instead.

Three crawlers worth their own callout:

• Applebot-Extended is a control token. It does not fetch pages of its own. Apple's regular Applebot does the crawling; blocking Applebot-Extended tells Apple not to use the content for generative-AI training, while still leaving you indexable for Siri, Spotlight, and Apple search. Roughly 6-7% of high-traffic sites block it today, mostly news publishers including The New York Times, The Financial Times, The Atlantic, Vox Media, and Condé Nast (Apple Support, 2025).
• Meta-ExternalFetcher can bypass robots.txt for user-initiated URLs. Meta states that facebookexternalhit and meta-externalfetcher may ignore robots.txt when a user explicitly provides a URL as context to a Meta AI product, the same carve-out that Perplexity-User and ChatGPT-User apply. A hard block on user-triggered fetches requires firewall rules because robots.txt alone is insufficient (Meta Developers, 2026).
• CCBot blocks propagate slowly. Common Crawl publishes snapshots quarterly, and older snapshots live forever in derivative training datasets like The Pile and RedPajama. Blocking CCBot today removes you from future snapshots; content you already published remains in circulation for years (Common Crawl, 2025). Treat it as a long-horizon decision. It cannot remove prior snapshots immediately.

Recommended robots.txt Configuration

For businesses that want search visibility and also choose to allow potential-training and grounding use, this is a clear starting point:

# Standard search engine crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# OpenAI (all three tiers)
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic (all three tiers)
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Perplexity (both tiers)
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Google AI features
User-agent: Google-Extended
Allow: /

# Amazon AI
User-agent: Amazonbot
Allow: /

# Default: allow everything else
User-agent: *
Allow: /

# Block private/admin areas
Disallow: /admin/
Disallow: /api/
Disallow: /auth/

Sitemap: https://yoursite.com/sitemap.xml

How to Selectively Allow or Block AI Crawlers

The three-tier framework enables a nuanced strategy. You can allow search discovery while blocking potential-training use. User-requested fetchers have separate behavior:

# ALLOW search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-requested fetchers
User-agent: Claude-User
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Perplexity-User
Allow: /

# BLOCK potential-training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Gemini training and grounding
User-agent: Google-Extended
Disallow: /

Note: Blocking GPTBot or ClaudeBot expresses a preference against potential training use. It does not make search results stale when the separate search crawlers remain allowed. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because the fetch is user requested.

Google-Extended has a wider scope: Google says this control covers future Gemini model training and grounding in Gemini Apps and Vertex AI. Blocking it does not affect Google Search, but it can reduce Gemini Apps and Vertex AI grounding visibility for your content.

Compliance caveat: Bytespider (ByteDance) claimed robots.txt compliance but was observed accessing disallowed paths on test sites within 30 days of applying a block. Consider firewall-level blocking for crawlers you don't trust to honor robots.txt.

How to Edit robots.txt on Popular Platforms

WordPress

Edit via Yoast SEO plugin: SEO → Tools → File editor → robots.txt. Or create/edit the file at your site root.

Squarespace

Go to Settings → SEO → scroll to "Additional Robots.txt Rules" and add your AI crawler rules there.

Wix

Go to Dashboard → Settings → SEO (Google) → SEO Tools → Robots.txt Editor.

Shopify

Shopify auto-generates robots.txt. Edit it via theme.liquid or use a Shopify robots.txt app.

Next.js / Vercel

Create a robots.ts file in your app/ directory or add a static robots.txt in public/.

Common Mistakes

1. Using a wildcard Disallow that blocks AI crawlers User-agent: * Disallow: / blocks everything, including AI crawlers.
2. Not checking platform defaults: Some CMS platforms add AI crawler blocks automatically. Always check after setup.
3. Checking training bots instead of search bots: GPTBot and ClaudeBot govern potential training use. Check OAI-SearchBot andClaude-SearchBot when diagnosing search visibility.
4. Forgetting to add a sitemap reference: Always include Sitemap: https://yoursite.com/sitemap.xml at the end of robots.txt.

How to Verify Your Configuration

1. Visit yoursite.com/robots.txt in your browser
2. Check that OAI-SearchBot, Claude-SearchBot, and PerplexityBot are allowed for public pages
3. Run a free Technical Audit It checks all AI crawlers and shows exactly which are blocked

Frequently Asked Questions

What is GPTBot?

GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. It does not control ChatGPT search inclusion. OpenAI uses OAI-SearchBot for search and ChatGPT-User for certain user-triggered visits. Each setting is independent.

Should I allow AI crawlers on my website?

Businesses that want current AI search visibility should allow the relevant search crawlers, including OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Potential-training access through GPTBot and ClaudeBot is a separate site-owner choice. Google-Extended controls both Gemini model training and grounding in Gemini Apps and Vertex AI. User-triggered fetchers form a third category and may follow different robots.txt behavior.

Does Squarespace block AI crawlers?

Yes, some Squarespace sites block AI crawlers by default in their robots.txt. Check your site's robots.txt to confirm, and contact Squarespace support if you need to modify it.

What AI crawlers should I allow?

For search visibility, allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Claude-User supports user-requested retrieval. ChatGPT-User and Perplexity-User also support user actions, but their providers say robots.txt may not always apply. Decide separately whether to allow GPTBot and ClaudeBot for potential training. Google-Extended controls both Gemini model training and grounding in Gemini Apps and Vertex AI, without affecting Google Search.

How do I check if AI crawlers are blocked on my site?

Visit yoursite.com/robots.txt and look for Disallow rules targeting OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Check firewall rules against each provider's published IP ranges. Or use Foglift's free Technical Audit It automatically checks AI crawler access as part of the AI Readiness Score.

Sources & Further Reading

1. OpenAI, “Overview of OpenAI Crawlers”: OAI-SearchBot, GPTBot, and ChatGPT-User
2. Anthropic crawler controls, updated April 7, 2026: ClaudeBot, Claude-SearchBot, and Claude-User
3. Perplexity crawler documentation: PerplexityBot for search and Perplexity-User for user requests
4. Google crawler documentation: Google-Extended controls Gemini training and grounding, with no effect on Google Search
5. BuzzStream, “Which News Sites Block AI Crawlers in 2025? [New Data],” 2025. 79% of top news sites block AI training bots; 70.6% of blocking sites still cited; 23.1% traffic decline for blockers.
6. TechnologyChecker.io, “We Analyzed robots.txt Across Cloudflare's Network: Which AI Crawlers Get Blocked Most and Why,” Q1 2026. ClaudeBot doubled crawl rate Q3 2025 → Q1 2026.
7. Search Engine Journal, “Cloudflare Report: Googlebot Tops AI Crawler Traffic,” 2025. 89.4% of AI crawler traffic is training/mixed; 2.2% real-time user queries.
8. Search Engine Land, “Googlebot dominates web crawling in 2025 as AI bots surge,” 2025. GPTBot share decreased from 35.46% to 28.97% as blocking increased.
9. Paul Calvano, “AI Bots and Robots.txt,” Aug 2025. Analysis of AI crawler blocking patterns across the web.
10. Search Engine Journal, “Complete Crawler List For AI User-Agents,” Dec 2025. Verified user-agent list from real server logs with IP validation.
11. Search Engine Journal, “Anthropic's Claude Bots Make Robots.txt Decisions More Granular,” 2026. Three-bot framework: ClaudeBot, Claude-SearchBot, Claude-User.
12. ALM Corp, “ClaudeBot, Claude-User & Claude-SearchBot: Anthropic's Three-Bot Framework,” 2026. Each bot honors robots.txt independently.
13. Apple Support, “About Applebot” (article 119829) and “Applebot model training and individual privacy rights” (article 120320), updated 2025. Applebot-Extended is an AI-training opt-out control. Applebot continues to index for Siri, Spotlight, and Apple search regardless.
14. Meta for Developers, “Meta crawler documentation” (developers.facebook.com/docs/sharing/webmasters/crawler), 2024-2026. Meta-ExternalAgent launched July 2024 for training; Meta-ExternalFetcher handles user-initiated fetches and may bypass robots.txt when a user supplies an explicit URL.
15. DuckDuckGo Help Pages, “Is DuckAssistBot related to DuckDuckGo?” 2025. DuckAssistBot/1.2 crawls on-demand for DuckAssist answers; robots.txt changes take effect within 72 hours; data is not used for model training.
16. Common Crawl, “CCBot” (commoncrawl.org/ccbot), 2025. CCBot now runs on dedicated IP ranges with reverse DNS for verification; Common Crawl snapshots feed derivative training datasets (The Pile, RedPajama, C4), so blocks propagate slowly.

Related Reading

• How to Optimize Your Website for ChatGPT
• What Is Generative Engine Optimization (GEO)?
• GEO vs SEO: What's the Difference?
• How to Appear in AI-Generated Answers
• Free Technical Audit: What to Check

Related: Learn about AEO (Answer Engine Optimization), the framework for making your content extractable by AI answer engines.

Fundamentals: Learn about GEO (Generative Engine Optimization) and AEO (Answer Engine Optimization) (the two frameworks for optimizing your content for AI search engines).