In 2026, the web crawler landscape is no longer just Googlebot and Bingbot. Over a dozen distinct AI user-agents actively crawl the web. Some exist to provide real-time search citations (driving high-intent buyer traffic), while others scrape training datasets for foundational models.
Understanding the exact function of each user-agent is critical: a single misplaced Disallow: / can completely wipe out your presence in ChatGPT Search, Perplexity, and Apple Intelligence.
Visual AI Robots.txt Builder
Select from 12+ bots with one-click presets and copy the generated rules.
The 12 Major AI Crawlers (2026 Breakdown)
| User-Agent | Owner | Primary Purpose | Recommendation |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT Search indexing & citations | ALLOW |
| GPTBot | OpenAI | Foundational model training data | OPTIONAL |
| PerplexityBot | Perplexity | Live AI search citations | ALLOW |
| Claude-Web | Anthropic | Real-time web lookups in Claude | ALLOW |
| ClaudeBot | Anthropic | Anthropic Claude model training | OPTIONAL |
| Applebot-Extended | Apple | Apple Intelligence & Siri search | ALLOW |
| Google-Extended | Gemini training (does not affect Search) | OPTIONAL | |
| Meta-ExternalAgent | Meta | Llama & Meta AI data scraping | OPTIONAL |
| CCBot | Common Crawl | Open web dump (high bandwidth scraper) | DISALLOW |
| Bytespider | ByteDance | Aggressive scraper for Doubao LLMs | DISALLOW |
The Recommended 2026 robots.txt Template
This rule set maximizes citations in ChatGPT Search, Perplexity, and Apple Intelligence while blocking heavy background scrapers and protecting admin routes:
# ========================================================
# Production AI Search robots.txt Specification
# ========================================================
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /private/
Allow: /
# --- Allow AI Search & Citation Engines ---
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Applebot-Extended
Allow: /
# --- Disallow Heavy Training / Bulk Scrapers ---
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlTest Your Current robots.txt Configuration Live
Verify if your live server is correctly serving these directives to all 12 bots without firewall interference.