How to allow GPTBot in robots.txt?

Add 'User-agent: GPTBot' followed by 'Allow: /' in your robots.txt file. To allow live ChatGPT search citations without model training, allow OAI-SearchBot instead.

Does blocking GPTBot affect my Google SEO ranking?

No. Google uses Googlebot to crawl and rank pages for Google Search. Blocking GPTBot or other AI crawlers has zero effect on your traditional Google SERP rankings.

Why is PerplexityBot not citing my website?

PerplexityBot requires Allow access in robots.txt. If your robots.txt has Disallow: / or Cloudflare WAF issues a 403 challenge, Perplexity will cite competitors or Reddit threads instead.

What is the difference between Googlebot and Google-Extended?

Googlebot indexes content for Google Search. Google-Extended controls whether Google can use your content to train Gemini and Vertex AI foundational models.

AI Robots.txt Generator — Configure Crawler Rules for ChatGPT, Claude & Perplexity

Strategy Presets:

1. Global & General Settings

Universal Crawlers

XML Sitemap URL (Optional)

Disallowed Paths (One per line)

2. AI Crawler Permission Matrix

Toggle permission for each major AI engine individually.

12 Crawlers

// robots.txt Preview

Place at: /robots.txt Test site live →

AI Crawlers Reference & Robots.txt Cheat-Sheet

Standardized syntax and recommendations for the major search engines and foundation model crawlers.

OAI-SearchBot Citations

Powers real-time search results and source links in ChatGPT Search. Recommended: Always Allow.

User-agent: OAI-SearchBot
Allow: /

PerplexityBot Citations

Indexes content for Perplexity AI answers and footnotes. Recommended: Always Allow.

User-agent: PerplexityBot
Allow: /

GPTBot Training

Collects training data for OpenAI models. Disallowing does not affect ChatGPT Search citations.

User-agent: GPTBot
Disallow: /

ClaudeBot Training

Anthropic model training crawler. Use Claude-Web to control interactive browsing in Claude chat.

User-agent: ClaudeBot
Disallow: /

Google-Extended Training

Controls Gemini and Vertex AI training. Disallowing does not affect Googlebot Search indexation.

User-agent: Google-Extended
Disallow: /

Applebot-Extended AI / Siri

Controls content ingestion for Apple Intelligence and Siri generative search features.

User-agent: Applebot-Extended
Allow: /

Step-by-Step Platform Setup & Troubleshooting Guides

Shopify Fix

Fix robots.txt.liquid blocking GPTBot & Perplexity.

Next.js App Router

Generate dynamic app/robots.ts & /llms.txt routes.

Cloudflare WAF

Bypass HTTP 403 & Bot Fight Mode for AI search.

Is Your robots.txt Working on Your Live Website?

Robots directives can be easily overridden by CDN caching, Cloudflare WAF, or server header misconfigurations.

Run Live 12-Crawler Test Now — Free