Strategy Presets:

1. Global & General Settings

Universal Crawlers

XML Sitemap URL (Optional)

Disallowed Paths (One per line)

2. AI Crawler Permission Matrix

Toggle permission for each major AI engine individually.

12 Crawlers

// robots.txt Preview

Place at: /robots.txt Test site live →

AI Robots.txt Best Practices & Guidelines

Why You Shouldn't Use a Blanket "Disallow: /"

Blocking all AI user-agents prevents your website from being cited in ChatGPT Search, Perplexity, and Apple Intelligence answers. Instead of a total block, the industry standard is to Allow search citation bots (OAI-SearchBot, PerplexityBot, Claude-Web) while optionally disallowing general training scrapers (CCBot, Bytespider).

How to Install on Common Platforms

Shopify: Navigate to Online Store > Themes > Edit Code > Templates and add/edit robots.txt.liquid.
WordPress: Use an SEO plugin (RankMath / Yoast) under General Settings > Edit robots.txt, or upload the file directly to your web server root.
Next.js: Create app/robots.ts or place robots.txt directly inside your /public folder.

Is Your robots.txt Working on Your Live Website?

Robots directives can be easily overridden by CDN caching, Cloudflare WAF, or server header misconfigurations.

Run Live 12-Crawler Test Now — Free