How to Allow GPTBot & PerplexityBot Through Cloudflare WAF & Managed Challenges

One of the most frustrating bottlenecks in generative search optimization is discovering that your robots.txt is completely open, yet AI search engines report your website as inaccessible. In over 60% of verified cases, the blocker is not your origin web server—it is Cloudflare's Managed Challenges returning HTTP 403 or soft captchas to automated crawlers.

Why Cloudflare Challenges Fail AI Search Bots

When an end user asks ChatGPT or Perplexity for purchase advice or documentation references, the AI engine triggers a real-time retrieval crawler (OAI-SearchBot or PerplexityBot). Because these crawlers operate under strict headless latency budgets (<1.8 seconds), they cannot solve Turnstile interactive puzzles or wait for JavaScript hydration shells. Cloudflare interprets this timeout as a failed challenge and drops the connection.

Creating a WAF Custom Skip Rule

To allow legitimate AI search crawlers without turning off Bot Fight Mode for malicious scrapers, configure a Cloudflare Custom WAF Rule:

1. Log in to your Cloudflare Dashboard and select your domain.
2. Navigate to Security > WAF > Custom Rules.
3. Click Create Rule, name it Allow AI Search Crawlers, and paste the expression below:

(http.user_agent contains "OAI-SearchBot" or
 http.user_agent contains "PerplexityBot" or
 http.user_agent contains "Claude-Web" or
 http.user_agent contains "Applebot-Extended")

Under the Choose Action dropdown, select Skip, and check the following security components to bypass:

• WAF Managed Rules
• Browser Integrity Check
• Security Level (Managed Challenge)

Verifying with Live Request Logs

Navigate to Security > Analytics & Logs. Trigger a test search in ChatGPT or use our free diagnostic tool. Filter by User Agent: OAI-SearchBot and verify that requests return HTTP 200 OK with action Skip rather than Managed Challenge.