Next.js & React AI Search Optimization — App Router AEO Guide

01

Accidental `'use client'` at Route Roots (The JS Wall)

Placing 'use client' at the top of your page file turns the entire subtree into Client-Side Rendering (CSR). When an AI search engine fetches your URL with a raw HTTP GET request, it receives an empty <div id="__next"> shell. AI crawlers do not wait for React hydration bundles.

// Correct App Router Architecture
// page.tsx (Server Component by default — renders full HTML payload)
import InteractiveWidget from './Widget'; // Client Component boundary pushed to leaf

export default async function ProductPage() {
  const data = await fetchContent();
  return (
    <article>
      <h1>{data.title}</h1> {/* Raw HTML immediately readable by GPTBot */}
      <p>{data.summary}</p>
      <InteractiveWidget />
    </article>
  );
}

02

Native `app/robots.ts` Configuration

Next.js has first-class support for programmatic robots.ts generation. Make sure you don't use a catch-all block that accidentally excludes AI user-agents.

// app/robots.ts

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/' },
      { userAgent: ['GPTBot', 'OAI-SearchBot', 'PerplexityBot', 'ClaudeBot'], allow: '/' }
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  };
}

03

Programmatic `/llms.txt` via Route Handler

Serve a real-time, clean markdown index of your documentation or product catalog using a Next.js App Router Route Handler:

// app/llms.txt/route.ts

export async function GET() {
  const content = `# Your Project\n\n> Core proposition\n\n## Documentation\n- [API](/docs/api)`;
  return new Response(content, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' }
  });
}

Next.js AI Visibility: Break the JavaScript Wall

The 3 Critical Traps in Next.js Apps

Accidental `'use client'` at Route Roots (The JS Wall)

Native `app/robots.ts` Configuration

Programmatic `/llms.txt` via Route Handler

Audit Your Next.js App for AI Search

The 3 Critical Traps in Next.js Apps

Accidental 'use client' at Route Roots (The JS Wall)

Native app/robots.ts Configuration

Programmatic /llms.txt via Route Handler

Audit Your Next.js App for AI Search

Accidental `'use client'` at Route Roots (The JS Wall)

Native `app/robots.ts` Configuration

Programmatic `/llms.txt` via Route Handler