Generating Dynamic robots.ts and llms.txt in Next.js 14/15 App Router (2026 Guide)

In Next.js 14 and 15, handling search engine crawlers and emerging AI agents requires a cleaner, typed approach. Placing a static robots.txt inside the /public directory prevents environment-specific domain overrides and makes dynamic user-agent rules difficult to maintain.

1. Typed app/robots.ts Implementation

Create a new file at app/robots.ts. Using the native MetadataRoute.Robots type guarantees valid output for both Googlebot and modern AI search engines:

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://yoursite.com';

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/'],
      },
      {
        userAgent: ['OAI-SearchBot', 'PerplexityBot', 'Claude-Web'],
        allow: '/',
      },
      {
        userAgent: ['GPTBot', 'CCBot'],
        disallow: '/',
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

2. Serving /llms.txt via Route Handler

AI agents look for https://yoursite.com/llms.txt to understand your site structure. Create app/llms.txt/route.ts to serve this file with exact text/plain headers and cache controls:

import { NextResponse } from 'next/server';

export const dynamic = 'force-static';

export async function GET() {
  const content = `# Your Project Name
> High-performance developer platform optimized for AI search.

## Core Documentation
- [Quickstart Guide](https://yoursite.com/docs/quickstart): Fast onboarding guide.
- [API Reference](https://yoursite.com/docs/api): REST and GraphQL endpoints.
- [Pricing](https://yoursite.com/pricing): Transparent tiers and billing.
`;

  return new Response(content, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  });
}

By declaring export const dynamic = 'force-static', Next.js prerenders the route at build time, eliminating edge latency budgets for incoming crawlers.