Does llms.txt help me get cited by AI?

llms.txt is a bonus, not a prerequisite. It helps AI understand your site structure, but if your pages are blocked by robots.txt or rendered only via JavaScript, no llms.txt will save you. Fix access first.

My SEO is fine — why isn't that enough for AI search?

Google renders JavaScript and has decades of crawl history with your site. AI engines are newer, often don't execute JavaScript, and may be blocked by default firewall rules. Ranking #1 on Google and being citable inside an AI answer are two separate checks.

Why ChatGPT Doesn't Recommend Your Website (and How to Fix Each Reason)

Q: How do I check if ChatGPT can read my website?

Open yoursite.com/robots.txt and look for GPTBot, OAI-SearchBot or a wildcard Disallow: / rule. Then compare your homepage HTML fetched as a normal browser versus fetched as GPTBot — if the bot gets a fraction of the bytes, your content needs JavaScript to render.

You rank #1 on Google. Traffic is fine. Then a customer tells you they asked ChatGPT for a recommendation in your category — and your name never came up. You try it yourself. Three different phrasings, three answers, none of them you.

This is not an edge case anymore. A growing share of buyers ask an AI before they buy, and AI answers cite a handful of sources while ignoring everyone else. The uncomfortable truth: most websites are invisible to AI for reasons that have nothing to do with their content quality — and most of those reasons live at the front door of the site, where they can be found and fixed in under an hour.

Here are the five causes we see most often, ordered by how frequently they turn out to be the culprit.

Reason 1: Your robots.txt blocks AI crawlers (often accidentally)

robots.txt is a text file at yoursite.com/robots.txt telling crawlers what they may access. Many site owners blocked "aggressive" bots years ago during a SEO scare, or inherited overly strict rules from a website builder's defaults. Some builders ship templates that block everything except Googlebot.

The names to look for:

GPTBot / OAI-SearchBot / ChatGPT-User — OpenAI's crawler, its search index, and user-initiated browsing
ClaudeBot — Anthropic
PerplexityBot — Perplexity
Google-Extended — controls Gemini's use of your content
CCBot, Applebot-Extended — Common Crawl (feeds many models) and Apple Intelligence

A full-site block looks like this:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /        ← this one blocks EVERYONE, including every AI bot

The fix: delete those lines, or replace them with explicit permission. This is also what we recommend for your own site — being unambiguous helps conservative parsers:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Ten minutes, and it's the single highest-leverage change most sites can make.

Reason 2: Your content hides behind JavaScript

Modern site builders love client-side rendering: the server sends a nearly empty HTML shell, and your visitor's browser runs JavaScript to fill in the page. Humans see a beautiful site. An AI crawler that doesn't execute JavaScript sees a blank wall.

We test this by fetching the same homepage twice — once as a normal browser, once as GPTBot. On JS-heavy sites, the bot version regularly comes back with under 1% of the real content. One large, well-known deployment platform serves over 500 KB to browsers and about 3 KB to bots.

The fix: get your key content into the initial HTML — server-side rendering (SSR), static pre-rendering, or at minimum for your homepage and top product pages. Most frameworks (Next.js, Astro, Hugo) support this natively; many hosting platforms offer pre-rendering add-ons.

Reason 3: A leftover noindex tag

<meta name="robots" content="noindex"> is an explicit "do not index me" sign. It ends up on live sites more often than you'd think: staging flags left on, a CMS setting like "discourage search engines" still checked after launch.

The fix: view your homepage source, search for "noindex". If you find it, remove it — then check whether your site settings (WordPress "Search visibility" checkbox, Webflow/Squarespace indexing toggles) are also involved.

Reason 4: Your firewall or CDN blocks bot user-agents

Sometimes robots.txt says "come on in," but the server slams the door anyway. DDoS protection and WAF ("web application firewall") services often block any request whose user-agent looks automated — and GPTBot looks automated. We've seen sites where even a plain browser request from a datacenter IP got a 403.

The fix: whitelist known AI crawler user-agents in your WAF/CDN rules. If you use Cloudflare, check the Bots section; if you're on shared hosting, ask your provider. If a firewall intercepts everything, even our diagnostic probes get blocked — which itself is worth knowing.

Reason 5: The door is open, but nobody cites you anyway

If all four checks above pass and AI answers still ignore you, the cause moves from access to citation. AI answers lean heavily on certain source types: community threads (Reddit dominates), comparison/listicle articles, review platforms, and established media. A brand-new or thin site can be fully crawlable and still not be part of the answer network yet.

This is the hardest of the five to fix, but it starts with knowing who does get cited when buyers ask about your category — because those are the places where your presence would change the answer. That analysis (which third-party domains appear across dozens of buyer questions) is the most valuable part of a proper AI visibility report, and it's exactly what we run for you.

Your 30-minute self-check

Open yoursite.com/robots.txt — any Disallow lines naming AI bots, or a bare Disallow: /?
View page source on your homepage — Ctrl+F "noindex".
Ask ChatGPT to browse your site (or use any "fetch as bot" tool) — does it see your actual text?
Check your CDN/WAF logs for blocked requests from crawler user-agents.
Search your category in ChatGPT, Perplexity and Gemini — write down who does get cited.

Or let us run the front-door check right now — free, 30 seconds, nothing stored.

Check my site →

Frequently asked

How do I check if ChatGPT can read my website?

Look at robots.txt for GPTBot/OAI-SearchBot rules, then compare your homepage HTML fetched as a normal browser versus as GPTBot. If the bot version is a tiny fraction of the size, your content needs JavaScript to render — that's your problem.

Does llms.txt help me get cited?

It's a bonus, not a prerequisite. It helps AI systems understand your site structure, but no llms.txt file will save you if crawlers are blocked or your pages are empty without JavaScript. Fix access first.

My SEO is fine — why isn't that enough?

Because Google and AI engines are different customers. Google executes JavaScript and has decades of crawl history with you. AI engines are newer, often skip JavaScript, and trip over default firewall rules. Ranking #1 and being citable are two separate checks.