Your homepage looks great in a browser. Fonts load, animations play, the product copy fades in beautifully. And yet when you ask ChatGPT about your category, your competitor gets cited and you don't.
One common reason: most AI crawlers never execute JavaScript. They request your page, take the raw HTML that comes back, and start reading. If your content only exists after a browser runs your app, what the bot receives may be an empty shell — a blank wall with a script tag on it.
We measure this gap on every site we audit. In one recent test, a well-known developer-tools company's homepage served our browser around 512 KB of HTML. Fetching the same URL with GPTBot's user agent returned about 3 KB. That site happens to serve bots a compact markdown version on purpose — content parity by design, which is fine. But when we run the same test on ordinary business sites and get 3 KB back, it's usually not a strategy. It's an accident, and the content simply isn't there.
Why AI bots skip your JavaScript
Rendering a modern web app is expensive: it means running a full browser engine per page. Google can afford a dedicated rendering pipeline because search has been their business for twenty-five years. AI crawlers mostly don't have one:
- GPTBot (OpenAI's training crawler) and OAI-SearchBot (the crawler behind ChatGPT's citations) fetch raw HTML.
- ClaudeBot (Anthropic) and PerplexityBot likewise collect the HTML as served.
- ChatGPT-User does browse interactively when a user asks it to open a specific page — but it isn't the channel that decides who gets cited in answers.
If the raw HTML contains no headlines, no product description, no pricing — there is nothing to quote. You won't show up in training data, in search indexes built from crawls, or in cited sources.
The two-minute test
Test 1: view-source (no tools needed)
- Open your homepage in a browser.
- Press
Ctrl+U(or right-click → View Page Source). This shows the raw HTML — exactly what a crawler receives first. - Search inside it (
Ctrl+F) for your main headline and a sentence of your core product copy.
Found them? Good — your front door is open. Only seeing a nearly empty <body> with a <script> tag? That's the blank wall.
Test 2: ask for the page as a bot
curl -sL -A "GPTBot" https://yoursite.com | wc -c
curl -sL https://yoursite.com | wc -cThe first number is roughly what OpenAI's crawler receives; the second is a plain fetch. Then eyeball the actual content:
curl -sL -A "GPTBot" https://yoursite.com | grep -i "<h1">How to read the result:
- Similar sizes, real text present → server-rendered or static. You're readable.
- Tiny bot response, big browser page → JS wall. Check whether what the bot gets is meaningful content or just a shell.
- Bot gets less but it's clean markdown/text → some sites deliberately serve bots a simplified version with the same substance. That counts as readable — parity of content is what matters, not byte equality.
Four ways to fix a blank wall
1. Move rendering to the server (SSR)
Frameworks like Next.js, Nuxt, SvelteKit and Rails can render HTML on the server per request. Bots get full content; your app still hydrates in the browser. This is the standard fix and usually a configuration-level change, not a rewrite.
2. Pre-render to static HTML (SSG)
If your pages don't change per visitor — marketing sites, docs, blogs — build them to plain HTML ahead of time (Astro, Next.js static export, Hugo). Fastest option for bots and humans alike, and the hardest one to get wrong.
3. Prerendering middleware
Tools like Prerender.io detect known bot user agents and serve them a cached, fully rendered snapshot while browsers keep getting the SPA. Less elegant than SSR, but useful when rebuilding isn't feasible soon. One caution: keep the snapshot genuinely equivalent to what users see — serving bots thin or different content is cloaking territory.
4. Fix the highest-value pages first
You may not need every route server-rendered this week. Homepage, pricing, and your three most important product pages carry almost all of your citable surface area. Render those, and the long tail can follow.
Want the wall test plus robots, noindex and llms.txt checked at once — free, 30 seconds, nothing stored?
Run the front-door check →Your self-check list
- Ctrl+U on your homepage: is your headline in the raw HTML?
curl -sL -A "GPTBot" https://yoursite.com | wc -c: within shouting distance of the browser version?- Same test on your pricing page — it's the page buyers ask AI about most.
- Already passed all three? Then check the other front-door issues: robots.txt blocks (post #1 covers all five) and whether an llms.txt would help you now.
Frequently asked
Do AI crawlers like GPTBot render JavaScript?
Generally no. Most AI crawlers fetch the raw HTML and extract text without executing scripts. Googlebot has a dedicated rendering pipeline; GPTBot, ClaudeBot, PerplexityBot and most others don't operate one at scale. Client-rendered content largely doesn't exist for them.
How do I check if my site works without JavaScript?
View-source (Ctrl+U) and look for your main copy in the raw HTML, or fetch the page with curl -A "GPTBot" and compare against what a browser sees. Both take under two minutes.
Does client-side rendering hurt regular SEO too?
It can. Google can render JS, but in a slower second pass — fresh pages may sit unrendered for days or weeks. SSR or static generation removes that queue risk for search engines and makes you readable to AI crawlers at the same time.