JavaScript Rendering and AI Crawlers: Why AI Engines Need Raw HTML
By the Leadee team at Netbase JSC · Updated
AI crawlers such as GPTBot, ClaudeBot and PerplexityBot fetch a page and read the HTML exactly as the server sent it — none of the major AI crawlers execute JavaScript. If your content only appears after client-side JavaScript runs, these bots see a mostly empty page no matter how good that content is. The fix is server-side rendering, static generation, or prerendering, so the facts a buyer needs are already in the HTML on arrival.
Key facts
- Over one month on Vercel's network, GPTBot made 569 million requests and Claude made 370 million, and none of the major AI crawlers tested — OpenAI's OAI-SearchBot/ChatGPT-User/GPTBot, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider, and Perplexity's PerplexityBot — executed JavaScript. (Vercel, 2024)
- ChatGPT's crawlers fetched a JavaScript file (without running it) on 11.5% of requests versus 23.84% for Claude's — meaning they sometimes download the script as plain text but never execute it, so its output never reaches them. (Vercel, 2024)
- Google renders JavaScript pages in a second wave: Googlebot first crawls and parses the raw HTML, then queues every 200-status page for rendering in a headless Chromium browser before using the rendered HTML to index it — a process that can take seconds or longer per page. (Google Search Central, 2026)
- Leadee's own check measures the ratio of visible text to total HTML in the raw server response, before any script runs, and fails a page outright when that ratio and paragraph count both come back too low — see the AI crawlability check.
Why do AI crawlers read raw HTML instead of running JavaScript?
Running a full browser engine per request — the only way to execute JavaScript and see what a user sees — is expensive at the scale AI crawlers operate at. Vercel's measurement of its own network found GPTBot alone made 569 million requests in a single month; fetching and parsing raw HTML text is cheap at that volume, spinning up a headless browser for each one is not.
That is a deliberate trade-off, not an oversight: the same Vercel data shows these crawlers will sometimes fetch a `.js` file as plain text (ChatGPT's bots did this on 11.5% of requests, Claude's on nearly a quarter) without ever executing it — they are reading bytes, not running code, even when the bytes happen to be a script.
How do I check what an AI crawler actually sees on my page?
Open view-source in your browser (not the regular "Inspect" panel, which shows the DOM after JavaScript has already run) — view-source shows exactly the HTML your server sent, before any script executes, which is the closest free approximation to what a non-JavaScript crawler receives.
If your page's main content — the product description, the service details, the facts a buyer actually needs — is missing from view-source and only appears in the rendered page, an AI crawler cannot read it. A near-empty `<body>` with a single `<div id="root"></div>` and a pile of `<script>` tags is the clearest sign of a client-side-only page.
- Right-click the page and choose "View Page Source" (or visit `view-source:https://yoursite.com/page`) — read it top to bottom for your actual content, not just the `<head>`.
- Search the raw source for a sentence you know is on the page. If you cannot find it there but can see it in the browser, JavaScript put it there after the fact.
- Run Leadee's AI crawlability check, which measures this programmatically: the ratio of visible text to total HTML, whether an `<h1>` exists, and whether there are enough real paragraphs in the raw response.
What are my server-side rendering options, by framework?
- Next.js / React: use Server Components or `getServerSideProps`/`generateStaticParams` so the page HTML is built on the server or at build time, not assembled in the browser after a client-side fetch.
- Vue / Nuxt: Nuxt's server-side rendering mode (the default in recent versions) renders full HTML per request instead of shipping a client-only single-page app.
- Angular: Angular Universal server-renders the initial page so the first response already contains real content.
- A fully static site (plain HTML, Hugo, Jekyll, Eleventy): already safe by default — there is no client-side rendering step to worry about.
- A JavaScript-only single-page app you cannot easily rewrite: a prerendering service (a headless-browser snapshot served only to known crawler user agents) is a workaround, not a long-term fix — it adds a maintenance burden and can drift out of sync with what users actually see.
What if only part of my site is client-rendered?
Most real sites are mixed: a server-rendered marketing site built in Next.js or WordPress next to a client-side-only interactive tool, dashboard, or configurator bolted on later. Check each template separately with view-source rather than assuming one result applies everywhere — a product catalogue page can be fully server-rendered while the "request a quote" calculator next to it is not, and that difference only matters for the calculator if the facts a buyer needs live inside it.
Prioritise the pages an AI engine would actually need to answer a buyer's question — product and service pages, pricing, locations, certifications — over interactive widgets whose content is secondary or decorative. A broken calculator costs you a feature; a client-side-only product page costs you the citation entirely.
What should I not do to "fix" this?
- Do not serve a different, simplified version of the page only to a crawler's user agent while leaving the real page client-rendered — that is cloaking, and if an AI engine ever detects the mismatch it damages trust far more than a slow fix would.
- Do not assume adding schema markup compensates for missing visible content — schema describes facts that should already be readable in the page text, not a replacement for it; see schema markup for B2B.
- Do not rely on a sitemap or llms.txt entry to "tell" an AI engine about content it cannot actually fetch and read — those files point to pages, they do not carry the content themselves.
- Do not treat a prerendering/snapshot service as permanent — plan the real server-side-rendering or static-generation migration, since a snapshot service is one more system that can silently fall out of date.
Questions
Does Googlebot have the same problem as GPTBot or ClaudeBot?
Less so. Googlebot does eventually render JavaScript in a second wave using a headless Chromium browser, though that rendering is queued and can take seconds or longer per page. AI crawlers such as GPTBot and ClaudeBot currently skip that step entirely, so for them there is no second chance.
My site uses a JavaScript framework — does that automatically mean AI crawlers cannot read it?
No. The framework itself is not the problem; whether it renders on the server or only in the browser is. Next.js, Nuxt and Angular Universal can all produce server-rendered HTML from the same codebase a client-only setup would otherwise ship.
Is a prerendering/snapshot service a real fix?
It works as a stopgap, but it is a second system to maintain and a potential cloaking risk if it drifts from what users see. Treat it as a bridge to real server-side rendering, not the destination.
How much of my content needs to be in the raw HTML?
The content a buyer would actually need to answer their question — what you make, what it costs, where you operate, how to contact you. Decorative or interactive elements that add no factual content can safely stay client-rendered.
Can I just check one page and assume the rest of my site is the same?
Not safely — a site can mix templates (a server-rendered product page next to a client-only interactive tool), so check your highest-traffic and highest-value pages individually, or run an automated check across several at once.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email [email protected] or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.
See what AI crawlers actually see on your pages
Run the free check to measure how much of your content is readable in the raw HTML, before any JavaScript runs.