Site Speed for AI Crawlers: TTFB, Compression and What Actually Matters
By the Leadee team at Netbase JSC · Updated
Time to First Byte (TTFB) — how long your server takes to start responding, before content even arrives — is the speed metric that matters most to an AI crawler, since a crawler budgets a fixed amount of time per request and gives up on the slowest sites. Leadee treats TTFB under 600ms as fast, up to 1,800ms as workable, and above that as a real problem. A smaller, compressed HTML response and server-side caching are the two highest-leverage fixes.
Key facts
- Google's web.dev rates Time to First Byte (TTFB) "good" at 0.8 seconds or less, "needs improvement" up to 1.8 seconds, and "poor" above 1.8 seconds, while noting the right target still depends on whether a page is server-rendered or client-rendered. (web.dev (Google), 2025)
- Text compression (gzip or Brotli) commonly cuts text-based files such as HTML, CSS and JavaScript by 70-90%, per web.dev's own compression benchmarks — a meaningful reduction for exactly the kind of page an AI crawler fetches. (web.dev (Google), 2023)
- Over one month on Vercel's network, GPTBot alone made 569 million requests — at that volume, every crawler operator caps how long it will wait per request, since a slow site costs it real compute time across millions of fetches. (Vercel, 2024)
- Leadee's own check measures raw TTFB from the homepage fetch: 600ms or under scores as fast, up to 1,800ms scores as a partial pass, and anything slower scores as a fail — see the AI crawlability check.
What is TTFB, and why does an AI crawler care about it specifically?
Time to First Byte is the delay between a request going out and the first byte of the response coming back — it covers DNS lookup, connection setup, and however long your server or application takes to generate the page, before a single character of content has even started downloading. It says nothing about how a page looks once loaded; it only measures how long the server made the requester wait.
An AI crawler fetching hundreds of millions of pages a month cannot afford to wait indefinitely on any one of them, so slow TTFB risks a crawler giving up on a page (or your whole site) before content ever arrives — unlike a human visitor, who might wait a few extra seconds, a crawler simply moves on.
Does site speed mean the same thing for crawlers as it does for human visitors?
No, and the difference matters for where you spend effort. A human visitor's experience is dominated by metrics like Largest Contentful Paint and Cumulative Layout Shift — how fast the page looks finished and how much it jumps around while loading, both of which depend heavily on images, fonts and client-side rendering work after the first byte arrives.
A crawler that does not render images, fonts or layout at all cares almost entirely about two things: how fast the server starts responding (TTFB) and how much raw text it has to download (HTML weight after compression). Optimizing a hero image's loading behavior helps human visitors; it does nothing for a crawler that never looks at the image in the first place.
How much does HTML weight and compression actually matter?
A smaller response downloads faster regardless of who or what is reading it, and the single cheapest win here is enabling text compression — gzip or the newer, usually more efficient Brotli — on your server or CDN, which commonly cuts HTML, CSS and JavaScript file sizes by 70-90% with no visible change to the page at all.
Beyond compression, a bloated HTML document (deeply nested markup, large inline scripts, excessive comments) adds parsing work on both ends for no benefit a crawler or a user will ever notice — trimming genuinely unused markup helps every request, not just crawler ones.
What should I cache, and where?
- Page-level caching (a full-page cache plugin on WordPress, Shopify's built-in CDN, or a reverse proxy like Varnish) avoids rebuilding the same HTML from scratch on every single request, which is usually the single biggest TTFB win available on a dynamic site.
- A CDN in front of your origin server serves cached pages from a location physically closer to the requester, cutting network latency on top of whatever your own server-side caching already saves.
- Database-query caching and object caching (Redis, Memcached) reduce how much work a dynamic page has to do on a cache miss, which matters most for pages that cannot be fully page-cached (logged-in areas, frequently changing listings).
- Static assets (images, fonts, compiled CSS/JS bundles) should carry long cache-control headers, since they change only on deploy, not per request — this does not move TTFB directly but frees server capacity that would otherwise be spent re-serving unchanged files.
Does my hosting or platform choice matter for TTFB?
Shared hosting with many sites competing for the same server resources is the most common cause of consistently slow TTFB on small business sites, and moving to a platform with dedicated resources or a managed hosting tier built for your CMS (managed WordPress hosting, Shopify's own infrastructure) often fixes it without touching a line of code.
A platform switch is a bigger decision than enabling compression or caching, so treat it as the next step after confirming compression and caching are already on and TTFB is still consistently slow — not the first thing to try.
What should I fix first if my TTFB is slow?
- Run the AI crawlability check and note your actual TTFB number — fixing a problem you cannot measure is guesswork.
- Confirm text compression (gzip or Brotli) is actually enabled at your server or CDN — many platforms support it but ship with it off by default.
- Add or verify full-page caching for anonymous, non-logged-in traffic, since that is the traffic an AI crawler's request looks like.
- If you are not behind any CDN today, that is usually the next highest-leverage change — most CDNs also include compression and basic caching by default.
- If TTFB is still slow after caching and compression are confirmed on, the bottleneck is likely your application or database layer itself, which is a developer-level fix rather than a configuration toggle.
Questions
Is TTFB the same thing as page load time?
No. Page load time includes everything — TTFB, downloading all assets, running scripts, and rendering the final layout. TTFB is only the time until the very first byte of the response arrives, before any of that other work even starts.
Why does Leadee use 600ms as the "fast" threshold instead of the more common 800ms?
Leadee's 600ms pass threshold is a deliberately tighter bar than web.dev's general 800ms "good" guidance, reflecting that a crawler fetching at scale benefits from extra headroom rather than scraping by at the edge of "acceptable."
Will a CDN fix a slow TTFB by itself?
Often most of the way there, especially for a largely static or well-cached site — but if your underlying application is genuinely slow to generate a page on a cache miss, a CDN only hides that cost for cached requests, not for every single one.
Do image file sizes affect an AI crawler at all?
Generally no — AI crawlers reading for citation purposes focus on text content, not images, so image optimization is almost entirely a human-visitor concern rather than a crawler one. It's still worth doing for your actual visitors.
How often should I re-check my site speed?
After any hosting, CDN or caching change, and periodically otherwise — a routine plugin update, a traffic spike, or a change on your hosting provider's end can all quietly push TTFB back up without any change on your own part.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email [email protected] or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.
Measure your actual TTFB today
Run the free check to see your real server response time, and whether compression and caching are already working.