How Page Speed Affects Citation Rates, and which AI Crawlers Time Out the Fastest

Everyone knows page speed matters for SEO. But most teams assume AI crawlers behave like Googlebot: patient, persistent, willing to wait. They don’t. We analyzed 50 days of server logs from 30 sites and found that AI retrieval bots abandon slow pages 2.3 times faster than Googlebot does. Perplexity-User starts timing out at 1,200 milliseconds. By 1,800 milliseconds, nearly one in ten requests fails. Across 200 tracked prompts, not a single page that logged a timeout error ever appeared in an AI citation. If your pages are slow, they’re not just slow. They’re invisible.

TL;DR

• AI retrieval bots time out 2.3x faster than Googlebot. Median timeout TTFB: 1,850ms vs. 4,200ms.
• Perplexity-User is the most aggressive. 499 errors begin at roughly 1,200ms.
• Above 1,800ms TTFB, 9–12% of AI bot requests fail with a 499.
• Pages under 800ms were cited at a 34% rate. Pages above 1,800ms: 6%.
• Zero pages that logged a 499 error ever appeared in a citation.
• If your content relies on client-side JavaScript, some AI crawlers can’t read it at all. That’s not a speed problem. It’s a visibility problem.

Methodology

We analyzed two datasets. The first is the stronger one.

Server logs. Thirty sites shared 50 days of log data. We filtered for four AI bots: ChatGPT-User, Perplexity-User, Claude-Web, and OAI-SearchBot. Googlebot was included as a reference. Across roughly 10,000 requests, we recorded TTFB, HTTP status, page URL, and bot type.

Citation tracking. From those requests, we identified roughly 300 pages crawled five or more times by retrieval bots. We tracked 200 prompts relevant to those pages across ChatGPT, Perplexity, and Claude, re-querying daily. For each page, we calculated a citation rate segmented by TTFB bucket.

A note on limitations. The server log data is observational. We didn’t control TTFB experimentally. The citation sample is small and we couldn’t fully isolate speed from content quality or domain authority. Treat the citation numbers as directional, not precise. The timeout data is the stronger finding.

Finding 1: AI Crawlers Time Out 2.3x Faster Than Googlebot

Across all 30 sites, the median TTFB of requests that returned a 499, the HTTP status code for “client closed request” meaning the crawler gave up waiting, was 1,850ms for AI bots. For Googlebot, it was 4,200ms.

That gap isn’t small. It’s the difference between a page that loads in under two seconds and one that takes over four. AI crawlers are simply less patient.

The reason is architectural. Googlebot has been optimized over two decades to crawl the web at scale. It queues requests, retries, and comes back later if a page is slow today. AI retrieval bots don’t work that way. They’re fetching pages in real time because a user is waiting for an answer. ChatGPT-User doesn’t have a queue. It has a person staring at a blinking cursor. If your page doesn’t respond quickly enough, the bot moves on and the answer gets built without your content.

“Page speed matters uniquely for AI search. Systems like ChatGPT and Perplexity request pages in real time, and slow pages get skipped.”

Tom Capper reached the same conclusion in his analysis of 515 million LLM bot traffic events at Moz. Our data puts specific numbers on the gap he identified. The practical implication is straightforward: a TTFB that’s fine for Google may already be too slow for AI retrieval. The bar is higher.

Median TTFB of 499'd requests, by bot

Finding 2: Perplexity-User Is the Most Aggressive

So AI crawlers give up faster than Googlebot. But are they all equally impatient?

Not even close. Perplexity-User starts logging 499 errors at roughly 1,200ms TTFB. ChatGPT-User and Claude-Web begin seeing them at roughly 1,600ms and 1,700ms respectively. OAI-SearchBot, which does background indexing rather than real-time retrieval, is the most patient at roughly 2,200ms.

Architecture explains the difference. Perplexity uses a streaming UX. The user watches the answer build in real time, sentence by sentence. The retrieval has to be fast because the entire experience is synchronous. There’s no buffering, no “loading” state to hide behind. ChatGPT and Claude have more buffered response models. They can tolerate slightly longer fetches without the user noticing. OAI-SearchBot, as OpenAI documents, is a background indexing bot, not responding to a user at all. It’s building an index, and it can afford to wait. Perplexity draws the same line between its user-triggered bot and its indexing bot.

What this means for your strategy: if Perplexity is an important traffic source for your vertical, your speed bar is higher than if you’re primarily optimizing for ChatGPT. You can’t treat “AI search” as one thing with one threshold.

499 error rate by TTFB in milliseconds

Finding 3: The 499 Curve Is Nonlinear

This isn’t a gradual slope. Below 1,200ms TTFB, 499 errors are essentially zero, between 0.1% and 0.4% of requests. Between 1,200ms and 1,800ms, they climb steadily to about 5%. Above 1,800ms, they spike to 9–12%.

The difference between a page at 1,700ms and a page at 1,900ms isn’t incremental. It’s the difference between “usually works” and “fails roughly one in ten times.” Most performance work focuses on shaving milliseconds: going from 600ms to 400ms, or 1,200ms to 1,000ms. For AI crawlers, the real ROI is in staying below 1,800ms entirely.

The 1,800ms line is worth noting. It aligns closely with Google’s own “poor” TTFB threshold for human users. Under 800ms is “good,” anything above 1,800ms is “poor.” That’s not a coincidence. It’s a natural timeout boundary for HTTP clients that need real-time responses. If your pages are above 1,800ms, you’re in the danger zone for both human users and AI crawlers, but the crawlers will drop you faster.

499 error rate by TTFB bucket, by bot

Finding 4: Citation Rate Drops with TTFB

The server logs show the mechanism: crawlers timing out on slow pages. But does that actually translate to fewer citations?

We looked at the roughly 300 pages in our sample that were crawled five or more times by retrieval bots. For each one, we measured how often it appeared in AI responses for 200 tracked prompts, then grouped the results by TTFB bucket.

Pages with TTFB under 800ms were cited 34% of the time. At 800–1,200ms, the rate dropped to 28%. At 1,200–1,600ms: 21%. At 1,600–1,800ms: 14%. Above 1,800ms: 6%.

The relationship is clear. Faster pages get cited more. But we want to be upfront about the limitations here. This is a small sample. We couldn’t fully control for content quality, domain authority, or how well each page matched the tracked prompts. The specific percentages have wide confidence intervals. Don’t quote the 34% as a benchmark. Quote the direction.

Why include it at all? Because the two datasets reinforce each other. The server logs show the mechanism. The citation data shows the consequence. Together, they’re more compelling than either alone.

Citation rate by TTFB in milliseconds

Finding 5: A Single 499 Means You’re Invisible

A page times out. The crawler moves on. Does it matter?

This was the starkest finding in the entire study. Across all 200 tracked prompts and all three engines, not a single page that logged a 499 error from an AI retrieval bot subsequently appeared in a citation for that engine. Zero.

If the crawler can’t fetch your page, your content doesn’t exist for that query. There’s no partial credit. There’s no “the engine remembers you from last time.” The retrieval is real-time and the timeout is absolute.

If you see 499 errors in your server logs from ChatGPT-User, Perplexity-User, or Claude-Web, those aren’t warnings. They’re confirmations that your content was excluded from answers your site should have been part of.

Some AI Crawlers Can’t Read JavaScript at All

Our server log analysis focused on TTFB, how long the server takes to respond with the first byte. But there’s a second problem that may be worse: some crawlers never see your content in the first place.

Existing research has already confirmed that some AI crawlers don’t execute JavaScript at all. The Peec AI Deep Research teardown showed that Deep Research’s text browser renders zero JavaScript. Content loaded client-side is simply absent from the page text the agent reads. And Semrush’s GEO research found that AI crawlers broadly struggle with JavaScript, noting that client-side rendered content may be invisible to AI systems. Other AI crawlers vary. Some execute JS. Some don’t. The safe assumption is that any content requiring JavaScript to appear may be invisible to a meaningful portion of AI retrieval traffic.

This compounds the TTFB problem. Even for crawlers that do execute JavaScript, client-side rendering adds latency. A page with a 400ms HTML TTFB might have an effective TTFB of 1,800ms or more once JS execution is factored in, pushing it past the cliff for Perplexity-User and into problematic territory for ChatGPT and Claude.

The fix: server-side rendering or static generation for critical content. At minimum, make sure your core content, headings, body text, structured data, is present in the initial HTML response, not loaded asynchronously. If you’re not sure whether your content survives without JavaScript, test it. Open your page in a text-mode browser or run curl against it. What you see is what some AI crawlers see.

What This Does and Doesn’t Prove

What it does prove:

AI retrieval bots have significantly lower timeout thresholds than Googlebot. This is measurable in server logs and consistent across all 30 sites in our sample. Perplexity-User is the most latency-sensitive of the major AI crawlers, and the architecture explains why. A 499 error from an AI retrieval bot is a reliable signal that your content was excluded from an answer. We found zero false positives in our citation cross-reference.

What it doesn’t prove:

The citation curve is correlational, not causal. Faster pages get cited more, but speed isn’t the only variable. Content quality, authority, and how well a page matches the prompt all matter. The specific TTFB thresholds, 1,200ms and 1,800ms, are observed medians from a 30-site sample. Your infrastructure, your CDN, and the bots hitting your site may behave differently. We didn’t study Google’s AI features. AI Overviews and AI Mode use index-based retrieval rather than real-time page fetches, so the timeout mechanism doesn’t apply the same way. And bot behavior changes. Crawler timeout thresholds aren’t documented APIs. They’re observed behavior that can shift with infrastructure updates.

What to Do About It

If you do nothing else:

1. Check your server logs for 499 errors from AI bots. Filter for ChatGPT-User, Perplexity-User, Claude-Web, and OAI-SearchBot. Any 499 is a page that was excluded from an AI answer. Fix those pages first.
2. Get your TTFB under 800ms. This is Google’s “good” threshold, and our data shows no citation penalty below it. It’s the safest target across all engines.
3. If Perplexity matters to your vertical, get under 1,200ms. Perplexity-User starts timing out earlier than the others. The bar is higher, and ignoring it means leaving citations on the table.

When you have more time:

4. Audit your most-cited pages for TTFB. If your highest-value content sits in the 1,200–1,800ms range, you’re losing citations to faster competitors with similar content quality. These are your highest-ROI fixes.
5. Check whether your content survives without JavaScript. Use a text-mode browser or curl to see what AI crawlers that skip JS actually receive. If your core content isn’t in the initial HTML, it doesn’t exist for those crawlers.
6. Monitor bot-specific timeout patterns, not just aggregate speed. A page that’s fast enough for ChatGPT might still be too slow for Perplexity. Segment your monitoring by user agent.
7. Set up ongoing 499 alerting. A sudden spike in 499s from AI bots is an early warning that your infrastructure is slipping past the timeout cliff, and your AI visibility is dropping with it.