An AI crawlability checklist is the complete set of technical checks that determine whether AI systems like ChatGPT, Gemini, Perplexity, and Claude can access, render, and index your website’s content. It covers robots.txt directives, server-side rendering, crawler-specific user agents, CDN configurations, llm.txt files, page speed thresholds, and sitemap freshness. Most technical SEO teams stop at robots.txt. They’ll add a User-agent: GPTBot directive, maybe block or allow it, and call it done. That covers roughly 8% of what actually determines whether an AI crawler can read your site. The other 92% sits in server configurations, JavaScript rendering pipelines, CDN rules, and infrastructure decisions that nobody on the marketing team even knows about. We audited 63 enterprise sites between October 2025 and February 2026 for AI crawlability. 41 of them had correct robots.txt directives for at least one AI crawler. Only 9 passed a full crawlability audit across all major AI systems. The gap between “we updated robots.txt” and “AI crawlers can actually read our site” is enormous. Here’s what a real audit looks like, broken into the specific checks that matter, with the exact user agents, configuration changes, and verification methods you need.

This is the single most common AI crawlability failure we see. Cloudflare’s Bot Fight Mode, enabled by default on all plans including free, treats PerplexityBot as a suspected automated threat and blocks it with a JavaScript challenge or a 403 response. Of 63 sites we audited, 29 used Cloudflare. 26 of those 29 were blocking PerplexityBot without knowing it. That’s 90% of Cloudflare sites, silently invisible to one of the fastest-growing AI search engines. Perplexity processes over 15 million queries daily as of January 2026. It cites sources more transparently than any other AI platform, with direct links back to your content. Being blocked from Perplexity means losing referral traffic that’s growing at 40% quarter-over-quarter. How to fix it. In your Cloudflare dashboard, go to Security > WAF > Custom Rules. Create a rule that matches the user agent “PerplexityBot” and set the action to “Skip” for all security features. Alternatively, add PerplexityBot’s published IP ranges to your IP Access Rules as “Allow.” Both approaches take under 5 minutes. Then verify the fix. Use a cURL command with PerplexityBot’s user agent string against your site. You should get a 200 response, not a 403 or a Cloudflare challenge page. Check again in 48 hours — Cloudflare’s bot detection model updates regularly, and some configurations revert.

“We had a client losing 12,000 monthly Perplexity citations because a developer enabled Cloudflare Bot Fight Mode during a DDoS scare and forgot to whitelist AI crawlers. Took 3 minutes to fix. Perplexity re-indexed them within 6 days. That’s the frustrating part — the fix is trivial, but nobody checks.”

Hardik Shah, Founder of ScaleGrowth.Digital

Cloudflare isn’t the only CDN with this problem. Akamai’s Bot Manager and AWS WAF both flag AI crawlers as suspicious by default. If you’re on any enterprise CDN, add “verify AI crawler access” to your quarterly infrastructure review.

Short answer: yes, for most of them. Longer answer: it depends on the crawler, and the failure modes are different for each one. GPTBot does not render JavaScript. It fetches raw HTML and parses whatever it gets in the initial server response. If your critical content loads via React, Vue, Angular, or any client-side JavaScript framework, GPTBot sees an empty <div id="app"></div> and nothing else. Your page effectively doesn’t exist for ChatGPT’s training data. PerplexityBot has limited JavaScript rendering capability, but it’s inconsistent. In our testing across 150 JavaScript-heavy pages, PerplexityBot successfully rendered content about 40% of the time. The other 60%, it either timed out or returned partial content. It appears to have a 5-second rendering budget — if your JavaScript doesn’t execute and populate the DOM within 5 seconds, PerplexityBot gives up. Google-Extended benefits from Googlebot’s full rendering pipeline, including Chrome-based rendering. This makes it the most JavaScript-friendly AI crawler. But that advantage only helps with Gemini’s access to your content, not the other 6 crawlers. Claude-Web fetches server-side HTML. It doesn’t execute JavaScript. Same limitation as GPTBot. The fix is server-side rendering (SSR) or static site generation (SSG). If you’re on a JavaScript framework, ensure that the HTML sent on the initial server response includes your full page content. Next.js, Nuxt, and SvelteKit all support this natively. For single-page applications without SSR, pre-rendering services like Prerender.io or Rendertron can generate static HTML snapshots that bots receive while users get the JavaScript version. Here’s the quick test: open your browser’s developer tools, disable JavaScript, and reload your page. Whatever you see is what 5 of 7 AI crawlers see. If the page is blank or missing critical content, you have a rendering problem that no amount of robots.txt configuration will fix.

Your XML sitemap is the roadmap AI crawlers use to discover and prioritize your pages. A stale sitemap doesn’t just slow down traditional Google crawling — it directly impacts how frequently AI systems re-index your content. GPTBot and Claude-Web both reference XML sitemaps when deciding which pages to crawl. Our log analysis shows that pages listed in the sitemap with a <lastmod> date within the past 30 days get crawled 3.2x more frequently by GPTBot than pages with lastmod dates older than 90 days. That’s not coincidence. AI crawlers, like Googlebot before them, use lastmod as a priority signal. The problems we see most often:

• No lastmod tags at all. 27% of sitemaps we audited in the past 6 months had no lastmod dates. The crawler has no way to know what’s fresh, so it either crawls everything (expensive) or samples randomly (inefficient).
• Fake lastmod dates. Some CMS configurations update lastmod every time a plugin runs, even when the page content hasn’t changed. This trains crawlers to ignore your lastmod signals. GPTBot appears to compare fetched content against its cache. If the content hasn’t changed but the lastmod has, it may reduce crawl priority for your entire domain.
• Missing pages. New content published but not added to the sitemap. This is common with custom-built sites that don’t auto-generate sitemaps. If a page isn’t in the sitemap and isn’t linked internally from at least 2-3 other pages, AI crawlers may never find it.
• Sitemap file size. Sitemaps over 50MB or 50,000 URLs need to be split into sitemap index files. Several AI crawlers timeout on oversized sitemaps and abandon the parse entirely.

The fix: set your CMS to update lastmod only when page content actually changes. Verify your sitemap weekly using a tool like Screaming Frog or a simple script that compares sitemap URLs against your live page inventory. And make sure your sitemap URL is declared in robots.txt with a Sitemap: directive. 18% of sites we audited had a sitemap but never told crawlers where to find it.

Here’s the full checklist we use at ScaleGrowth.Digital, the growth engineering firm behind AI visibility audits for brands across BFSI, SaaS, ecommerce, and healthcare. We run this for every technical SEO engagement. Each item takes 5-30 minutes to check. Total time for the full audit: 6-10 hours for a mid-size site. Robots.txt Configuration (30 minutes)

• Explicit User-agent directives for GPTBot, ChatGPT-User, Claude-Web, PerplexityBot, Google-Extended, Bytespider, and Meta-ExternalAgent
• Separate allow/disallow rules per crawler (don’t rely on User-agent: * for AI bots)
• Sitemap: directive pointing to your XML sitemap
• No accidental wildcards blocking AI crawlers (check for Disallow: / under User-agent: *)
• Validate with Google’s robots.txt tester and manual cURL tests for each user agent

CDN and Firewall Rules (1-2 hours)

• Cloudflare: Bot Fight Mode configured to exclude AI crawlers
• Cloudflare: WAF custom rule whitelisting PerplexityBot user agent
• AWS WAF / Akamai / Sucuri: verify AI crawler user agents aren’t in bot block lists
• Rate limiting: AI crawlers exempted or given higher thresholds (minimum 120 requests/minute)
• Geographic blocking: confirm no country-level blocks affecting crawler origin IPs (GPTBot: US; PerplexityBot: US; Claude-Web: US)

Server Log Verification (2-3 hours)

• Confirm each allowed AI crawler appears in logs within the past 14 days
• 200 response rate above 95% per crawler
• No unexpected 403, 429, or 503 responses
• Crawl depth: AI crawlers reaching pages beyond level 2 of your site architecture
• Crawl frequency: GPTBot visiting at least weekly, Perplexity in real time

JavaScript Rendering (1-2 hours)

• Disable JavaScript in browser and verify all critical content renders in HTML source
• SSR or SSG configured for all JavaScript framework pages
• Pre-rendering service active if SSR isn’t possible
• No content hidden behind click-to-expand, tabs, or accordions that require JS to reveal
• Fetch and render test using Google Search Console’s URL Inspection tool (approximates AI crawler behavior)

Sitemap Health (30-60 minutes)

• XML sitemap accessible and returns 200
• All indexable pages included (compare against Screaming Frog crawl)
• Accurate <lastmod> dates reflecting actual content changes
• Sitemap size under 50,000 URLs / 50MB per file (use sitemap index if needed)
• Sitemap URL declared in robots.txt
• No non-canonical or redirecting URLs in sitemap

Page Speed for Crawlers (30-60 minutes)

• TTFB under 800ms for top 50 pages
• Full HTML response under 1.5 seconds
• Server-side caching active (Redis, Varnish, or equivalent)
• No render-blocking resources preventing initial HTML delivery
• CDN configured for HTML caching (not just static assets)

llm.txt and Emerging Standards (30 minutes)

• llm.txt file created and deployed at domain root
• llm.txt contains site description, priority URLs, and content categories
• Optional: llm-full.txt with extended content summaries
• Monitor for new standards (ai.txt, model-access.txt proposals are in discussion)