How to Audit Your Site’s LLM Crawlability
LLM crawlability is the audit Google’s tooling does not run for you. Search Console reports on Googlebot. Bing Webmaster reports on Bingbot. Neither reports on GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, or any of the smaller agents that feed retrieval indices for citation engines. A 25,000-page NBFC we audited had 4,431 broken internal links and 71% of crawled pages returning 403 due to a WAF misconfiguration. The classical SEO toolset surfaced none of this in its LLM-specific impact. This piece walks through the eight-checkpoint audit we run on every property entering an AI visibility engagement, with the exact log lines, robots directives, and verification commands to use.
The Eight Checkpoints
The audit runs sequentially. Each checkpoint blocks the next. A failure at Checkpoint 2 means Checkpoint 3 to 8 cannot be trusted, because the data they produce depends on the bots being allowed in the first place.
Checkpoint 1: Identify every LLM crawler your property currently sees. Pull the last 30 days of access logs and group requests by user-agent. The agents to look for are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, FacebookBot, ApplebotExtended, and CCBot. Count requests per agent per day. A property in 2026 should be seeing daily traffic from at least four of these. If the count is zero for any of them, Checkpoint 2 is where the cause sits.
Checkpoint 2: Verify robots.txt and meta robots directives. Fetch /robots.txt and confirm that User-agent declarations either permit or explicitly deny the LLM crawlers above. Many enterprise properties shipped a 2023-era robots.txt that disallowed GPTBot in response to a board memo. The decision may still hold for some brands, but it should be a conscious one. Where the intent is “allow”, confirm there is no User-agent: * Disallow: / pattern that overrides specific bot allowances through processing order.
On the Angular fintech we audited, robots.txt and sitemap.xml were both being served as Angular HTML because a Router intercept rule was catching the requests before the static file handler reached them. The file existed on disk. The crawler never got it. Verifying robots.txt means fetching it from outside the application, with a curl command, with the user-agent string of the crawler you care about, and reading the actual response body.
Checkpoint 3: Validate render parity for retrieval-grade content. Open a representative URL with curl as a generic agent. Note the visible text count and the presence of structural elements (H1, paragraphs, schema). Open the same URL with a headless browser that runs JavaScript. Compare. A delta of more than 70% between pre-JS and post-JS text is a render gap, and most LLM crawlers do not execute JavaScript. The Angular SPA audit surfaced a pre-JS word count of approximately 1 against a post-JS count of approximately 1,200, which is the most extreme form of this failure.
The verification command, in shape:
# Pre-JS view (what most LLM crawlers see) curl -A "GPTBot/1.0" -s https://example.com/category/page-x | wc -w # Post-JS view (what a real browser sees) # Use Playwright or Puppeteer with domcontentloaded plus 4s wait # Compare word counts
Checkpoint 4: Walk the sitemap and resolve each URL. Pull sitemap.xml. Pull every URL it declares. Record the response status. The 25K-page NBFC audit found 3,620 sitemap waste URLs (404, 301, redirect chains, soft-404s). Each waste URL trains the retrieval crawler that your sitemap is unreliable, and an unreliable sitemap is rebudgeted downward over time. The sitemap is the LLM crawler’s first map of the property. Inaccuracy compounds.
Checkpoint 5: Verify canonical consistency and entity resolution. Pull a stratified sample of 50 URLs. For each, check the canonical tag, the og:url, the JSON-LD url field, and the sitemap entry. All four should agree. The 25K-page case had 81% of pages with no canonical at all, which made it impossible for the retrieval layer to resolve which URL represented the entity. Add to this the cross-locale check: hreflang errors at 78% (4,330 broken hreflang links across the property) tell the crawler the locale map cannot be trusted.
Checkpoint 6: Audit schema and structured data presence. Pull the structured-data report for a stratified sample. Confirm one Organization markup on the homepage, type-appropriate schema on every leaf, and dateModified populated and accurate on every Article and Dataset URL. The Angular fintech audit surfaced 0 Open Graph tags across 3,677 stock pages. This is not a schema-only failure. It is a render failure that prevents schema from existing in the page the crawler reads.
Checkpoint 7: Test crawler responses to common WAF and bot management rules. Many enterprise stacks (Akamai, Cloudflare, AppTrana, F5) ship with default rules that block unknown user-agents. The 25K-page NBFC case had 71% of crawled pages returning 403 because GTranslate’s WAF rules were too aggressive. Send a sample of 50 requests using each LLM crawler’s published user-agent and IP range. Record status codes. A status code distribution skewed toward 403 or 429 is a WAF problem, not a content problem.
The remediation for a WAF problem is rarely “allow everything”. It is a documented allow list of verified crawler user-agents and IP ranges, refreshed quarterly as engines publish updated lists. OpenAI publishes GPTBot’s IP ranges. Anthropic publishes ClaudeBot’s. Verifying the user-agent against the published IP range eliminates spoofed-agent attack vectors while preserving real crawler access.
Checkpoint 8: Run a citation panel and correlate with crawl evidence. The final check ties crawl health to commercial outcome. Run a 100-prompt panel across ChatGPT, Claude, AI Overview, AI Mode, and Perplexity. Record citation rate per engine. Cross-reference low citation rate engines against the per-engine crawler evidence in Checkpoints 1 to 7. ChatGPT citation rate at 8% against AI Overview at 15.6% indicated, in the NBFC case, that GPTBot was being treated more aggressively by the property’s WAF than Googlebot-extended. The asymmetry was diagnosable from logs.
The Quick-Look Diagnostic Sheet
What Each Checkpoint Tells You
| Checkpoint | Pass signal | Fail signal |
|---|---|---|
| C1 Bot presence | Daily traffic from 4+ LLM crawlers | Zero requests from any major crawler |
| C2 robots.txt | Explicit allow or deny, no SPA shadow | File served as HTML or contradictory rules |
| C3 Render parity | Pre-JS to post-JS delta under 30% | Pre-JS word count near zero, post-JS rich |
| C4 Sitemap accuracy | Greater than 95% 200-status URLs | Sitemap waste at 10% or more |
| C5 Canonical coherence | Canonical, og:url, JSON-LD, sitemap agree | Greater than 5% of sampled URLs disagree |
| C6 Schema presence | Type-appropriate schema with current dateModified | Missing or invalid schema on core templates |
| C7 WAF posture | Documented allow list, refreshed quarterly | 403 or 429 rates above 5% for verified bots |
| C8 Citation panel | Stable citation rate above category baseline | Wide engine spread or zero citation on key prompts |
What the Audit Surfaces That Other Audits Miss
Three patterns recur. The first is the WAF asymmetry described above. Enterprise stacks ship rules tuned for security teams, not for content teams. The cost shows up only in LLM citation rate, which is rarely the metric a security review is run against. The Akamai case, the AppTrana case in a separate engagement, and the Cloudflare default Bot Management profile have all produced this pattern.
The second is render-gap masquerading as a content problem. The Angular fintech with 0 Open Graph tags across 3,677 pages did not have a markup template problem. The markup existed in the Angular component. It rendered into the DOM on user interaction. It did not exist in the HTML the crawler received. A content team brief that asked for “better metadata” would never have surfaced the underlying SSR gap.
The third is sitemap waste at scale. A property that has been live for more than three years almost always carries sitemap entries for URLs that no longer exist or that 301 to current pages. Removal feels low-impact. The compound effect on crawler trust over six to twelve months is the opposite. Cleaning a sitemap is the cheapest move available, and the most often skipped.
Running the Audit Yourself
The audit is reproducible. A small team can run Checkpoints 1 to 6 with curl, a sitemap parser, a Lighthouse export, and a headless browser script. Checkpoint 7 requires access to logs and to WAF configuration. Checkpoint 8 requires the prompt panel and the engine accounts.
Typical execution time for a 1,000-page property runs to about three working days for a single auditor. A 25,000-page property requires a five-act methodology that took us several weeks across audit, fix specification, and validation. The scale-up is not linear because deeper properties carry more cross-locale and cross-template variance, each of which has to be sampled and verified.
Brands that want the audit run under our methodology can request it through the AI visibility audit service. Render and canonical issues that block downstream fixes are addressed in the technical SEO audit. Vertical context for BFSI and fintech crawl patterns is documented in BFSI growth engineering and fintech growth engineering.
Practitioner Takeaway
- Pull 30 days of access logs and group by user-agent this week. Identify which LLM crawlers you are seeing and which you are not. Zero traffic from a named crawler is the diagnostic, not the data.
- Fetch /robots.txt with curl and confirm the actual response body. If it returns HTML instead of plain text, your routing is misconfigured and Checkpoint 2 fails.
- Run a 10-URL render parity test. Compare pre-JS and post-JS word counts. A delta above 70% is a render gap that blocks LLM extraction.
- Cross-check a 50-URL sample for canonical agreement across canonical, og:url, JSON-LD url, and sitemap. Disagreement on more than 5% of URLs is an entity resolution failure.
- Compute 403 and 429 rates for verified LLM crawler user-agents in your logs. Above 5% is a WAF problem, not a content problem.
Frequently Asked Questions
How often should an LLM crawlability audit be run?
Quarterly for active properties, with a lighter monthly review of Checkpoints 1, 4, and 8. The full eight-checkpoint audit is run at engagement start, after any major site migration, and after any WAF or CDN configuration change. Annual audits miss too much in the LLM space, where engine behaviour shifts on weekly cadences.
Do I need to allow every LLM crawler, or can I be selective?
Selectivity is fine if it is intentional. Some brands deny GPTBot for training but allow OAI-SearchBot for indexing. Others allow all retrieval bots and deny training-specific agents. The discipline is to document the choice, audit the actual response codes against the choice, and revisit the decision quarterly as crawler taxonomy evolves.
What is the relationship between LLM crawlability and classical Googlebot crawlability?
Overlapping but not identical. A site Googlebot crawls cleanly may still fail LLM crawlability if the WAF treats Googlebot-extended or named LLM agents differently. The render-gap check is shared because Googlebot also struggles with heavy CSR, but most modern Google crawls handle JavaScript more thoroughly than most LLM agents.
Will allowing LLM crawlers reduce my classical search traffic?
No causal link has been established in published research or in observed client data. Allowing LLM crawlers exposes content to citation engines without removing it from Google’s index. The substitution effect on click-through rates is a separate question that depends on AI Overview behaviour, not on crawler permissions.
What is the highest-impact fix when the audit fails on multiple checkpoints?
Render parity, almost always. A site that fails Checkpoint 3 cannot recover Checkpoints 5 to 8 regardless of how well-marked-up the source HTML is, because the crawler is not reading the source HTML. SSR or static rendering for the crawler-visible viewport is the first remediation in nearly every multi-checkpoint failure case.
If your access logs show LLM crawlers absent or your citation panel shows engine asymmetry you cannot explain, the audit is the diagnostic. Request the eight-checkpoint run, the per-engine evidence, and the fix-priority specification.