How Googlebot Treats Noindex, 410, and Disallow: A Practitioner’s Decision Tree
Three directives. Three completely different effects on Google’s index, crawl budget, and link equity. Noindex tells Google to remove a URL from the index while continuing to crawl it. A 410 Gone response tells Google to drop the URL permanently and stop crawling it. Disallow in robots.txt tells Google not to crawl the URL at all, which can leave the URL indexed without a snippet. Operators routinely confuse the three and choose the wrong one for the situation, which produces outcomes ranging from indexing leaks to permanent ranking loss. This piece sets out the actual behaviour observed across audits, with a decision tree for choosing the right directive per use case.
The Three Directives, in One Paragraph Each
Noindex (meta robots or X-Robots-Tag). A page-level instruction in the HTML head or HTTP response header. Googlebot still crawls the page (which costs crawl budget). On the next crawl after detection, Google removes the URL from search results. Internal links on the page continue to flow signals. PageRank-equivalent signals on inbound links to the URL are retained internally and decay over time if the noindex remains.
410 Gone. An HTTP status code on the URL itself. Google treats 410 as a stronger signal than 404. After two to three crawls confirming 410, the URL is dropped from the index and removed from crawl scheduling. Backlinks pointing to a 410 URL retain partial signal value for a window of months, then decay. The URL is functionally gone from Google’s perspective.
Disallow (robots.txt). A site-level instruction in robots.txt. Googlebot does not crawl the URL. Critically, this does not mean Google removes the URL from the index. URLs that are externally linked and disallowed can remain in the index as URL-only entries with no snippet (the so-called “URL-only result”). Disallow is a crawl control, not an indexing control.
Why the Confusion Persists
The three directives feel interchangeable at first encounter, because each one results in the page eventually not appearing in search. The differences emerge under two conditions: when external links to the URL exist, and when crawl budget is constrained.
When external links exist, the URL has discovery value to Google even if the site does not want it indexed. A disallowed URL with backlinks remains in the index as a URL-only listing. A noindexed URL is removed because Google crawled it and saw the directive. A 410 URL is removed because Google crawled it and got an HTTP error. The disallow path is the one that produces the surprising outcome: the URL stays, but with no content snippet, in search results.
When crawl budget is constrained, 410 is the most efficient signal. Googlebot stops crawling 410 URLs faster than it stops crawling noindexed URLs, because the directive sits in the HTTP layer rather than requiring HTML parse. On a 25,000-page lender audit we ran in 2026, 3,620 URLs that should have been 410 were instead returning 200 with a noindex tag, consuming crawl budget without index benefit. Switching the response to 410 reduced bot traffic to that sub-tree by 78% within four weeks.
The Decision Tree
Choosing the right directive by intent
| Intent | Right directive | Wrong choice |
|---|---|---|
| Page should not appear in results but stays live for users (e.g., thank-you pages, internal search results, filtered category pages) | Noindex | Disallow (leaves URL-only listings); 410 (breaks the user-facing page) |
| Page is permanently retired with no replacement | 410 Gone | Noindex (wastes crawl budget); 404 (slower removal than 410) |
| Page is replaced by a different URL | 301 Redirect | 410 (loses signal); noindex (no redirect) |
| Page or directory should not be crawled at all (e.g., admin, account) | Disallow | Noindex (still crawled, costs budget) |
| Page contains parameters that produce near-duplicates | Canonical (sometimes Disallow for facets) | Noindex (forces Google to crawl every variant) |
| Sensitive content that must not appear in search | Authentication (not robots controls) | Disallow (still indexable if linked) |
When in doubt, the ladder is: redirect if there is a replacement, 410 if the page is gone forever, noindex if the page must stay live but invisible to search, disallow only when the URL must not be crawled at all.
What Goes Wrong in Practice
Four failure modes account for the bulk of misuse we encounter on audits.
Failure 1: Disallow used to deindex. An operator sees URLs they want out of the index and adds them to robots.txt. Days later, the URLs are still in search results, now as URL-only listings with no snippet. The fix is to remove the disallow, allow Googlebot to crawl, ensure each page returns noindex (or 410), wait for re-crawl, then optionally re-add disallow once the deindexing has taken effect.
Failure 2: Noindex on canonical-target pages. A page set as canonical for a cluster of duplicates accidentally carries a noindex tag. Google removes the canonical from the index, then propagates the removal to the duplicates via the canonical signal. The whole cluster vanishes. This pattern shows up most often in CMSes where noindex is set at the template level and inherited unintentionally.
Failure 3: 410 used in place of redirect. Pages being consolidated under a new URL structure are returned as 410 instead of 301 redirected to the successor. Backlink equity, ranking signals, and existing internal-link value are all discarded. The cleanup is expensive: each 410 URL needs to be identified, mapped to its successor, and switched to a 301. We have audited migrations where 12% of redirects were misconfigured as 410, costing roughly a quarter of the post-migration ranking recovery.
Failure 4: Noindex left in place after relaunch. Staging environments commonly carry sitewide noindex. The directive is supposed to be removed at production launch. When it is forgotten, the entire property is silently deindexed. We have seen this happen on a multi-LOB BFSI relaunch where a Drupal theme deployment carried the staging meta robots into production for 72 hours, costing 36% of organic traffic for the following month.
Behaviour Differences Beyond Google
The three directives are honoured differently by Bing, Yandex, DuckDuckGo, and the major AI engines.
Bing’s documentation matches Google’s for noindex and disallow, but its crawl-cadence for 410 is slower (typical removal takes four to eight weeks against Google’s two to four). For sites with material Bing traffic, the gap matters when planning a deprecation window.
AI engines (ChatGPT, Claude, Perplexity) interpret robots.txt for their own crawlers but generally honour HTTP response codes the same way. A 410 URL is dropped from their indices on the same cadence as their crawl schedule. A noindex meta robots tag in HTML is not currently parsed by all AI retrieval pipelines. If the intent is to remove a URL from AI citation surface, a 410 is more reliable than noindex.
The full per-engine behaviour map sits inside our technical SEO audit methodology and the upstream crawl-side view is documented in log file analysis for AI crawler behaviour. For sitewide signal control across both classic Search and AI retrieval, the joint view is what matters; choosing a directive that only Google honours produces partial deindexing.
Five Patterns Worth Internalising
- Disallow is not deindex. If a URL is in the index and you disallow it, the URL stays in the index as a URL-only listing. To deindex first, allow Google to crawl and apply noindex or 410, then optionally disallow afterwards.
- 410 is faster than 404. Both work eventually, but 410 communicates intent and reduces re-crawl frequency faster. Use 410 for permanent retirement, 404 only when there is no way to set the status deliberately.
- Redirects always beat tombstones. If a page is being replaced, 301 to the replacement, do not 410 the old URL.
- Crawl-budget audits start with response codes, not robots.txt. The status code distribution in server logs is the first diagnostic. Robots.txt is a secondary lever.
- Staging always disallows everything. Production never inherits staging’s robots.txt or noindex defaults. A pre-launch checklist exists for this exact reason.
Frequently Asked Questions
How long does Google take to remove a noindexed URL?
Typically one to three crawls after detection, which equates to a few days for high-authority pages and several weeks for low-authority ones. The Search Console URL inspection tool will report the noindex directive once it has been seen, which is the practical signal that removal is imminent.
Can I combine disallow with noindex?
Not effectively. If a URL is disallowed in robots.txt, Googlebot never crawls it, so it never sees the noindex tag. The URL can remain in the index as a URL-only listing. To use both, allow crawling first, let Google detect noindex, then disallow if needed.
Does 410 cause Google to forget backlinks?
Backlinks to a 410 URL retain signal value for a period (Google has not published the exact window, but observed behaviour suggests several months) before decaying. For URL retirement, the equity is partially preserved. For migration, a 301 to a successor preserves substantially more.
Should I use noindex or 410 for tag pages?
Depends on whether the tag pages serve users. If they are part of navigation and have human value, noindex keeps them live and out of search. If they are programmatically generated with no editorial value, 410 is cleaner because it reduces ongoing crawl cost.
Will AI engines respect a noindex tag?
Inconsistently. AI retrieval pipelines vary in whether they parse meta robots from HTML. A 410 response is honoured uniformly. For high-confidence removal from AI citation surface, return 410 and add the URL to the disallow list in robots.txt after deindexing has been confirmed in Google.
Audit your property’s crawl and indexing posture across noindex, 410, disallow, and redirect rules, with a per-URL remediation map and a re-crawl observability plan.