Disallow and noindex both seem to keep pages out of search, but they do different jobs. Disallow is a robots.txt rule that tells crawlers not to request a URL. Noindex is a page-level directive that tells search engines not to show a URL in their results. One controls crawling, the other controls indexing.
The difference matters most when people use them together. If a URL is disallowed in robots.txt, Google can't fetch it, so it never sees the noindex tag on it. The page you tried to hide can stay in search results for months.
This guide compares the two, shows which one to use in common situations, and explains how to remove pages from Google without getting stuck. If you're new to robots.txt, start with my guide on what a robots.txt file is and how it works.
Disallow vs noindex at a glance
| Disallow | Noindex | |
|---|---|---|
| What it controls | Crawling: whether a bot may request the URL | Indexing: whether the URL may appear in search results |
| Where it goes | robots.txt file at the root of the host | Meta robots tag in the HTML <head>, or an X-Robots-Tag HTTP header |
| Can the URL still appear in Google? | Yes, if other pages link to it (usually with no description) | No, once Google recrawls the page and sees the directive |
| Does Google see the page content? | No | Yes, it has to crawl the page to read the directive |
| Are links on the page discovered? | No | Yes at first, but long-term noindexed pages are crawled less |
| Works on PDFs, images and other files? | Yes, by path or file extension | Only through the X-Robots-Tag header |
| Saves crawl budget? | Yes | No, the page still has to be crawled |
| How fast it takes effect | When Google refreshes robots.txt, usually within 24 hours | When Google next crawls each URL, from days to months |
How Disallow works
Disallow is a rule inside a robots.txt group. It tells a crawler not to request any URL whose path starts with the value you give it:
User-agent: *
Disallow: /internal-search/Googlebot obeys this and never downloads those pages. But Disallow says nothing about indexing. If Google finds links to a blocked URL, from your own pages, other sites or your sitemap, it can still index the URL based on those links alone. Because Google can't read the page, the search result usually has no description and just the URL like this:
That's why the Page indexing report in Search Console has two separate statuses. "Blocked by robots.txt" means Google respected the block and didn't index the URL. "Indexed, though blocked by robots.txt" means it indexed the URL anyway from external signals.
For syntax, precedence and wildcards, see my robots.txt guide.
How noindex works
Noindex tells search engines they may crawl a page but should not show it in results. There are two ways to send it.
Meta robots tag
Add the tag to the <head> of an HTML page:
<meta name="robots" content="noindex">To target only Google, change name="robots" to name="googlebot". You can combine values in one tag, for example content="noindex, nofollow".
X-Robots-Tag HTTP header
You can also send noindex as an HTTP response header. This is the only option for non-HTML files such as PDFs, images and videos, and it's useful for applying noindex to many URLs through server configuration.
X-Robots-Tag: noindexFor example, this Apache rule adds the header to every PDF on the site:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>Noindex in robots.txt doesn't work
Google used to unofficially honour a Noindex: line in robots.txt. It stopped supporting it on 1 September 2019, along with other unofficial rules such as nofollow and crawl-delay. If your robots.txt still has Noindex: lines, they do nothing for Google.
Whichever method you use, Google has to crawl the URL to see the noindex. That one requirement is what makes combining it with Disallow a problem.
Why you shouldn't use Disallow and noindex together
It feels like extra protection: noindex the page, then block it in robots.txt so crawlers stay away completely. In practice, the two cancel each other out. Here's what happens:
- You add noindex to a page that is already indexed.
- You also disallow the URL in robots.txt.
- Googlebot checks robots.txt, sees the block and doesn't request the page.
- Because it never downloads the page, Google never sees the noindex.
- The URL stays in the index, based on what Google already knew and on the links pointing to it.
In Search Console, this usually shows up as "Indexed, though blocked by robots.txt". If you see that status on pages you meant to remove, take the Disallow rule off, let Google recrawl the pages and process the noindex, and only then decide whether you still need the block.
Pro Tip
Give each URL one job. If you want a page out of the index, it must stay crawlable. If you want to stop crawling, accept that the URL may still be indexed from links.
Which one to use: a scenario guide
Start with one question: do you want to stop Google showing the page, or stop Google spending time on it? The first is a noindex job, the second is a Disallow job. And for several common situations, neither is the right tool.
| Situation | Use | Why |
|---|---|---|
| Internal site search results | Disallow | Endless URL combinations with no search value. Blocking saves crawl budget. If they're already indexed, noindex them first. |
| Faceted navigation and filter parameters | Disallow, or canonical for facets worth consolidating | Stops crawlers wasting time on near-infinite combinations. |
| Thin tag, author or date archives | Noindex | Keeps them out of results while Google can still crawl them. |
| Thank-you, cart and account pages | Noindex | Users need them, but they shouldn't be search landing pages. |
| PDFs and other files | X-Robots-Tag: noindex | You can't add a meta tag to a non-HTML file. |
| Duplicate or near-duplicate pages | rel="canonical" | Consolidates signals to the preferred URL instead of throwing them away. |
| Pages you've deleted for good | 404 or 410 | A removed page should return an error, not stay live with noindex. |
| Pages that have moved | 301 redirect | Sends users and signals to the new URL. |
| Staging sites and private content | Password protection | Neither directive stops people, and Disallow doesn't stop indexing. |
| Indexed pages you want removed and never crawled again | Noindex first, then Disallow | Covered step by step in the next section. |
How to remove pages from Google, then stop crawling them
When a section of a site needs to go from indexed to both unindexed and uncrawled, order matters:
- Make sure the URLs are crawlable. Remove any Disallow rule that covers them.
- Add noindex with a meta robots tag or an X-Robots-Tag header.
- Wait for Google to recrawl. Spot-check URLs with URL Inspection, or watch the "Excluded by 'noindex' tag" count grow in the Page indexing report. Listing the URLs in a temporary XML sitemap can help Google find them sooner; remove it once they've dropped out (see my sitemap best practices).
- Add the Disallow rule, if crawl budget matters. For most small sites it doesn't, and leaving noindex in place is enough.
One caveat on step 4: once the URLs are blocked, Google can index them again as URL-only results if plenty of links still point to them. For pages with lots of internal or external links, keeping noindex without Disallow is the safer choice.
If you need a page hidden urgently, use the Removals tool in Search Console. It hides URLs from Google Search for about six months but doesn't remove them from the index, so you still need noindex, a 404 or 410, or password protection behind it.
Nuances worth knowing
Long-term noindex tends to become nofollow
"noindex, follow" is a common setting for archive pages, meant to keep them out of search while still passing link signals. Google's John Mueller has explained that Google crawls long-term noindexed pages less often and may eventually treat them like noindex, nofollow. Don't let a noindexed page be the only route to important content. Link to that content from indexable pages too.
Disallowed pages are a dead end for links
Google can't see the links on a page it isn't allowed to crawl. Links pointing to a disallowed URL still count towards it, but nothing flows onward. Blocking a heavily linked section can cut off the pages it links to.
Blocking resources can break rendering
Disallowing JavaScript, CSS or API endpoints that a page needs can leave Google looking at an empty or broken page. Check the rendered HTML in URL Inspection before blocking any resource path.
What this means for AI search
Google's AI Overviews and AI Mode draw on Google's index, so a noindexed page won't appear there either. To keep an indexed page's content out of AI features and snippets while staying in search, use nosnippet, data-nosnippet or max-snippet instead.
Other AI crawlers are a separate decision. Disallowing GPTBot or ClaudeBot controls whether those bots fetch your pages and has no effect on Google indexing. Noindex is aimed at search indexes and isn't a reliable way to opt out of AI training. See how to block AI crawlers with robots.txt for the user-agent tokens.
Other search engines
Bing supports Disallow and both forms of noindex, and it works much the same way: it needs to crawl a page to see noindex, and a blocked URL can still be indexed from links. The main difference is that Bing respects Crawl-delay, which Google ignores.
How to check which directive is applied
- Page indexing report. The statuses "Blocked by robots.txt", "Excluded by 'noindex' tag" and "Indexed, though blocked by robots.txt" tell you which signal Google acted on.
- URL Inspection. Shows whether crawling is allowed, whether indexing is allowed, and the rendered HTML Google saw.
- robots.txt tester. Paste your rules and a URL to see whether it's blocked, and by which rule.
- Response headers. X-Robots-Tag doesn't appear in the page source, so check the headers in your browser's dev tools (Network tab) or with curl:
curl -I https://www.example.com/whitepaper.pdfFAQs about Disallow vs noindex
Can I put noindex in robots.txt?
No. Google stopped supporting Noindex: rules in robots.txt in September 2019. Use a meta robots tag or an X-Robots-Tag HTTP header instead.
Should I disallow pages that have a noindex tag?
Not while you want them removed from the index. Google has to crawl a page to see its noindex, so blocking it in robots.txt stops the noindex from working. Once the pages have dropped out, you can add a Disallow rule if you need to save crawl budget.
What's the difference between noindex and nofollow?
Noindex is about the page itself: it tells search engines not to show that page in results. Nofollow is about links. In a meta robots tag, nofollow tells search engines not to follow any links on the page. As rel="nofollow" on a single link, it applies only to that link. Nofollow doesn't stop the page from being indexed, and noindex doesn't stop links on the page from being followed.
Does nofollow stop a page from being crawled or indexed?
No. Since March 2020, Google treats nofollow as a hint for crawling and indexing rather than a strict rule. A URL linked with rel="nofollow" can still be crawled and indexed, especially if it's linked from anywhere else. To keep a page out of search, use noindex on the page itself.
Should I use "noindex, follow" or "noindex, nofollow"?
Usually just noindex. Follow is the default, so "noindex, follow" behaves the same as noindex alone. Add nofollow only if you specifically don't want search engines following the links on that page, which is rare.
How long does noindex take to work?
It depends on how often Google crawls the page. Frequently crawled pages can drop out within days, while deep pages may take weeks or months. For individual URLs, you can speed things up with Request Indexing in URL Inspection.
Which is better for crawl budget, Disallow or noindex?
Disallow. Noindexed pages still have to be crawled for Google to see the directive, so noindex doesn't save crawl budget. Crawl budget is mainly a concern for large sites with many thousands of URLs.
