Indexed, Though Blocked by robots.txt: What It Means and How to Fix It

"Indexed, though blocked by robots.txt" is a Google Search Console status for URLs that Google has indexed even though your robots.txt file doesn't allow Googlebot to crawl them. Google knows the URL exists, it just has never been allowed to read the page.

It looks contradictory, but it's working as designed. robots.txt controls crawling, not indexing. If Google finds enough signals pointing at a blocked URL, it can add that URL to its index without ever fetching the page.

Whether you need to do anything depends on one question: do you want this page in Google or not? This guide explains why the status appears, how to find the rule and the links causing it, and the right fix for each answer.

What "Indexed, though blocked by robots.txt" means

You'll find the status in the Page indexing report in Search Console. Because the URLs are indexed, it's shown as a warning about indexed pages rather than in the list of reasons pages aren't indexed.

The status combines two facts:

• Indexed: the URL is in Google's index and can appear in search results.
• Blocked by robots.txt: a Disallow rule matches the URL, so Googlebot doesn't request it.

Since Google can't read the page, it has no title tag, content or meta description to work with. When one of these URLs shows up in search, it usually has a title Google built from anchor text or the URL itself, and the snippet reads "No information is available for this page".

It's easy to confuse this with a similar-sounding status:

Why Google indexes pages it can't crawl

A Disallow rule tells Googlebot not to request a URL. It says nothing about whether the URL can be indexed. Google's own documentation is clear on this: a page blocked by robots.txt can still be indexed if other pages link to it.

Google learns about a blocked URL from signals that don't require crawling it:

• Internal links from crawlable pages on your own site, including navigation, filters and footer links.
• External links from other websites.
• Your XML sitemap, if it lists URLs that robots.txt blocks.
• Redirects, canonicals or hreflang on other pages that point to the blocked URL.
• Earlier crawls, if the page was indexed before you added the Disallow rule.

The more of these signals a URL has, the more likely Google is to treat it as worth indexing. That's why the status tends to show up on pages that are linked site-wide, such as login, cart, search and filter URLs.

Disallow vs Noindex: What's the Difference and When to Use Each

Is it a problem?

Not always. The status isn't a penalty, and it doesn't hurt the rest of your site. What matters is whether the outcome matches what you want for those URLs:

The first case is the urgent one. If important pages carry this status, they're competing in search with no title, no description and no content Google can rank.

How to find the affected URLs and the rule blocking them

1. Export the URLs from Search Console

In Search Console, go to Indexing > Pages, open the "Indexed, though blocked by robots.txt" warning and export the example URLs. Sort them by path and look for patterns. In most audits I run, a handful of directories or URL parameters account for nearly all of them.

2. Check whether they get impressions

In the Performance report, filter by page (use "URLs containing" with the directory or parameter you found). URLs with real impressions are worth fixing first.

3. Find the rule that matches

Run a few example URLs through URL Inspection. Under Crawl, you'll see "Crawl allowed? No: blocked by robots.txt". It won't tell you which rule is responsible, so paste your robots.txt and a URL into my robots.txt tester to see the exact matching line. The robots.txt report in Search Console shows which version of the file Google last fetched.

Rules match by prefix, so a short rule can block far more than intended:

User-agent: *
# Meant to block /p?id= product previews
Disallow: /p

# Also blocks /products/, /pricing/ and /press/

When rules conflict, Google follows the most specific one (the longest matching path), and Allow wins a tie. My robots.txt guide covers matching, wildcards and precedence in detail.

4. Find where the links come from

Note where Google is finding the URLs, because you'll often need to fix that too. Check your XML sitemap for blocked URLs, crawl the site with a tool such as Screaming Frog to find internal links to them, and check your backlink tool for external links.

How to fix “Indexed, though blocked by robots.txt”

Scenario 1: The page should be indexed

If the URL is a real page you want in search, the block is the bug.

1. Edit the Disallow rule. Delete it, or make it more specific so it only covers the URLs you meant to block. Adding an Allow rule for the path you want crawled also works.
2. Test the new file. Check the affected URLs and a few URLs you still want blocked, so the fix doesn't open up too much.
3. Ask Google to refetch robots.txt. Google generally caches robots.txt for up to 24 hours. You can request a recrawl from the robots.txt report in Search Console.
4. Request indexing for the most important URLs with URL Inspection, then click Validate fix in the Page indexing report.

Here's the earlier example rewritten to block only the preview URLs:

User-agent: *
Disallow: /p?id=

Once Google can crawl the pages, they'll move to the normal indexed state with proper titles and snippets.

Scenario 2: The page should not be indexed

This is the case that catches people out. The instinct is to block the URL harder, but robots.txt is the reason Google can't see any instruction to drop it. To get the URL out of the index, Google needs to crawl it first.

1. Remove the Disallow rule covering the URLs.
2. Add noindex with a meta robots tag or an X-Robots-Tag header (examples below).
3. Wait for Google to recrawl. The URLs should move from "Indexed, though blocked by robots.txt" to "Excluded by 'noindex' tag". Spot-check with URL Inspection.
4. Only then decide whether to block crawling again. For most sites, leaving noindex in place is enough. If you re-add Disallow on heavily linked URLs, they can end up indexed from links again.

Add the meta tag to the <head> of HTML pages:

<meta name="robots" content="noindex">

For PDFs and other non-HTML files, or for whole sections at once, send the HTTP header instead. For example, in an nginx config:

location /internal-search/ {
  add_header X-Robots-Tag "noindex" always;
}

Noindex isn't the only option. Depending on why the URL exists, one of these may be the better tool:

• Return a 404 or 410 if the page shouldn't exist at all.
• Put it behind a login if the content is private. Password protection is the only reliable way to keep content away from everyone, not just Google.
• Redirect or canonicalise it if it's a duplicate of another page. Like noindex, Google has to crawl the URL to see either signal.

Either way, stop sending signals to the URL: remove it from your XML sitemap and from internal links where you can.

Scenario 3: The blocked URLs don’t matter

Plenty of sites have thousands of URLs with this status that never cause a problem. Faceted navigation, internal search results, sort parameters and cart or account pages are typical.

If they get no impressions and the block is saving crawl budget on a large site, keeping the Disallow rule is a reasonable choice.

Two clean-ups are still worth doing:

• Remove blocked URLs from your XML sitemap. A sitemap should only list pages you want indexed. Listing blocked URLs sends Google mixed signals.
• Cut unnecessary internal links to them, or reduce how often they're linked. Fewer links means fewer URLs Google decides to index.

Common causes I see in audits

• A sitemap that disagrees with robots.txt. The CMS adds every URL to the sitemap and robots.txt blocks some of them. This is the most common source.
• Blocking pages to remove them. Someone adds Disallow to pages that were already indexed, hoping they'll drop out. They don't, because Google can no longer see that anything changed.
• Noindex and Disallow on the same URL. The noindex never gets read. This is the classic case covered in Disallow vs noindex.
• Staging rules shipped to production. A Disallow: / copied from a staging site blocks everything. Pages that were already indexed show this status while everything else stops being crawled. Check robots.txt straight after every launch or migration.
• Blocked URLs that redirect. If an old URL redirects but is also disallowed, Google can't see the redirect, so the old URL can stay indexed instead of passing its signals to the new one.
• Overly broad prefix or wildcard rules, like the Disallow: /p example above.

How to validate the fix

After you've made the change, open the warning in the Page indexing report and click Validate fix. Google rechecks the affected URLs over the following days and weeks, and the report shows each one as passed or failed. Large sites can take longer, because Google has to recrawl every URL before the count drops.

If the count doesn't fall, check that Google has picked up the new robots.txt (the robots.txt report shows the fetch date), and that nothing else is still blocking or pointing at the URLs.