Are you encountering a bunch of “Crawled – not currently indexed” statuses with RSS feed URLs in your Google Search Console? Don’t panic. These are not errors but rather RSS feeds that facilitate subscription through various news aggregator software.
In most cases, you don’t need to do anything.
However, if you wish to manage them to optimise your crawl budget or for other reasons, here’s what you need to know and how you can take action.
What are RSS Feed URLs?
RSS feed URLs are simply URLs that serve as RSS feeds, providing information about new posts on your website. They are not problematic per se but can consume your crawl budget unnecessarily.
An example of what an RSS Feed URL look like:
https://aubreyyung.com/seo/feed/If you are using WordPress as your CMS, you will likely see these individual RSS feeds as WordPress automatically creates them.
How RSS Feed URLs Impact Indexing Status
RSS feed URLs provide updates about new posts on your website, but it is not necessary for search engine indexing.
When Google’s crawlers spend resources on crawling these URLs without indexing them, it can impact your website’s crawl budget.
This allocation of resources to non-essential URLs might affect the indexing of more important pages on your site, potentially impacting your site’s overall visibility in search results.
If you notice there are too many URLs not being indexed because of “Crawled – not currently indexed”, consider disable them.
How to remove /feed/ URLs from Search Console?
If you notice RSS feed URLs in your Google Search Console coverage report, you can’t remove those. Simply because Google already crawled and found those pages, so Google will notify you about the status of them.
However, most of the time these URLs are listed as “Crawled – not currently indexed”. As these pages don’t need to be indexed, this is a normal behavior and you don’t need to worry about them.
How to Manage RSS Feed URLs
1. Editing the robots.txt File
You can directly manage RSS feed URLs by accessing your website’s root folder and locating the robots.txt file. By adding a Disallow directive for RSS feed URLs, you can instruct search engine crawlers not to crawl these pages.
Here’s how you can do it in the robots.txt file:
User-agent: *
Disallow: /*/feed/Simply paste this directive into your robots.txt file to prevent crawlers from accessing RSS feed URLs.
2. Using a WordPress Plugin
If you are using WordPress as your CMS, managing RSS feed URLs is even easier. There are different plugins that help you disable your RSS feed or edit them.
For example, you can install “Disable Everything” to disable the RSS feed option. Or alternatively, many popular SEO plugins (such as Yoast, RankMath) offer similar control over the crawling.
Refer to the documentation or support resources provided by the plugin developer for detailed instructions on how to disable the RSS feed setting.
3. Setting a Noindex Tag for Your RSS Feed
Another method to manage RSS feed URLs is by setting a noindex tag specifically for your feed pages. This approach instructs search engine crawlers not to index these pages, thereby preventing them from appearing in search results.
However, most of the time, Google won’t index your feed page. So this is just a safeguard measure in case they are indexed.
<meta name="robots" content="noindex, follow">DO NOT combine noindex and disallow:
Using disallow will stop Google from crawling your feed page and thus they won’t see the noindex tag.
Is an RSS feed considered duplicate content?
Feed URLs themselves are not typically considered duplicate content in the traditional sense. They serve as a mechanism for delivering updates about new posts or content on your website to users who have subscribed to your RSS feed through feed readers or news aggregators.
However, in some cases, if the content of your feed URLs mirrors the content of your website pages exactly, it could potentially be flagged as duplicate content by search engines. This scenario is rare because feed URLs usually contain excerpts or summaries of your content rather than the full content itself.
To avoid any potential issues with duplicate content, ensure that the content in your feed URLs is distinct from the content on your website pages. Providing unique excerpts or summaries in your feed can help prevent any duplicate content concerns.
Conclusion
RSS feed URLs are not a cause for concern, but managing them can enhance your website’s performance.
Whether through direct editing of the robots.txt file or using WordPress plugins, you can efficiently handle RSS feed URLs and improve your website’s SEO.
Remember, while it’s optional, optimizing your crawl budget is always a good practice in maintaining a healthy website.
