Broken links, also known as dead links or 404 errors, are hyperlinks that no longer work, resulting in a frustrating user experience. To find and fix broken links with Screaming Frog, enter your website’s URL, start a crawl, then go to the “Response Codes” tab, and select “Client Error (4xx)” to identify broken links. To fix them, update or redirect the broken URLs on your website.
In this article, I will show you how to identify broken links and remove 404 pages using Screaming Frog.
Step 1: Download and Install Screaming Frog SEO Spider
If you haven’t already, you can download Screaming Frog SEO Spider from their official website (https://www.screamingfrog.co.uk/seo-spider/). After downloading, follow the installation instructions for your operating system. Once the installation is complete, you’re ready to start using Screaming Frog.
You can only crawl up to 500 URLs with the Screaming Frog SEO Spider free version. It is enough for a smaller site, but if your website has more than 500 pages, you won’t be able to analyze all of them using the free version of this tool. For unlimited crawling and access to advanced features, you would need to purchase a licence (currently at €239 per year).
Step 2: Crawl the website with Screaming Frog
Paste your website’s URL in the “Enter a URL to Spider” field and click the “Start” button. Screaming Frog will then begin to crawl the website, and you will be presented with a report of all the URLs and associated data.
If you only need to analyse specific folders (for example, Blog only) or exclude pages (such as author pages), you can use the “Include” and “Exclude” options in the tool.
In the “Crawl Config”, you can specify rules based on URL patterns to focus the crawl on certain sections of your website while excluding others. This allows you to control which parts of your site are crawled and which are ignored during the crawling process.
Step 3: Filter for Client Errors
Once the crawl is complete, it’s time to filter the results to find out which pages have broken links. Go to the “Response Codes” tab. Select the “Client Error (4xx)” filter, which will display all the broken links with 4xx HTTP status codes, including 404 (Not Found) errors.
Your report will display any 4XX HTTP status codes found during the crawl. These codes indicate URLs that couldn’t be found or had an error in the request. By looking at the URLs listed under Client Errors, you can identify which pages on your website have broken links.
Step 4: Analyze your list of Broken Links
Once you’ve identified broken links through the report, it’s time to fix them. You can either update or remove the broken links altogether. Updating the links involves finding a new URL for the same content or replacing the broken URL with new content. If you can’t find a suitable replacement URL, remove the link altogether. A broken link is much worse than no link at all.
Depending on the nature of the broken link, you can take the following actions:
- Fix: If the broken link is pointing to a valid page on your site with a different URL, update the link to the correct URL.
- Redirect: If the destination page no longer exists, but you have a relevant replacement page, set up a 301 redirect to guide users to the new page.
- Remove: If the link is no longer necessary, remove it from the referring page.
Step 5: Fix your Broken Links
For websites with numerous broken links, you can export all your inlinks with Screaming Frog.
Go to “Bulk Export” in your main navigation, select “Response Codes,”, then “Internal” and choose “Internal Client Error (4xx) Inlinks” to export all 404 pages found. You can repeat the same step to investigate other types of issues, such as 301 and 500 errors.
Open the exported data in Excel or Google Sheet, where you’ll see the “Source” (the page where the broken link is linking from) and the “Destination” (the non-working URL). You can now proceed to fix these in bulk by adjusting the source page or resolving issues with the destination page.
I will usually add a column for the checkbox to track my progress.
Step 6: Re-crawl to validate fixes
Finally, it’s essential to re-crawl the website after fixing the broken links to confirm that the issue has been resolved. This is done by simply re-crawling the website in Screaming Frog and filtering for Client Errors once again. If the issue has been fixed, the report should show a decrease in Client Error URLs. If the issue persists, it may require additional investigation to determine the root cause of the problem.
Conclusion
Broken links can negatively impact the user experience of a website and may affect its search engine rankings. Therefore, it’s essential to regularly check for broken links. Screaming Frog is an effective tool for finding and fixing broken links.
By following the steps above, you should be able to identify the broken links on your website and take appropriate action to fix them. Doing so will improve your website’s crawlability, provider better user experience, and maintain a healthy website.
