Gary Illyes from Google recently warned about the implications of soft 404 errors on web crawling, emphasizing the importance of proper error management to improve SEO and overall site performance.

Key Takeaways

• Soft 404 Errors Waste Resources: These errors mislead web crawlers, leading to inefficient resource usage.
• Correct HTTP Status Codes Are Crucial: Using accurate status codes is vital for effective crawling and indexing.
• Addressing Soft Errors Can Enhance SEO: Tackling these errors improves site performance and visibility.

In a recent LinkedIn post, Google Analyst Gary Illyes pointed out two critical issues affecting web crawlers: soft 404 errors and various types of “crypto” errors. While they may seem trivial, these errors can significantly harm your SEO efforts.

What Are Soft 404 Errors?

Soft 404 errors occur when a web server mistakenly returns a “200 OK” HTTP status code for pages that either do not exist or contain error messages. This miscommunication confuses web crawlers, leading them to waste resources on content that is either absent or unhelpful.

Illyes likens this scenario to visiting a coffee shop where every item on the menu is out of stock. While this might frustrate customers, it creates a much bigger problem for web crawlers.

He explains, “Crawlers rely on status codes to determine if a fetch was successful, even if the page content consists solely of an error message. They might repeatedly try to access the same page, wasting your resources, and if there are many such pages, resource consumption increases exponentially.”

The Hidden Costs of Soft 404 Errors

The repercussions of soft 404 errors extend beyond merely wasting resources for crawlers. Illyes notes that such pages are unlikely to appear in search results because they are filtered out during the indexing process.

To combat this issue, he advises webmasters to return the appropriate HTTP status code whenever a server or client encounters an error. This allows crawlers to accurately evaluate the situation and optimize their resource allocation.

Illyes also cautions against rate-limiting crawlers with messages like “TOO MANY REQUESTS SLOW DOWN,” as crawlers do not interpret such textual instructions.

Why This Matters for Your Site

Soft 404 errors can dramatically influence a website’s crawlability and indexing. By rectifying these issues, crawlers can focus on fetching and indexing valuable content, potentially boosting your site’s visibility in search results.

Eliminating soft 404 errors also leads to more efficient use of server resources, as crawlers won’t waste bandwidth by repeatedly accessing error pages.

How to Identify and Fix Soft 404 Errors

To effectively identify and resolve soft 404 errors on your site, follow these steps:

1. Regularly Review Crawl Reports: Analyze your website’s crawl reports and logs to identify pages that return an HTTP 200 status code despite displaying error messages.
2. Implement Proper Error Handling: Ensure your server delivers the correct HTTP status codes for error pages (e.g., 404 for “not found,” 410 for “permanently removed”).
3. Utilize Google Search Console: Monitor your site’s coverage using tools like Google Search Console to identify any pages flagged as soft 404 errors.

By proactively addressing soft 404 errors, you can improve your website’s crawlability, indexing, and overall SEO performance.