404 error handling and soft 404

What the server answers for an address that does not exist. It has to be a 404: a missing page that answers 200 gets indexed as a real one.

• Help center

When someone asks for a page that does not exist, the server must say so with HTTP 404 (or 410 for “gone for good”). The error page can look however you like — friendly text, a search box, links home — but its status code has to be an error. When a missing page answers 200, that is a “soft 404”: search engines treat it as a real page, index thin duplicates of your error screen, and spend crawl budget on addresses that should not exist. Monitoring tools also miss broken links, because every URL appears to work.

How SiteScan checks it

The scanner requests a random path that cannot exist on the site and reads the status of the final answer:

• green: 404 or 410;
• amber “Soft 404”: a 2xx status — including a redirect to the home page that ends in 200;
• amber with a status code: anything else, for example 500 or 403, which tells crawlers the server is broken or forbidding access rather than that the page is missing.

How to fix

Make the catch-all route return a real 404 status: in nginx error_page 404 /404.html;, in a single-page app let the server or the framework set the status for unknown routes instead of always serving index.html with 200. Do not redirect unknown addresses to the home page. Background and edge cases are in the article on soft 404 errors.