robots.txt file

The file at /robots.txt that tells crawlers which addresses they may fetch and where the sitemap is. A rule that disallows the home page hides the whole site from search.

• Help center

robots.txt is a plain-text file at the root of the site that well-behaved crawlers read before anything else. It holds User-agent groups with Disallow and Allow rules, and Sitemap: lines that point to your sitemaps. It controls crawling, not indexing: a disallowed page can still appear in results if other sites link to it. To keep a page out of the index, use noindex instead.

How SiteScan checks it

The scanner fetches /robots.txt and evaluates the rules the way Google and Bing do: the longest matching rule wins, and Allow beats Disallow at equal length.

• Red, "Blocks crawlers": the rules for all crawlers (User-agent: *) disallow the page the scan started from, usually the home page. That hides the site from search.
• Green: the file exists; the value is its size.
• Amber, "Missing": no file. Crawlers then assume everything is allowed, but you lose the place to declare the sitemap.

A server error, a refusal (401, 403, 429) or an HTML error page is not treated as a missing file; the row says "Not checked".

A safe starting point

User-agent: *, an empty Disallow: and Sitemap: https://example.com/sitemap.xml, one per line. A leftover Disallow: / from a staging site is the most common cause of a red row. More in robots.txt and sitemaps; see also CSS and JS for crawlers.