robots.txt and sitemap.xml: two files that decide what gets indexed

The difference between blocking a crawl and blocking an index, why a stale sitemap wastes crawl budget, and the one-line mistake that hides a whole section.

SEO · Published: 6 September 2026 · 4 min read · Марк Лебедев

These two files are the shortest configuration on your site and the easiest place to do lasting damage. A single line in robots.txt can remove a section from search, and it will not look like anything is wrong — the pages still load, the links still work, and traffic simply stops.

robots.txt does not do what most people think

It controls crawling, not indexing. A blocked URL can still appear in results — without a description, because the crawler was never allowed to read it. If your goal is "this page must not appear in search", robots.txt is the wrong tool: use <meta name="robots" content="noindex"> and make sure the page is crawlable, because a crawler that cannot fetch the page will never see the noindex.

That combination is the classic mistake:

Disallow: /private/          # in robots.txt
<meta name="robots" content="noindex">   # on the page

The two cancel out. The engine cannot fetch the page, so it never learns it should drop it.

What belongs in robots.txt

Search endpoints, faceted filter URLs, endless calendars, cart and account paths — anything that generates infinite variations and no value. Keep it short and specific. Two rules to keep in mind: the most specific matching rule wins, and blocking /assets/ or /static/ can stop the crawler rendering your pages at all, which affects how they are evaluated.

Also point to your sitemap from it:

Sitemap: https://site.com/sitemap.xml

What belongs in the sitemap

Only URLs you actually want indexed: status 200, canonical to themselves, not blocked in robots.txt, not noindex. Every URL that fails one of those tests is a wasted hint, and a sitemap full of wasted hints gets trusted less.

lastmod is worth setting honestly — meaning it changes when the content changes, not on every build. A sitemap where every page claims to have been modified today teaches the crawler to ignore the field. priority and changefreq, by contrast, are largely ignored by the major engines; do not spend time tuning them.

Over 50,000 URLs or 50 MB, split into several files behind a sitemap index. Generate it from the same source that generates your routes, so a deleted page disappears from both at once.

The failures worth checking for

• URLs in the sitemap that redirect, 404, or are blocked in robots.txt.
• Pages that exist and are linked from navigation but appear in no sitemap.
• A Disallow: / left over from a staging environment and deployed to production. This one costs a site its entire search presence, and it happens more often than anyone admits.
• A sitemap that has not changed since the last CMS migration.

Verify from outside

The check that matters is not "does the file look right" but "does what it says match the site". Crawl the site, take the link graph, take the sitemap, and compare the two sets: pages in the crawl but not the sitemap, URLs in the sitemap that the crawl cannot reach or that answer with anything but 200. The disagreements between those two lists are, reliably, the most valuable SEO findings a site has.