Online website scanning: how a scanner reads your site

What happens during online website scanning: how pages are picked and rendered, what robots.txt means, and why a report can differ from your browser.

Infrastructure · Published: 9 October 2026 · 6 min read · Анна Ковалёва

You paste an address, wait a minute and get a report. Between those two moments a scanner makes dozens of HTTP requests, opens a headless browser, reads several files nobody ever looks at and decides which of your pages are worth reading at all. Knowing how online website scanning works inside is not trivia: it explains most of the "but on my machine it looks fine" moments and tells you how far a report can be trusted.

Step one: the entry page and the browser

Every scan starts from a single address. The scanner requests it, follows the redirects to the final URL and records each hop — this is where a chain like http → https → www → /en/ first shows up (see redirect hops).

The home page is then opened in a real headless Chromium. That matters for two reasons. First, modern sites build part of their content with JavaScript, and only a browser sees the result. Second, many hosts sit behind a CDN or a WAF that shows a "checking your browser" page to anything that does not run scripts. A browser lets that challenge finish. The same session also records console errors and failed requests, which a plain HTTP client never sees.

Step two: choosing which pages to read

A scanner is not a search engine. It does not try to read the whole site; it reads a sample. Candidates come from the links on the entry page and from the sitemap, and before anything is picked the list is cleaned up:

• addresses are normalised, so /page, /page/ and /page?utm_source=x are one page, not three; only query parameters that can name a page of their own (for example page, category, product) are kept;
• URL traps are skipped: endless calendars, paths that repeat the same segment, absurdly long or deep addresses, redirect loops;
• files that are not pages — images, PDFs, archives — are left out.

From the cleaned pool a handful of pages is picked. The free scan reads a few pages, which is enough to catch problems that live in templates: if the product template lacks a canonical, every product page lacks it. A repeat check of a watched site reads the same pages again, so two reports are comparable.

Step three: being a polite robot

A scan should never look like an attack. SiteScan sends one recognisable User-Agent, so a site can allow or refuse it on purpose, keeps a pause between requests to the same host, retries after 429 and transient 5xx while honouring Retry-After, and works within a fixed time budget. If a site answers too slowly, the scan stops at the limit and the report says which pages and checks were not reached, instead of silently pretending they were fine.

When a protection layer answers instead of the site, the page is marked as blocked, not as an error of yours. The report's crawl section shows the User-Agent, the pace, retries and every 403 split into "WAF" and "the site itself".

What robots.txt means for a scan

robots.txt is addressed to search crawlers that index the web on their own schedule. An on-demand scan that you start for your own site is a different kind of visit, and for the scanner robots.txt is above all an object of inspection. The report reads it and tells you whether it blocks crawlers, whether it hides CSS and JavaScript from them, whether it names a sitemap and whether that sitemap is valid. A single stray Disallow: / left over from staging is one of the most expensive mistakes a site can make, and it is exactly what a scan is good at catching; more in sitemap and robots.txt.

Why the report may differ from your browser

Location and IP. The scanner connects from a data centre. Geo-redirects, regional CDNs and anti-bot rules may treat it differently from your home connection.

No cookies, no login. You see the site as a returning, logged-in visitor who accepted the cookie banner. The scanner is a first-time anonymous visitor every time.

Caches. Your browser and your CDN may still hold yesterday's version. On the other side, a report is reused for fifteen minutes so that a shared link does not trigger a new crawl; press Rescan after a deploy.

Lab versus field. Speed metrics come from Google PageSpeed Insights for the home page: a lab run on an emulated device and, if the site has enough traffic, real-user data from the Chrome UX Report. A lab LCP of 3.1 s and a field LCP of 2.2 s are both true; read where the data comes from and the thresholds in Core Web Vitals.

Content after load. Anything injected long after the page settles, or only after a click, may be missing.

How to get the most out of a scan

1. Start from the address real visitors use, ideally the final one after redirects.
2. Check the crawl section first: if pages were blocked, fix access before reading scores.
3. Treat findings on one template as findings on all pages of that template.
4. After a fix, rescan and compare — do not rely on the old report.

To try it, run a free scan of your site; the product page lists what each analyser checks, and an online website check explains what the result looks like.