CSS and JS blocked in robots.txt

Whether robots.txt lets Googlebot fetch the CSS and JavaScript the pages need. Google renders a page before it judges it, and blocked assets mean it sees the page unstyled.

• Help center

Google does not judge a page from its HTML alone: it renders the page the way a browser does, with its stylesheets and scripts, and only then decides what the page contains and whether it works on a phone. If robots.txt forbids Googlebot from fetching those files, it sees an unstyled page or one whose JavaScript content never appears. A layout that looks fine to visitors can then fail mobile-friendliness and lose content in the index.

How SiteScan checks it

During the crawl the scanner collects the CSS and JavaScript files the pages load and tests each path against the rules in robots.txt that apply to Googlebot (its own group if there is one, otherwise User-agent: *). Green "Allowed" means none is blocked; red "Disallowed" gives the number of blocked files. The row appears only when robots.txt exists and assets were found.

How to fix

• Remove broad rules that cover asset folders, such as Disallow: /wp-includes/, Disallow: /assets/, Disallow: /static/ or Disallow: /_next/.
• If a folder must stay closed, open the files explicitly in the same group: Allow: /*.css$ and Allow: /*.js$.
• Then check a page with the URL Inspection tool in Google Search Console and look at the rendered screenshot.

Related: the robots.txt row and robots.txt and sitemaps.