AI website audit: a checklist for AI search and assistants

An AI website audit in six steps: crawler access in robots.txt, content without JavaScript, llms.txt, structured data, clear answers and measurement.

SEO · Published: 4 October 2026 · 5 min read · Ольга Мирошник

Answer engines and chat assistants now send real visitors — and, more often, answer without sending anyone. Whether your site is the source they quote depends on things you can check. Our article on how AI assistants read a site explains the principles; this one is the working checklist. An AI website audit takes an afternoon, and you can repeat it every quarter.

Step 0. Decide what you want

Before touching robots.txt, separate two questions. Do you allow your content to be used for training models? And do you want to appear in AI search answers with a link back? They are controlled by different crawlers, and many sites want "no" to the first and "yes" to the second. Write the decision down; the rest of the audit checks whether the site actually implements it.

Step 1. Access: robots.txt, firewall, status codes

The main user-agent tokens, as published by their operators:

• OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (fetches on a user's request).
• Anthropic: ClaudeBot, Claude-SearchBot, Claude-User.
• Perplexity: PerplexityBot, Perplexity-User.
• Google: Google-Extended is a robots.txt token, not a crawler — it governs use in Gemini models; AI Overviews follow ordinary Googlebot rules.
• Others: Applebot-Extended, CCBot (Common Crawl, a frequent training source).

A typical "no training, yes search" policy:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Then check that reality matches the file. A CDN "bot fight" mode or WAF often returns 403 or a challenge page to these agents regardless of robots.txt. Test with curl -I -A "OAI-SearchBot" https://example.com/ and look at the status. See also the robots.txt reference.

Step 2. Content without JavaScript

Most AI crawlers do not execute JavaScript. Open a key page with scripts disabled, or look at the raw HTML: are the headline, the main text, prices and dates there? If the page is an empty shell filled in by the client, the assistant sees nothing. Server-side rendering or prerendering fixes this.

Step 3. llms.txt and llms-full.txt

llms.txt is a short Markdown map of the site at the root: who you are, the key sections, links to the pages that matter. llms-full.txt is the long form, typically for documentation. It is a convention, not a standard, and no assistant is obliged to read it — but it costs an hour and removes guesswork.

Step 4. Structured data and attribution

• Organization markup with name, logo and sameAs on the home page; Article with author and dateModified on articles; Product with price and availability in the shop. Details in our guide to structured data.
• One address per piece of content: a correct canonical, so the citation points to the right URL.
• A visible author, publication date and the company name the same way everywhere.

Step 5. Clarity of content

Assistants quote fragments. A page is quotable when the answer comes first and the explanation second; when headings are questions or plain statements, not slogans; when numbers, terms and conditions are written out in text rather than hidden in images or accordions. Read your top ten pages and ask: can one paragraph be lifted out and still be correct?

Step 6. Measure and repeat

Look in server logs for the user agents from step 1 and their status codes — 403s and 5xx are your first fixes. In analytics, segment referrals from AI services. Repeat the audit after every major release: a new firewall rule or a client-side rewrite can undo everything silently.

Running an AI website audit with SiteScan

1. Start a free scan. The report shows whether robots.txt blocks crawlers and whether llms.txt and llms-full.txt are present.
2. For each page, the report lists the structured data found, the canonical, title and description — enough to cover steps 3 and 4.
3. AEO analysis is part of the deeper audit types; see pricing.
4. To turn findings into tasks and have an assistant explain them, follow our guide to an AI website check.