Sitemoth AI search readiness

Sitemoth checks how AI tools such as ChatGPT, Claude, Perplexity and Gemini can reach a site. The check runs after every crawl and fills the AI search tab. Open the full view from Analyze, AI search readiness.

AI crawler access

Sitemoth reads robots.txt for 19 AI crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and Applebot-Extended. For each one it shows the operator, what the crawler does and whether it is allowed, partly blocked or blocked. Sitemoth also reads noai and noimageai values and per-crawler meta tags and X-Robots-Tag headers.

Blocking a crawler that only collects training data is a business choice, so Sitemoth lists it as information. Blocking a search or user-triggered crawler keeps pages out of AI answers, so Sitemoth flags it as a warning.

llms.txt

Sitemoth fetches /llms.txt and /llms-full.txt and checks the format: an H1 title, an optional summary, and sections of links. It checks each link against the crawl. Generate llms.txt builds a file from the crawl, grouped by top-level folder and ranked by link score, for you to edit and save.

JavaScript-only content

Most AI crawlers do not run JavaScript. With rendering on, Sitemoth flags pages whose title, H1, meta description, canonical, main text or internal links appear only after scripts run.

AI crawlers in server logs

After a server log import, Sitemoth compares the pages AI crawlers request with the pages Googlebot requests, and lists AI crawler hits on errors and redirects.

sitemoth ai-readiness example.sitemoth
sitemoth llms-txt example.sitemoth -o llms.txt

Back to getting started with Sitemoth · Sitemoth home