Sitemoth
Features
Tabs across the data, filters inside each tab, a detail pane for the selected URL. If you have used a desktop crawler before, nothing here needs explaining. If you have not, every issue explains itself.
Crawling
Spider, list and sitemap modes
Start from a URL, a pasted list, or your XML sitemaps. Sitemap-only URLs are crawled after linked pages, so orphans are found and the URL limit favours what users can reach.
Scope control
Subdomains on or off, stay inside the start folder, include and exclude regexes, parameter stripping, depth and URL limits, robots.txt respected or ignored.
Authentication
Basic auth, cookies, custom headers. Staging sites and members areas crawl from your own machine with your own access.
Speed and politeness
Parallel requests with a per-host delay. Rate limiting by Cloudflare or your server is detected and reported instead of being misread as SEO issues.
Resources too
Images, CSS, JavaScript, PDFs and other files get status, size and inlinks. External links are checked.
JavaScript rendering
Headless Chromium with network idle detection, configurable settle time, resource blocking, custom scripts to dismiss banners or click load-more, optional screenshots.
Tabs and filters
| Tab | Filters |
|---|---|
| Internal | HTML, JavaScript, CSS, Images, PDF, Other, Indexable, Non-Indexable |
| External | HTML, Images, Other |
| Security | HTTP URLs, Mixed Content, Form on HTTP URL, Form Posts to HTTP, Unsafe Cross-Origin Links, Protocol-Relative Resource Links, Missing HSTS Header, Missing Content-Security-Policy, Missing X-Content-Type-Options, Missing X-Frame-Options, Missing Secure Referrer-Policy, Bad Content Type |
| Response Codes | Blocked by Robots.txt, Blocked Resource, No Response, Success (2xx), Redirection (3xx), Redirection (Meta Refresh), Redirection (JavaScript), Client Error (4xx), Server Error (5xx), Redirect Chain, Redirect Loop, Rate Limited, Internal Redirection (3xx), Internal Client Error (4xx), External Client Error (4xx) |
| URL | Non ASCII Characters, Underscores, Uppercase, Parameters, Over 115 Characters, Repetitive Path, Multiple Slashes, Contains Space, Session ID |
| Page Titles | Missing, Duplicate, Over 60 Characters, Below 30 Characters, Over 580 Pixels, Same as H1, Multiple, Outside <head> |
| Meta Description | Missing, Duplicate, Over 155 Characters, Below 70 Characters, Over 990 Pixels, Multiple |
| Meta Keywords | Missing, Present |
| H1 | Missing, Duplicate, Over 70 Characters, Multiple |
| H2 | Missing, Duplicate, Over 70 Characters, Non-Sequential |
| Content | Exact Duplicates, Near Duplicates, Low Content Pages, Soft 404 Pages, Readability Difficult, Lorem Ipsum Placeholder, Mojibake, Banned Phrases |
| Images | Over 100 KB, Missing Alt Attribute, Missing Alt Text, Alt Text Over 100 Characters, Missing Size Attributes, Broken |
| Canonicals | Contains Canonical, Self Referencing, Canonicalised, Missing, Multiple, Multiple Conflicting, Non-Indexable Canonical, Relative, Unlinked, Protocol Mismatch |
| Pagination | Contains Pagination, First Page, Non-Indexable Target |
| Directives | Index, Noindex, Nofollow, None, Noarchive/Nosnippet, Multiple Robots Tags, Refresh |
| Hreflang | Contains Hreflang, Missing Return Links, Non-200 Hreflang URLs, Missing Self Reference, Missing X-Default, Invalid Language Code, Non-Canonical Hreflang, Noindex Hreflang URLs, Multiple Entries, Inconsistent Language |
| JavaScript | Noindex Only in Rendered HTML, Canonical Only in Rendered HTML, Canonical Mismatch, Page Title Updated by JavaScript, Meta Description Updated by JavaScript, H1 Updated by JavaScript, Contains JavaScript Content, Contains JavaScript Links, Pages with JavaScript Errors, Render Timeout |
| Links | Pages With High Crawl Depth, Orphan URLs, Internal Outlinks With No Anchor Text, Non-Descriptive Anchor Text, Internal Nofollow Outlinks, Outlinks To Localhost, Pages Without Internal Outlinks, Pages With High External Outlinks, Pages With High Internal Outlinks, Only Nofollow Inlinks, Only One Inlink |
| Structured Data | Contains Structured Data, Missing, Parse Errors, Validation Errors, Validation Warnings, Mixed Formats |
| Sitemaps | URLs in Sitemap, URLs not in Sitemap, Orphan URLs, Non-Indexable URLs in Sitemap, Noindex URLs in Sitemap, Canonicalised URLs in Sitemap |
| Validation | Missing Doctype, Invalid Element in <head>, Charset After 1024 Bytes, Missing Viewport, Missing html lang, HTML Over 15 MB, Multiple head/body |
| Custom Search | Contains |
| Custom Extraction | Extracted |
Analysis after the crawl
Duplicates
Exact duplicates by content hash and near duplicates by similarity, with the closest match for each page.
Link graph
Inlinks, unique inlinks, outlinks, crawl depth by shortest path from the start page, and a link score (PageRank over internal links).
Redirects and canonicals
Chains, loops, redirects to error pages, canonicals pointing at redirected, erroring or noindex pages, unlinked canonical targets.
Hreflang
Return links, self reference, x-default, language codes, targets that are non-200, noindex or canonicalised, language mismatches.
Sitemaps
Discovered from robots.txt and common paths, nested indexes followed. Non-200, noindex and canonicalised URLs in sitemaps, indexable URLs missing from them, orphans.
Structured data
JSON-LD, microdata and RDFa detected; required and recommended properties checked for the common Google rich result types.
Output
Exports
CSV and Excel per tab and filter, an issues overview, and bulk exports: all inlinks, all outlinks, external links, 4xx and 5xx inlinks, redirect inlinks and chains, image alt reports, canonicals, hreflang, structured data, anchor text, sitemap URLs, custom extraction.
Crawl comparison
New and removed URLs, status and indexability changes, title, canonical and word count changes, new and resolved issues between two crawl files.
XML sitemap generator
From indexable 200 pages, with lastmod and optional image entries.
Custom extraction and search
CSS selectors or regular expressions pull any value from every page; custom search counts matches.
Content rules
Banned phrases flagged wherever they appear. Useful when positioning changes and old copy lingers.
Command line
sitemoth crawl, export, compare, sitemap, issues. Runs anywhere Node runs.
Compared with Screaming Frog
| Screaming Frog Free | Screaming Frog Paid | Sitemoth Free | Sitemoth Pro | |
|---|---|---|---|---|
| URL limit | 500 | Unlimited | Unlimited during the beta | Unlimited |
| Price | Free | £199 per user per year | Free | $149 once, a year of updates included |
| JavaScript rendering | No | Yes | Yes | Yes |
| Save and reopen crawls | No | Yes | Yes | Yes |
| Crawl comparison | No | Yes | Yes | Yes |
| Custom extraction and search | No | Yes | Yes | Yes |
| Crawl configuration (scope, speed, auth) | No | Yes | Yes | Yes |
| Scheduling and command line | No | Yes | Yes (CLI) | Yes (CLI) |
| Excel and CSV exports, bulk exports | Yes | Yes | Yes | Yes |
| Structured data validation | Yes | Yes | Yes | Yes |
| Issues with fix guidance | Yes | Yes | Yes | Yes |
| Google Search Console and Analytics | No | Yes | Not yet | Not yet |
| PageSpeed Insights | No | Yes | Not yet | Not yet |
| Accessibility and spelling checks | No | Yes | Not yet | Not yet |
| Crawl visualisations | Yes | Yes | Not yet | Not yet |
Screaming Frog figures from its public pricing and feature pages, October 2026. "Not yet" means it is on our roadmap and not in the current build.