Configuration
Limits and speed
| Setting | Meaning |
|---|---|
| Max URLs | Stop after this many fetched URLs. 0 means no limit (subject to your license). |
| Max crawl depth | Do not follow links deeper than this many clicks from the start page. |
| Parallel requests | How many requests are in flight at once. 5 is polite for most hosts; 1 or 2 for shared hosting. |
| Delay | Minimum gap in milliseconds between two requests to the same host, shared across all parallel requests. |
| Timeout and retries | Per request. Transport errors and rate limiting are retried; the last failure is recorded on the URL. |
Scope
Crawl subdomains treats blog.example.com as internal. Crawl outside the start folder lets a crawl that starts at /blog/ leave it. Include and exclude take JavaScript regular expressions, one per line. Query parameters to strip removes tracking parameters before URLs are queued, so ?utm_source=x variants collapse.
Resources and external links
Images, CSS, JavaScript and other files are fetched for status and size. External links are checked with a GET capped after the headers. All four are on by default.
robots.txt and nofollow
Blocked URLs are recorded with their reason rather than silently skipped. Nofollow links are recorded as outlinks but not crawled unless you turn on Follow internal nofollow.
Identity
The default user agent identifies Sitemoth. Browser-like headers make requests look like Chrome, which helps with bot protection on your own sites. Basic auth, cookies and custom headers are sent with every request, including rendering.
Content
Store HTML keeps raw and rendered HTML inside the crawl file so you can view source later and include it in bug reports. Banned phrases, custom extraction and custom search are described in Tabs and filters.