Full-site crawler

How to use it

1. 1Enter the address you want to checkType or paste it into the field marked "Address to check", including the https:// at the front. Give it the home page, or wherever you want the crawl to start. It stays on that host.
2. 2Press Check and wait a few secondsThe request runs from our server rather than from your browser, so your cache, your extensions and your login session cannot affect the answer. That is the point of it: you are seeing what an anonymous visitor sees.
3. 3Set how much of the site to visitMax pages and max depth bound the crawl, and the sitemap toggle decides whether we seed from sitemap.xml as well as from links. Start small — fifty pages tells you most of what a thousand will, and it is somebody else's server.
4. 4Let it run — it is deliberately unhurriedOne request at a time, with any Crawl-delay in robots.txt obeyed and a back off if the site starts refusing. Progress is shown as it goes, and partial results are kept, so nothing is lost if it stops early.
5. 5Read why it stopped before you read the numbersA crawl that hit your page cap has not seen the site, and every count below is "out of what we visited" rather than "out of your site". The result says which of the two happened, because reporting our own limit as your site's problem is a crawler's oldest failure mode.
6. 6Work the consolidated findings, not the page listIssues are grouped by rule with a count of affected pages, so "42 pages have no meta description" is one line rather than forty-two. Open a finding to see which pages, with the evidence from each.
7. 7Share or keep the result within 24 hoursThe result has its own link you can send to whoever needs to act on it. It is deleted 24 hours after the check ran, so save anything you need to keep — and re-run the check after a change rather than trusting an old link.

About this tool

Every other tool here answers a question about one page. Some questions cannot be answered that way at all: whether two pages share a title, whether an internal link is broken, whether a page exists that nothing links to. Those need the whole site in view, and that is what this does — it starts at the address you give it, follows the links it finds, and checks each page it reaches.

It crawls one site: the host you submitted. A link to another domain is recorded so you can see where your site points, and then left alone. It reads robots.txt before every address, not just the first, and if the site asks crawlers to stay out, the crawl does not start.

Understanding the result

Read the coverage line before anything else. "We crawled 43 pages and stopped because the page limit was reached" is the context for every number under it — a finding about 12 pages is 12 of the pages we looked at, not 12 of your site.

Every finding carries the number of pages checked alongside the number affected, because the two together are the only honest form of the statement. 17 of 20 pages missing a description is a broken template; 17 of 4,000 is a handful of stragglers.

The skipped list is not an error log. It is the addresses we found and deliberately did not fetch — off-site links, files that are not pages, anything robots.txt disallows, and anything past the limits you set. A crawler that showed only what it fetched would be presenting its own limits as the size of your site.

Duplicate titles and descriptions are grouped across the whole crawl. This is the finding that genuinely cannot come from a single-page check: no amount of looking at one page tells you another page says the same thing.

Example

https://example.com/ · at most 50 pages · 3 links deep

Crawled 43 pages — stopped because the page limit was reached

critical  2 of 43 pages returned an error
warning  12 of 43 pages have no meta description
warning   4 of 43 pages share a title with another page

11 addresses skipped: 7 off-site, 3 not HTML, 1 disallowed by robots.txt

Limitations

• A crawl finds pages by following links and by reading your sitemap. A page that nothing links to and no sitemap lists is invisible to any crawler, including this one — and including Google's. Leaving the sitemap option on is what gives us the best chance of finding those.
• Pages are read as delivered, without running JavaScript. On a site that builds its links in the browser, a crawl will find fewer pages than a visitor does. Our raw-vs-rendered tool shows the size of that gap for a single page.
• The checks run against each page are the ones that read the page we already downloaded. Checks that would fetch extra files per page — opening every link, downloading every icon — are excluded on purpose: across three hundred pages they would turn a polite crawl into thousands of requests against your server.
• One request at a time, with a pause between them, and any Crawl-delay in your robots.txt is honoured. If your server starts returning 429 or 503 we back off, and if it keeps doing so we stop and say so rather than keep pushing.
• Results are deleted after 24 hours, along with every page record the crawl made. Requests to private and internal addresses are refused by design.

Questions