Robots.txt tester

How to use it

1. 1Enter the address you want to checkType or paste it into the field marked "Address to check", including the https:// at the front. Give it any URL on the site — the tool finds that site's robots.txt itself.
2. 2Press Check and wait a few secondsThe request runs from our server rather than from your browser, so your cache, your extensions and your login session cannot affect the answer. That is the point of it: you are seeing what an anonymous visitor sees.
3. 3Read the verdict per crawlerThe same file can allow Googlebot and block Bingbot. The tool answers for each named crawler separately rather than giving one blended answer that is true for none of them.
4. 4Look at the exact line that decided itThe rule that matched is quoted with its line number. This is almost always the useful part: the usual cause of a blocked page is a line the owner did not know was in the file.
5. 5Remember what robots.txt cannot doIt controls crawling only. It cannot remove a page from an index, and blocking a page actually PREVENTS removal, because the noindex tag that would remove it can no longer be seen.
6. 6Share or keep the result within 24 hoursThe result has its own link you can send to whoever needs to act on it. It is deleted 24 hours after the check ran, so save anything you need to keep — and re-run the check after a change rather than trusting an old link.

About this tool

This tool fetches a site's robots.txt and answers one question with evidence: may this URL be crawled? It answers it separately for Googlebot, Bingbot and the others, because they can genuinely disagree — and names the exact line that decided each verdict.

It exists because robots.txt matching is not what most people expect. The rule that applies is not the first one that matches; only one user-agent group applies to a given crawler; and rules are not inherited from the wildcard group when a named one exists.

Understanding the result

Each crawler gets its own answer with the line number behind it, so you can open your own robots.txt and see the same line we did.

The longest matching path wins, and Allow beats Disallow on a tie. This is why a broad Disallow followed by a narrower Allow permits the URL — and why "first match wins" produces the wrong answer.

Lines that do not do what they look like are called out separately. A misspelled directive is ignored silently by every crawler, so the file reads as protective while protecting nothing.

The whole file is shown as served. A tester that will not show you the file is asking to be trusted.

Example

https://www.google.com/search?q=test

Googlebot: blocked · line 3 · Disallow: /search (User-agent: *)
Bingbot: blocked · same rule
Blocked from crawling is not removed from search.

Limitations

• This models the documented behaviour of major crawlers. A crawler is free to ignore robots.txt entirely, and badly behaved ones do.
• Crawl-delay is reported but Google ignores it; Bing and Yandex honour it. Set Google's crawl rate in Search Console instead.
• A robots.txt that redirects to another origin is followed, matching Google's behaviour, but it is usually accidental rather than intended.
• We test a fixed set of user agents. A rule targeting some other named bot is shown in the parsed file but not given its own verdict.

Questions