About RoftrlabsBot

How to identify it

Requests carry this user-agent string:

RoftrlabsBot/1.0 (+https://roftrlabs.com/crawler)

What it does

• Fetches only in response to a person submitting an address to one of our tools. There is no background or scheduled crawling of sites that nobody has asked about, and we do not build an index.
• Most tools fetch exactly one page. Our site crawler is the exception: one submission asks for up to a few hundred pages of that one site, one request at a time with a gap between them, within limits the person chose and we cap. It never wanders onto another domain, and the result states how many pages it fetched and why it stopped.
• Requests over HTTP and HTTPS only, on the standard ports 80 and 443.
• Follows at most five redirects, times out quickly, and stops reading a response once it passes a size ceiling.
• Reads pages. It never submits forms, never attempts to log in, and never tests credentials or scans for vulnerabilities.

How to block it

Add this to your robots.txt:

User-agent: RoftrlabsBot
Disallow: /

We read your robots.txt before fetching anything, and we stop when a rule names us. Anyone asking our tools to check that address is then told the site declines, and shown the rule and line number that said so. If you would rather block us outright, refusing the user-agent at your web server or firewall works too.

Exactly what we obey, and what we do not

We would rather state this precisely than let you assume something we do not do.

• A group naming RoftrlabsBot stops us completely — every tool, every address it covers, whether one page or many.
• A general User-agent: * rule does not stop us fetching a single page that someone has typed into one of our tools. A blanket Disallow: / is what a staging or pre-launch site looks like, and those are exactly the sites people need to check. It is one request, for an address a person just gave us. It does stop our site crawler, which is the automated bulk crawler such a rule is written for — a crawl of a site whose robots.txt disallows crawlers never starts, and the person who asked is told why.
• If we ever cannot read your robots.txt, one page still gets fetched. Nothing bulk does.

If you want us gone regardless of how the request reached us, name us. That is what the block above does.

Private and internal addresses

The crawler will not connect to loopback, private, link-local or reserved addresses, in either IPv4 or IPv6, and it re-checks every redirect against the same rules. An address that resolves into a private network is refused rather than fetched.

Getting in touch

If RoftrlabsBot is causing a problem on your site, contact Roftr Clouds LLP and we will look into it.