Roftr Labs

SEO

Robots and sitemap generator

Generate robots.txt and sitemap.xml together, with warnings for the rules that quietly empty a site out of search — and a cross-check that neither file can do on its own.

Runs entirely in your browser. Nothing you paste is uploaded.

Your site

Dates are optional and never invented. A URL with no date gets no lastmod — a sitemap where every date is today is one search engines learn to stop believing.

This runs entirely in your browser. Nothing you type is sent to us or to anyone else.

Enter your site address to generate robots.txt and a sitemap.

How to use it

  1. Fill in the formAdd the paths you want to allow or block, your sitemap address, and the URLs the sitemap should list. Nothing is sent anywhere: the file is built in your browser from what you type.
  2. Use the staging preset if the site is not public yetIt produces a robots.txt that blocks everything, and it turns the form red before you reach the output — because shipping that file to a live site is the most expensive mistake this tool can help you make.
  3. Read the cross-check between the two filesA URL listed in the sitemap while robots.txt blocks it is reported as critical. Neither file can see that on its own, and generating both together is the only reason this tool can.
  4. Notice what it refuses to emitNo priority and no changefreq — Google ignores both and says so. No invented lastmod either: a URL with no date you supplied gets no date, because stamping today onto everything is what makes the field stop being trusted site-wide.
  5. Read the warnings before you copy anythingWarnings appear ABOVE the generated code on purpose. A caution printed underneath output somebody has already copied has not been communicated, and the mistakes this tool warns about are the expensive kind.
  6. Copy or download the fileEach output block has its own Copy and Download button, so you can take one file without the other. Paste it where the block says it belongs.

About this tool

Two files govern how a search engine moves through your site. robots.txt says where crawlers may go; sitemap.xml says which pages you would like looked at. This builds both from the same form, which is the point — the commonest mistake in either file is one only visible when you hold them side by side.

It is the counterpart to our robots.txt tester and XML sitemap validator, which read these files on a live site. Same rules, pointed the other way. Everything runs in your browser; nothing you type is sent anywhere.

Understanding the result

robots.txt belongs at the root of your domain and nowhere else — example.com/robots.txt. A copy in a subfolder is never fetched by anything.

Warnings appear above the code, worst first. Some are corrections (a path with no leading slash is left out, because as written it matches nothing); some are facts about the file you are about to publish (a Disallow names the folder to anyone who reads it, and everyone can).

The cross-check is the part no single-file generator can do: a URL listed in your sitemap while robots.txt blocks it is reported as critical. You are asking a crawler to fetch a page and forbidding it in the same breath, and Search Console reports it as an error.

Example

Input

https://example.com · block /cart and /checkout · 3 URLs with dates

Output

User-agent: *
Disallow: /cart
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

Limitations

  • There is deliberately no priority or changefreq field. Google has stated it ignores both, and Bing has said the same of priority. Emitting them would make the file look considered while teaching that tuning numbers moves crawling, which it does not.
  • lastmod is never filled in for you. A date is used only where you supply one, because a sitemap where every date moves at once is one Google learns to stop believing — and then your real updates stop counting too.
  • robots.txt is a request, not a control. Well-behaved crawlers honour it; nothing obliges anyone to. It is not access control, not a privacy measure, and not a way to hide anything.
  • The files are generated, not checked against your live site. Once they are published, run them through the robots.txt tester and the sitemap validator to confirm what is actually being served.

Questions

Will Disallow keep a page out of Google?

No, and this is the misunderstanding worth fixing. Disallow stops a page being crawled, not indexed. A blocked URL that is linked from anywhere else can still appear in results — as a bare address with no description, because Google was not allowed to read it. Worse, if the page is already indexed, blocking it prevents removal: Google cannot see a noindex on a page it may not fetch. To remove a page, leave it crawlable and put a noindex on it.

Can I hide my admin folder by disallowing it?

You would be doing the opposite. robots.txt is public and always at the same predictable address, so anyone curious about your site reads it first. Listing /admin/ there tells them exactly where to look. Protect it with authentication and keep it off the list entirely.

Why is there no priority or changefreq?

Because they do nothing. Google has said publicly that it ignores both values, and Bing has said the same of priority. They persist in generators because a file full of numbers looks like it was tuned. Leaving them out keeps the file honest and costs you nothing.

Should I block AI crawlers?

That is a business decision, not an SEO one, and this tool takes no position. If you choose to, the option adds the tokens their operators document. Two things to know: every one of those blocks is honoured by choice, so a crawler that ignores robots.txt is not stopped by it — and blocking Google-Extended affects Gemini and Vertex training only, not Search. Your ranking and indexing are unchanged.

What goes in lastmod?

The date the page content genuinely last changed, in W3C form — 2026-08-07, or 2026-08-07T09:30:00+05:30. Leave it out if you do not have a real date. A date written any other way is left out and named rather than guessed at, because 12/03/2026 is March in Chennai and December in Chicago.

My sitemap has more than 50,000 URLs.

Split it. One sitemap file may hold 50,000 URLs and 50 MB uncompressed; past that you publish several and list them in a sitemap index, which is a small file of files. This tool warns when you cross the line rather than emitting something that will be rejected.

Last updated 2026-08-16.

Related tools