Skip to content
Toolify
SEO

Robots.txt Generator

A robots.txt file tells search engine crawlers which parts of your site they may or may not access. This generator builds a valid file from simple options — allow or disallow paths, target specific user-agents, set a crawl delay, and include your sitemap URL — ready to drop at your site's root.

robots.txt
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Upload this file at the root of your domain, e.g. https://example.com/robots.txt.

How to use the Robots.txt Generator

  1. 1Choose crawler access. Allow everything, block everything, or add specific disallow paths.
  2. 2Add your sitemap. Include the full URL to your XML sitemap so crawlers can find it.
  3. 3Copy and deploy. Copy the file and upload it as robots.txt at your domain root.

Frequently asked questions

Where does robots.txt go?

It must live at the root of your domain, for example https://example.com/robots.txt, so crawlers can find it.

Does disallow guarantee a page won't appear in Google?

No. Disallow blocks crawling, but a URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta tag instead.