Skip to content
SnapTools

robots.txt Generator

Runs in your browser

Build a robots.txt file with disallow rules for a user-agent and an optional sitemap reference — the file crawlers check before indexing your site.

robots.txt
User-agent: *
Disallow: /admin
Disallow: /private

About this tool

robots.txt sits at your site's root and tells well-behaved crawlers which paths they may fetch — it's a request, not a security boundary, since it doesn't stop determined scrapers or hide content that's already linked elsewhere. Most sites need only a handful of rules: block admin and private paths, allow everything else, and point crawlers at the sitemap.

How to use

  1. Set the user-agent this rule block applies to ("*" means all crawlers).
  2. List the paths to disallow, one per line.
  3. Add your sitemap URL, then copy the result to /robots.txt at your site's root.

When to use this tool

  • Blocking crawlers from admin panels, staging paths, or internal search results.
  • Pointing search engines at your sitemap.xml.
  • Setting up crawl rules for a new site before launch.

Tips

  • robots.txt only requests that crawlers stay out — it does not password-protect anything. Never rely on it to hide sensitive content.
  • Blocking a page here also blocks it from passing link authority in some search engines; use a noindex meta tag instead if you want it de-indexed but still crawled.

FAQ

Where does this file need to live?
At the root of your domain — https://example.com/robots.txt — nowhere else works.
Does disallowing a page remove it from search results?
Not reliably. A disallowed page can still be indexed if it's linked elsewhere; use a noindex meta tag to actually remove it.

Related tools