Robots.txt File Generator

Generate valid RFC 9309 compliant robots.txt files with custom crawl rules, sitemap declarations, and one-click AI scraper protection (GPTBot, ClaudeBot, CCBot).

Blocks GPTBot, ClaudeBot, CCBot, PerplexityBot, and Google-Extended from training on your content.

/admin//api//private/
/
Generated robots.txt Preview

Robots.txt Specification & RFC 9309 Rules

The robots.txt file is placed at domain apex root (e.g. https://example.com/robots.txt) to instruct automated web crawlers which URL directories they may request. Formalized in 2022 under IETF RFC 9309, it controls crawl budget efficiency and protects private server endpoints from scraper overload.

Core Directives & Crawler Specificity Rules

  • User-agent: Declares the crawler identifier (e.g. User-agent: Googlebot or wildcard User-agent: *).
  • Disallow: Specifies path prefixes that matching bots must not crawl. An empty value (Disallow:) allows crawling of the entire domain.
  • Allow: Explicitly permits crawling of a specific sub-path inside an otherwise disallowed directory (e.g. Disallow: /blog/ with Allow: /blog/public/).
  • Sitemap: Fully-qualified absolute URL pointing to your XML sitemap index.

Blocking AI Scrapers vs. Preserving Search Rankings

AI companies deploy specialized user-agent strings specifically for model training separate from their web search crawlers:

  • GPTBot & ChatGPT-User — OpenAI scraping bots. Blocking these does not affect Google or Bing search indexing.
  • ClaudeBot & anthropic-ai — Anthropic AI training spiders.
  • Google-Extended — Google's AI training crawler (Gemini/Vertex AI). Disallowing Google-Extended does not stop standard Googlebot from indexing your pages for Google Search.

Technical SEO & Indexation Toolkit

Combine robots.txt management with our full indexation diagnostic suite:

Frequently Asked Questions

Does Disallow in robots.txt remove a page from Google Search?

No. Disallowing a URL stops search engine bots from crawling it, but if other websites link to that URL, Google can still index it as a bare URL without page snippet content. To guarantee removal, use a noindex meta tag.

Where must the robots.txt file be hosted?

It must be hosted at the exact root of your domain via HTTP/HTTPS on port 80/443 (e.g. https://example.com/robots.txt). Placing it in subdirectories (e.g. /assets/robots.txt) has no effect.

Does Googlebot obey Crawl-delay?

No. Googlebot ignores the Crawl-delay directive (crawl rate can be adjusted in Google Search Console). Bingbot and Yandex still respect standard Crawl-delay headers.