🤖 Robots.txt Checker

🔍 Fetches /robots.txt from any domain and parses every User-agent group, Allow/Disallow rule, Crawl-delay, and Sitemap declaration.

Robots.txt Architecture & the Robots Exclusion Protocol (RFC 9309)

The robots.txt file is a plaintext file placed at the root of a domain that instructs automated web crawlers which URL paths they are permitted or forbidden to request. Formalized under IETF RFC 9309, it serves as the foundational gatekeeper for crawl budget management and server resource preservation.

Core Syntax & Directive Rules

1. User-Agent Grouping

Directives apply to specific crawler identifiers. A group begins with one or more User-agent lines followed by Allow or Disallow directives:

# Allow Googlebot full access
User-agent: Googlebot
Disallow: /checkout/
Disallow: /api/

# Block aggressive AI scrapers
User-agent: GPTBot
User-agent: CCBot
User-agent: ClaudeBot
Disallow: /

# General fallback for all other crawlers
User-agent: *
Disallow: /private/
Allow: /

2. Rule Specificity and Matching Logic

When multiple rules in the same group match a requested URL, search engines follow the longest matching path rule, not line order.

  • Disallow: /catalog/ (9 chars)
  • Allow: /catalog/public/ (16 chars)

A request for /catalog/public/item.html will be allowed because the Allow pattern is longer and more specific than the Disallow pattern.

3. Wildcards (* and $)

  • * matches zero or more characters (e.g., Disallow: /*.pdf$ blocks all URLs ending in .pdf).
  • $ designates the end of the URL pattern (e.g., Disallow: /*?* blocks any URL containing query parameters).

The Sitemap Directive

You can declare one or more XML sitemaps anywhere in robots.txt. This helps search engines discover your sitemap index without manual submission:

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-news.xml

Verify the format and availability of your sitemaps using our XML Sitemap Checker.

Crucial Gotchas: What Robots.txt Does NOT Do

  • Robots.txt does not guarantee non-indexation: If external websites link to a disallowed path, Google may still index the URL without crawling the page content. To ensure complete removal from search results, use a noindex directive on an accessible page.
  • Robots.txt is public: Never use Disallow to conceal sensitive endpoints or hidden admin folders, as the file itself is publicly readable by anyone.
  • Crawl-delay is ignored by Googlebot: Google does not honor Crawl-delay in robots.txt. If your server experiences crawl overload from Google, adjust crawl rates inside Google Search Console settings.

Frequently Asked Questions

What happens if a website has no robots.txt file?

If a server returns a 404 (Not Found) for /robots.txt, search engine crawlers interpret this as unrestricted crawl access and will crawl all discoverable links on the domain.

Can I block CSS and JavaScript files in robots.txt?

No. Googlebot needs access to CSS and JavaScript assets to render pages accurately for mobile indexing and visual layout verification. Blocking resources in /static/ or /assets/ can harm your search rankings.

How quickly does Google update its copy of robots.txt?

Google typically caches robots.txt files for up to 24 hours. If you update critical disallow rules, you can request an immediate re-fetch using the robots.txt Tester inside Google Search Console.