About robots.txt Validator

Validate robots.txt syntax and check for common issues. Part of the DevTools Surf developer suite. Browse more tools in the Web / Frontend collection.

Use Cases

• Validate robots.txt before deploying a new website
• Debug why search engines are not indexing certain pages
• Audit crawler access rules during an SEO migration
• Ensure staging environments block search engine indexing

Tips

• Check for syntax errors that could block search engine crawlers
• Validate that your sitemap directive URL is correct
• Test specific user-agent rules against known crawlers

Fun Facts

• The robots.txt standard was proposed by Martijn Koster in June 1994 and has never become an official RFC, yet every major search engine respects it.
• Google officially documented its robots.txt parser as open source in 2019 (github.com/google/robotstxt), revealing how Googlebot interprets the spec.
• A missing robots.txt file returns a 404 to crawlers, which most search engines interpret as 'crawl everything' rather than 'crawl nothing.'

FAQ

What does it check?

Syntax (User-agent groupings, Allow/Disallow rules), common typos (lowercase user-agent, missing slash), and Sitemap directive placement.

Does it verify my URLs are blocked correctly?

Yes — enter a URL and the tool tells you which rule applies to each user agent. Useful for diagnosing why Google can/can't crawl a page.

Is Disallow case-sensitive?

Yes — `/admin/` and `/Admin/` are different paths. Google treats URLs as case-sensitive (except host). Check your actual URLs.

What about X-Robots-Tag?

Out of scope — that's an HTTP header. Use the HTTP Header Analyzer. robots.txt is for crawl directives; X-Robots-Tag is for index/snippet directives.