Validate robots.txt

Check robots.txt groups, path rules, sitemap lines, and crawler effects.

freeworks offlinenothing uploaded
Toolrobots.txt Validator
Input
Output
—
Put this on your own site

The frame below runs the same code as this page, in the reader's own browser. Nothing is sent to us, and nothing is sent to you.

Pick a dark background and the text and panels follow it, so the frame stays readable on a dark page.

Preview

How it works

Lines become user-agent groups, allow/disallow paths, crawl-delay extensions, and sitemap declarations; longest matching paths determine the rule for a crawler. The validator reports policy effects without crawling the site.

  • RFC 9309 defines group and longest-match behavior.
  • Crawl-delay remains implementation-specific.

Worked example

Validate robots.txt with sitemap directive
Check a robots.txt file with User-agent, Disallow, and Sitemap directives for syntax errors
Input
											User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Output
												robots.txt is valid!

When to use this

Deployments, SEO audits, and security reviews validate robots policy.

Edge cases

  • Disallow is not access control.
  • A specific Allow can override a broader Disallow.
  • Rules for one user-agent group do not automatically apply to another.

References