Generate robots.txt

Generate robots.txt groups with crawler rules and sitemap declarations.

freeworks offlinenothing uploaded
Toolrobots.txt Generator
Input
Output
—

How it works

User-agent groups, allow/disallow paths, crawl-delay extensions, and sitemap URLs become robots.txt lines separated into groups. The output controls compliant crawler behavior but does not authenticate requests.

  • User-agent * covers broad crawler scope.
  • Sitemap lines expose discoverable URLs.

Worked example

Standard robots.txt blocking admin and API paths
Generate a robots.txt that allows crawling but blocks admin, private, and API directories
Input
											Sitemap URL: https://example.com/sitemap.xml
Allow all: true
Disallow paths: /admin/
/private/
/api/
Allow paths: 
Crawl delay: 0
Additional bots:
Output
												User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

When to use this

Deployments, staging SEO workflows, and sitemap publishing generate robots files.

Edge cases

  • Disallow does not protect an endpoint from direct requests.
  • A path typo can block unintended content.
  • A group without User-agent has no defined scope.

References