Robots.txt Generator
Build robots.txt with crawler shortcuts, Allow and Disallow rules, sitemap URLs and optional Clean-param. Test a path before downloading.
Draft robots.txt Rules With a Public Exception
Crawler shortcuts
Google-Extended and Applebot-Extended are policy tokens for those providers. Set their rules separately from the search crawlers.
Separate crawler tokens with spaces or commas. Enter one path per line. Google ignores Crawl-delay. Yandex retired the Host directive.
Robots.txt Generator: how to use it
- Use a crawler shortcut or enter a product token. Use * for the fallback group and add a group for each different policy.
- Enter one Allow or Disallow path per line and optional absolute sitemap URLs. Enter a crawler and test path to preview the result.
- Generate robots.txt, review the reported decision and download the UTF-8 file. Publish it as /robots.txt on the site it applies to.
Limits and notes
Runs on this device. Builds up to 20 groups and a 500 KiB file. Supports Allow, Disallow, Sitemap, optional Crawl-delay and Yandex Clean-param. The path test follows core exclusion rules. Google ignores Crawl-delay. The tool does not contact your site or publish changes.
How your privacy works →Robots.txt Generator: common questions
Where should I upload robots.txt?
Serve it as /robots.txt at the root of the relevant origin. A different scheme, hostname or port has separate rules.
What does an empty Disallow rule do?
It adds no blocked path. The generator uses an empty Disallow line when a group has no path rules so that a following group stays separate.
Can I use robots.txt to keep documents private?
Use access controls for private files. Crawlers may ignore robots.txt and a blocked URL can still appear in search based on references.
Does Clean-param block a path?
Yandex can use Clean-param to disregard listed query parameters when finding duplicate URLs. The local path test applies Allow and Disallow rules.