Robots.txt Generator

Your workspaceProcessed on your device
Crawler shortcuts

Google-Extended and Applebot-Extended are policy tokens for those providers. Set their rules separately from the search crawlers.

Crawler group 1

Separate crawler tokens with spaces or commas. Enter one path per line. Google ignores Crawl-delay. Yandex retired the Host directive.

Test a path (optional)

A few things to know.

Runs on this device. Builds up to 20 groups and a 500 KiB file. Supports Allow, Disallow, Sitemap, optional Crawl-delay and Yandex Clean-param. The path test follows core exclusion rules. Google ignores Crawl-delay. The tool does not contact your site or publish changes.

  1. Use a crawler shortcut or enter a product token. Use * for the fallback group and add a group for each different policy.
  2. Enter one Allow or Disallow path per line and optional absolute sitemap URLs. Enter a crawler and test path to preview the result.
  3. Generate robots.txt, review the reported decision and download the UTF-8 file. Publish it as /robots.txt on the site it applies to.