Code · Made a little simpler

Robots.txt Generator

Build robots.txt with crawler shortcuts, Allow and Disallow rules, sitemap URLs and optional Clean-param. Test a path before downloading.

Free to useNo account neededProcessed on your device
Your workspaceProcessed on your device
Crawler shortcuts

Google-Extended and Applebot-Extended are policy tokens for those providers. Set their rules separately from the search crawlers.

Crawler group 1

Separate crawler tokens with spaces or commas. Enter one path per line. Google ignores Crawl-delay. Yandex retired the Host directive.

Test a path (optional)

Add this tool to your website

Paste this code into your page to add the tool and a visible ToolMellow credit link.

HTML embed code
Preview embed

Robots.txt Generator: how to use it

  1. Use a crawler shortcut or enter a product token. Use * for the fallback group and add a group for each different policy.
  2. Enter one Allow or Disallow path per line and optional absolute sitemap URLs. Enter a crawler and test path to preview the result.
  3. Generate robots.txt, review the reported decision and download the UTF-8 file. Publish it as /robots.txt on the site it applies to.

Limits and notes

Runs on this device. Builds up to 20 groups and a 500 KiB file. Supports Allow, Disallow, Sitemap, optional Crawl-delay and Yandex Clean-param. The path test follows core exclusion rules. Google ignores Crawl-delay. The tool does not contact your site or publish changes.

How your privacy works →

Robots.txt Generator: common questions

Where should I upload robots.txt?

Serve it as /robots.txt at the root of the relevant origin. A different scheme, hostname or port has separate rules.

What does an empty Disallow rule do?

It adds no blocked path. The generator uses an empty Disallow line when a group has no path rules so that a following group stays separate.

Can I use robots.txt to keep documents private?

Use access controls for private files. Crawlers may ignore robots.txt and a blocked URL can still appear in search based on references.

Does Clean-param block a path?

Yandex can use Clean-param to disregard listed query parameters when finding duplicate URLs. The local path test applies Allow and Disallow rules.

Guides