Robots.txt Generator
Your workspaceProcessed on your device
Crawler shortcuts
Google-Extended and Applebot-Extended are policy tokens for those providers. Set their rules separately from the search crawlers.
Separate crawler tokens with spaces or commas. Enter one path per line. Google ignores Crawl-delay. Yandex retired the Host directive.