Robots.txt Validator

Your workspaceProcessed on your device
Test a path (optional)

A few things to know.

Checks up to 500 KiB of UTF-8 text. Recognizes core exclusion rules, sitemap declarations and Crawl-delay syntax. The path test is a local simulation based on the parsed rules and exact product token. It cannot check deployed HTTP responses, provider-specific crawler aliases or indexing.

  1. Paste robots.txt or open a local UTF-8 file.
  2. To test a decision, supply a crawler product token and a path or absolute URL. Include the query string when a rule depends on it.
  3. Validate the file. Check the original line numbers and matched rule, then correct the text or download the JSON report.