Code · Made a little simpler

Robots.txt Validator

Check saved robots.txt syntax and sitemap declarations. Test a crawler product token against a path and see the matched rule and line number.

Free to useNo account neededProcessed on your device
Your workspaceProcessed on your device
Test a path (optional)

Add this tool to your website

Paste this code into your page to add the tool and a visible ToolMellow credit link.

HTML embed code
Preview embed

Robots.txt Validator: how to use it

  1. Paste robots.txt or open a local UTF-8 file.
  2. To test a decision, supply a crawler product token and a path or absolute URL. Include the query string when a rule depends on it.
  3. Validate the file. Check the original line numbers and matched rule, then correct the text or download the JSON report.

Limits and notes

Checks up to 500 KiB of UTF-8 text. Recognizes core exclusion rules, sitemap declarations and Crawl-delay syntax. The path test is a local simulation based on the parsed rules and exact product token. It cannot check deployed HTTP responses, provider-specific crawler aliases or indexing.

How your privacy works →

Robots.txt Validator: common questions

What happens when Allow and Disallow both match?

The longer normalized rule wins in this test. Allow wins a tie. The result names the matched line so you can check its scope.

When is the wildcard group used?

The wildcard group is the fallback when no group matches the crawler token. Multiple groups with the matching token are considered together.

Does the validator fetch the supplied URL?

It extracts the path and query string locally. It never opens the host or reads the deployed robots.txt file.

Guides