Robots.txt Validator
Check saved robots.txt syntax and sitemap declarations. Test a crawler product token against a path and see the matched rule and line number.
Find the robots.txt Rule That Controls a URL
Robots.txt Validator: how to use it
- Paste robots.txt or open a local UTF-8 file.
- To test a decision, supply a crawler product token and a path or absolute URL. Include the query string when a rule depends on it.
- Validate the file. Check the original line numbers and matched rule, then correct the text or download the JSON report.
Limits and notes
Checks up to 500 KiB of UTF-8 text. Recognizes core exclusion rules, sitemap declarations and Crawl-delay syntax. The path test is a local simulation based on the parsed rules and exact product token. It cannot check deployed HTTP responses, provider-specific crawler aliases or indexing.
How your privacy works →Robots.txt Validator: common questions
What happens when Allow and Disallow both match?
The longer normalized rule wins in this test. Allow wins a tie. The result names the matched line so you can check its scope.
When is the wildcard group used?
The wildcard group is the fallback when no group matches the crawler token. Multiple groups with the matching token are considered together.
Does the validator fetch the supplied URL?
It extracts the path and query string locally. It never opens the host or reads the deployed robots.txt file.