Find the Deciding Robots.txt Line for a Path
Admin ·
When a robots.txt file has both Allow and Disallow for related paths, check the crawler group first. Then see which path rule is the most specific match for the exact target you care about.
Robots TXT Rule Conflict Explainer
Trace two Googlebot groups for a preview file
Select Load example and Explain rules. The crawler is Googlebot and the compared path is /preview/sample.pdf. The Googlebot groups beginning on lines 4 and 7 are selected together. The wildcard group on line 1 is shown as not selected.
Line 5 is a shorter match. Lines 8 and 9 match the same path with the same normalized length. The Allow on line 9 decides. Download the trace to compare the deciding row with the original file.
RFC 9309, crawler product tokens and path rule matchingGoogle robots.txt interpretation and rule precedence
Add a query and see the deciding line change
Set the target to /preview/sample.pdf?draft=1 and explain again. The dollar sign at the end of the two file rules on lines 8 and 9 prevents those rules from matching the new suffix. Line 5 now disallows the path.
Return to /preview/sample.pdf and change the crawler to ExampleBot. There is no exact ExampleBot group, so line 1 applies as the wildcard group and its Disallow on line 2 decides.
RFC 9309, crawler product tokens and path rule matchingGoogle robots.txt interpretation and rule precedence
Check what evidence the trace can establish
The result ranks matching normalized rule paths by length. The asterisk matches a span of text and a terminal dollar sign anchors the end. Review parse findings when a source directive is malformed or outside the matcher’s scope.
Use the robots.txt file you saved from the relevant origin and confirm its actual response separately. The local trace can parse a supplied full URL, but it uses only the path and query. Search indexing and server behavior require separate evidence.
RFC 9309, crawler product tokens and path rule matchingGoogle robots.txt interpretation and rule precedence
Common questions
Is robots.txt rule order the deciding factor?
The most specific match controls this simulation. When equivalent rows make the same decision, the trace can display the first row for reference while the effective decision stays the same.
How is /robots.txt itself treated?
RFC 9309 treats that path as implicitly allowed. The trace explains the exception without ranking ordinary rules for that path.