Search crawling guides

Find why a robots.txt exception misses a URL with a query

·

When a clean product PDF loads but a tracked link to the same file is blocked, check the complete path and query. Test both against the same saved robots.txt snapshot and use the reported line to isolate the difference.

Robots.txt Validator

Compare a tracked PDF URL with the clean path

Choose Load example and Validate robots.txt. The default test targets /preview/sample.pdf?draft=1 for ExampleBot. The decision points to line 2, the blocked directory, because the Allow rule ends with $.

Remove ?draft=1 from Test URL or path and validate again. Line 3 now allows the file. If the query variant also needs to be crawlable, test a deliberate rule for that variant before changing the saved file. Removing $ makes the rule a prefix and may allow more URLs than intended.

One blocked prefix and one end-anchored exception
User-agent: *
Disallow: /preview/
Allow: /preview/sample.pdf$

RFC 9309, syntax and path matching

Separate malformed lines from a matched rule

Add a fourth line with Disallow: preview/sample.pdf and validate. The report flags the missing leading slash. The path test uses the supported rules it could parse, so correct reported errors before relying on the decision.

A Crawl-delay field can have valid syntax and still be ignored by some crawlers. Check the provider’s interpretation separately. Download the JSON report when you need to keep a copy of the findings and their original line numbers.

RFC 9309, syntax and path matchingGoogle interpretation of robots.txt

Check deployment when the saved file differs from production

Verify the exact scheme, hostname and port of the affected URL. They determine which /robots.txt endpoint applies. Check the current response, timestamps and any redirects using your site owner tools.

A saved-file decision only reflects the input. The deployed response, crawler cache, provider aliases and index status require separate checks.

Google interpretation of robots.txtRFC 9309, syntax and path matching

Common questions

Does URL case affect the test?

Path matching is case-sensitive. Crawler product tokens are matched case-insensitively.

Does a block remove a result already indexed?

Blocking crawling does not guarantee removal. A search engine can show a blocked URL based on links even when it cannot retrieve the page.

Sources

Put it into practice.

Related guides