Search crawling guides

Allow a public PDF inside a blocked robots.txt directory

·

A product catalog has an old preview directory with a single public sample. Build separate crawl groups, then compare the same path against a named crawler and the wildcard fallback before you deploy the file.

Robots.txt Generator

Build a fallback group and a named group

Choose Load example. The wildcard group blocks paths beginning with /preview/ and allows the exact file /preview/sample.pdf. Googlebot gets a separate group that blocks /calendar/. The Sitemap line points to the catalog host’s sitemap.

Generate robots.txt. The ExampleBot test for /preview/sample.pdf reports that line 3 allows it. Change the path to /preview/sample.pdf?draft=1 and generate again. Line 2 wins because the query string now follows the filename and the $ marker requires the end of the path and query.

The first group uses an exact public-file exception
User-agent: *
Disallow: /preview/
Allow: /preview/sample.pdf$

User-agent: Googlebot
Disallow: /calendar/

Sitemap: https://catalog.example/sitemap.xml

RFC 9309, Robots Exclusion ProtocolGoogle interpretation of robots.txt

Compare the named crawler with the fallback

Keep the query in the test path and change the token to Googlebot. Generate again. The result is allowed because the named group contains only a /calendar/ rule. Add /preview/ to the Googlebot group if that restriction also applies to it.

Use a separate policy for each crawler role you intend to control. OpenAI documents OAI-SearchBot for search and GPTBot for training. Anthropic likewise distinguishes Claude-SearchBot from ClaudeBot. Check each provider’s definitions before using the suggestions.

Google interpretation of robots.txtOverview of OpenAI crawlersAnthropic crawlers and site preferences

Add a sitemap and a separate training-crawler policy

Reset the draft. Enter https://catalog.example/store/ as the Site URL and choose Add site sitemap. The generated line is Sitemap: https://catalog.example/sitemap.xml. Edit the draft if your sitemap lives somewhere else.

Open Crawler shortcuts and add a blocked GPTBot group and an open OAI-SearchBot group. Enter ref /store/ as a Yandex Clean-param rule if that query parameter leaves store pages unchanged. Generate the file and test /store/item for GPTBot and OAI-SearchBot separately. The first has Disallow: / and the second has an empty Disallow line. Clean-param does not affect either test.

RFC 9309, Robots Exclusion ProtocolOverview of OpenAI crawlersYandex Clean-param directive syntax and examples

Check the saved file and its deployment location

Download robots.txt and open it in a text editor. Check the relevant hostname and path, then serve it from /robots.txt on that origin. Test the live response separately after deployment.

A Crawl-delay value is an optional crawler extension. Google ignores it. Bing can honor it, and Bing Webmaster Tools provides a crawl-control setting. Use server access controls for confidential destinations.

RFC 9309, Robots Exclusion ProtocolGoogle interpretation of robots.txtBing Webmaster Tools, Crawl Control

Common questions

Where does the sitemap shortcut put sitemap.xml?

It uses /sitemap.xml on the scheme, host and port you supply. The editable Sitemap URLs field lets you enter another path or additional sitemaps.

Does a sitemap declaration allow blocked paths?

No. A sitemap can point crawlers toward pages. Applicable path rules still govern whether crawlers that honor them retrieve each URL.

Do incoming links stop working for a disallowed URL?

The directive asks compliant crawlers to avoid retrieval. Browser visitors and links can still reach the URL if the server grants access.

Sources

Put it into practice.

Related guides