Search publishing guides

Read Robots Meta Directives From a Saved Page

·

When a search result unexpectedly disappears or loses its snippet, save the page source before changing anything. Review the general robots tag and the tags named for a particular crawler. More than one tag can contribute to the request.

Meta Robots Tag Tester

Read four tags in the saved HTML example

Select Load example and Inspect tags with Googlebot selected. The result lists four recognized tags. Line 3 contains general index, nofollow and max-snippet:50. Line 4 requests noindex, follow and nosnippet for Googlebot. The bingbot tag on line 5 is visible as other crawler evidence.

The Google summary reads Noindex requested, Nofollow requested and No text snippet requested. Google combines restrictions from applicable tags. The general snippet limit does not override the Googlebot nosnippet request.

Google Search Central, robots meta tags and combined directives

Inspect the body tag and switch the selected crawler

Line 7 places a Googlebot image preview size in the body. The report suggests moving it to the head while still showing large in the Google summary. Google documents that it can honor body robots meta tags.

Switch to Googlebot-News and inspect again. This view compares the general tag with tags named googlebot-news. There is no noindex request in that selection, and the general 50-character snippet limit remains visible. Bingbot mode shows its scoped evidence without applying Google’s precedence.

Google Search Central, robots meta tags and combined directivesBing webmaster guidance for robots and indexing directives

Match the report to the page you saved

Download the evidence and check the file’s source type and crawler label. Verify the reported line numbers in the saved page. Check HTTP response headers separately for X-Robots-Tag and review the rendered page if a script supplies tags after load.

A crawler must be able to read a page’s directive to act on it. Check deployed crawl access and the search provider’s inspection tools when you need evidence of the provider’s actual state.

Google Search Central, robots meta tags and combined directivesBing webmaster guidance for robots and indexing directives

Common questions

Why might the same page have several robots meta tags?

A base template, a page template and a provider-specific setting can each contribute a tag. Review them together before changing an individual directive.

Does the tester check unavailable_after dates?

It records an unavailable_after directive with a value. Check your date format and the provider’s interpretation separately.

Sources

Put it into practice.

Related guides