Crawl & Logs Updated 2026-11-16 3 min read Google · Bing · AI Snippets

Meta Robots vs. X-Robots-Tag: Directive Precedence & AI Snippet Controls

When an HTTP X-Robots-Tag contradicts HTML meta robots, the stricter restriction always wins. Learn exact directive combinations and data-nosnippet usage.

Dmitry K., Technical SEO Lead · SEO Space

Indexation benchmark targets

Conflict resolution
Most restrictive wins
Non-HTML assets
X-Robots-Tag header only
Partial exclusion
data-nosnippet attribute

How Search Engines Resolve Overlapping Robots Directives

If your CDN edge injects `X-Robots-Tag: noindex` while your React or CMS `<head>` outputs `<meta name="robots" content="index, follow" />`, Google obeys the strictest directive (`noindex`) every time. Always audit inherited proxy headers alongside origin HTML.

Conversely, if `robots.txt` blocks a URL via `Disallow`, crawlers never fetch the response body or HTTP headers—meaning neither `<meta>` nor `X-Robots-Tag: noindex` can take effect cleanly.

location ~* \.(pdf|xlsx)$ {
  add_header X-Robots-Tag "noindex, nofollow" always;
}

Fine-Grained Snippet Control With data-nosnippet & max-snippet

Use `data-nosnippet` on HTML containers holding boilerplate disclaimers or promo bars so snippets and AI extracts focus cleanly on core engineering proof.

Directive Scope Safe Use Case
`noindex, follow` Whole URL Multi-facet filters & internal search results
`max-snippet:-1` SERP & AI previews Commercial & editorial pages seeking full citations
`data-nosnippet` Single DOM node Hide legal disclaimers or dynamic widgets from snippets

Production QA Checklist Against Unintended Noindex

Run automated curl assertions scanning both raw response headers and HTML `<head>` tags across critical production templates immediately after every deployment.

  • Keep `index, follow` explicit or omitted cleanly on revenue landing pages.
  • Use `X-Robots-Tag` strictly on PDFs, feeds, staging hosts, and multi-attribute facet routes.
  • Monitor Search Console Excluded by noindex reports weekly for sudden spikes.

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.