Meta Robots vs. X-Robots-Tag: Directive Precedence & AI Snippet Controls
When an HTTP X-Robots-Tag contradicts HTML meta robots, the stricter restriction always wins. Learn exact directive combinations and data-nosnippet usage.
Dmitry K., Technical SEO Lead · SEO Space
Indexation benchmark targets
- Conflict resolution
- Most restrictive wins
- Non-HTML assets
- X-Robots-Tag header only
- Partial exclusion
- data-nosnippet attribute
How Search Engines Resolve Overlapping Robots Directives
If your CDN edge injects `X-Robots-Tag: noindex` while your React or CMS `<head>` outputs `<meta name="robots" content="index, follow" />`, Google obeys the strictest directive (`noindex`) every time. Always audit inherited proxy headers alongside origin HTML.
Conversely, if `robots.txt` blocks a URL via `Disallow`, crawlers never fetch the response body or HTTP headers—meaning neither `<meta>` nor `X-Robots-Tag: noindex` can take effect cleanly.
location ~* \.(pdf|xlsx)$ {
add_header X-Robots-Tag "noindex, nofollow" always;
} Fine-Grained Snippet Control With data-nosnippet & max-snippet
Use `data-nosnippet` on HTML containers holding boilerplate disclaimers or promo bars so snippets and AI extracts focus cleanly on core engineering proof.
| Directive | Scope | Safe Use Case |
|---|---|---|
| `noindex, follow` | Whole URL | Multi-facet filters & internal search results |
| `max-snippet:-1` | SERP & AI previews | Commercial & editorial pages seeking full citations |
| `data-nosnippet` | Single DOM node | Hide legal disclaimers or dynamic widgets from snippets |
Production QA Checklist Against Unintended Noindex
Run automated curl assertions scanning both raw response headers and HTML `<head>` tags across critical production templates immediately after every deployment.
- Keep `index, follow` explicit or omitted cleanly on revenue landing pages.
- Use `X-Robots-Tag` strictly on PDFs, feeds, staging hosts, and multi-attribute facet routes.
- Monitor Search Console Excluded by noindex reports weekly for sudden spikes.
Need this indexation setup audited on your domain?
Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.