AI Crawling Governance: robots.txt, Content-Usage Signals & Licensing Rules
Modern publishers separate live search citation bots from foundation-model training scrapers using explicit robots directives and HTTP usage headers.
Dmitry K., Technical SEO Lead · SEO Space
Indexation benchmark targets
- Bot separation
- Search vs Training agents
- HTTP signaling
- Content-Signal / TDMRep
- Legal sync
- Terms + machine policy match
Moving Beyond Binary "Allow All" or "Block Everything" Rules
Bluntly blocking every AI user-agent in `robots.txt` sacrifices real-time brand citations in ChatGPT Search, Perplexity, and Claude while leaving legacy training snapshots untouched. Conversely, allowing unrestricted scraping of proprietary research databases erodes licensing leverage.
Separating live citation indexers (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`) from bulk pre-training bots (`GPTBot`, `CCBot`, `Google-Extended`) paired with machine-readable HTTP headers gives legal and growth teams precise control.
# Allow live answer-engine search citations
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Restrict bulk pre-training ingestion on proprietary datasets
User-agent: GPTBot
Disallow: /proprietary-datasets/
User-agent: Google-Extended
Disallow: /proprietary-datasets/ Aligning Edge HTTP Headers with Public Crawler Policies
Standardizing machine-readable preferences across `robots.txt`, HTTP `X-Robots-Tag` / `Content-Usage` directives, and `.well-known/tdmrep.json` documents reduces compliance ambiguity across jurisdictions.
Define distinct access tiers across public marketing pages, documentation, and licensed datasets.
| Site Section | Search Bots (Google / OAI-Search) | Training Bots (GPTBot / CCBot) |
|---|---|---|
| Product, Pricing & Solutions | Allow (full SSR HTML) | Allow or governed by brand reach policy |
| Public Docs & Technical Guides | Allow (plus /llms.txt) | Allow for developer ecosystem familiarity |
| Paid / Proprietary Research Feeds | Summary preview allowed; raw DB restricted | Disallow + TDM reservation headers |
- Never apply blanket `noai` or `noimageai` meta tags across commercial landing pages unless your explicit goal is total exclusion from visual and generative summaries.
- Audit `robots.txt` updates quarterly against official crawler documentation—vendors introduce new specialized search bots frequently.
- Verify that CDN bot-mitigation rules honor your `robots.txt` allowlist rather than returning hidden `403` challenges.
Need this indexation setup audited on your domain?
Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.