AI Search (GEO) Field Note Updated 2026-11-16 3 min read robots.txt · TDMRep · Content-Usage · AI Licensing

AI Crawling Governance: robots.txt, Content-Usage Signals & Licensing Rules

Modern publishers separate live search citation bots from foundation-model training scrapers using explicit robots directives and HTTP usage headers.

Dmitry K., Technical SEO Lead · SEO Space

Indexation benchmark targets

Bot separation
Search vs Training agents
HTTP signaling
Content-Signal / TDMRep
Legal sync
Terms + machine policy match

Moving Beyond Binary "Allow All" or "Block Everything" Rules

Bluntly blocking every AI user-agent in `robots.txt` sacrifices real-time brand citations in ChatGPT Search, Perplexity, and Claude while leaving legacy training snapshots untouched. Conversely, allowing unrestricted scraping of proprietary research databases erodes licensing leverage.

Separating live citation indexers (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`) from bulk pre-training bots (`GPTBot`, `CCBot`, `Google-Extended`) paired with machine-readable HTTP headers gives legal and growth teams precise control.

# Allow live answer-engine search citations
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Restrict bulk pre-training ingestion on proprietary datasets
User-agent: GPTBot
Disallow: /proprietary-datasets/

User-agent: Google-Extended
Disallow: /proprietary-datasets/

Aligning Edge HTTP Headers with Public Crawler Policies

Standardizing machine-readable preferences across `robots.txt`, HTTP `X-Robots-Tag` / `Content-Usage` directives, and `.well-known/tdmrep.json` documents reduces compliance ambiguity across jurisdictions.

Define distinct access tiers across public marketing pages, documentation, and licensed datasets.

Site Section Search Bots (Google / OAI-Search) Training Bots (GPTBot / CCBot)
Product, Pricing & Solutions Allow (full SSR HTML) Allow or governed by brand reach policy
Public Docs & Technical Guides Allow (plus /llms.txt) Allow for developer ecosystem familiarity
Paid / Proprietary Research Feeds Summary preview allowed; raw DB restricted Disallow + TDM reservation headers
  • Never apply blanket `noai` or `noimageai` meta tags across commercial landing pages unless your explicit goal is total exclusion from visual and generative summaries.
  • Audit `robots.txt` updates quarterly against official crawler documentation—vendors introduce new specialized search bots frequently.
  • Verify that CDN bot-mitigation rules honor your `robots.txt` allowlist rather than returning hidden `403` challenges.

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.