AI Search (GEO) Field Note Updated 2026-11-16 3 min read OAI-SearchBot · PerplexityBot · Claude

Robots.txt Governance for AI Bots: Search Citations vs. Model Training Crawlers

Accidentally blocking OAI-SearchBot when trying to restrict training bots hides your pricing and brand recommendations inside ChatGPT Search.

Dmitry K., Technical SEO Lead · SEO Space

Indexation benchmark targets

Live AI search
Allow: / (citations)
Training policy
Optional selective disallow

Differentiating Retrieval Crawlers from Pre-Training Scrapers

OpenAI, Anthropic, and Perplexity operate separate user agents for live search grounding versus base-model training. Keep search-grounding agents explicitly allowed across commercial pages so AI buyers see accurate live URLs and rates.

Blanket blocking every AI user-agent in `robots.txt` often backfires—cutting your brand out of live ChatGPT Search and Perplexity citations even when you only intended to restrict bulk pre-training scraping.

  • Separate live retrieval search bots (`OAI-SearchBot`, `PerplexityBot`) from bulk model training agents (`GPTBot`, `CCBot`) in `robots.txt`.
  • Verify real crawler authenticity via IP ranges/ASN rather than trusting spoofed `User-Agent` headers.
  • Monitor AI bot hit frequency and 403/429 error rates alongside Googlebot in server logs.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Granular robots.txt Blueprint: Allow Citations, Govern Training

Tailor access policies cleanly: permit live search retrieval crawlers across commercial solutions, pricing, and public guides so your brand receives clickable citations in real-time answers, while governing bulk training bots according to your licensing stance.

Live retrieval crawlers time out rapidly when origin servers take longer than 1–2 seconds or emit Cloudflare JS challenges. Serve clean cached SSR HTML under 400ms and monitor security logs weekly so legitimate `OAI-SearchBot` fetches never hit silent blocks.

# Allow live AI search grounding & citations
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Govern bulk training corpus crawling separately
User-agent: GPTBot
Disallow: /private-datasets/

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.