AI Search (GEO) Engineering Playbook Updated 2026-11-16 3 min read OAI-SearchBot · GPTBot · PerplexityBot · ClaudeBot

AI Crawler Log Analysis: Parsing OAI-SearchBot, PerplexityBot & ClaudeBot Traffic

Treating all AI bots identically in WAF rules hides 403 blocks against live search fetchers and masks which URLs drive real-time RAG lookups.

Dmitry K., Technical SEO Lead · SEO Space

Indexation benchmark targets

User-agent split
Search vs Training bots
IP verification
ASN / signed headers
Status target
<1% 403/429 on search bots

Separating Live RAG Fetchers from Bulk Model Scrapers

Modern CDN and Nginx access logs mix three distinct AI traffic classes: bulk training crawlers (`GPTBot`, `CCBot`, `Google-Extended`), automated live search indexers (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`), and synchronous user-triggered fetchers (`ChatGPT-User`, `Perplexity-User`).

Blocking or rate-limiting synchronous user-triggered fetchers with aggressive Bot Fight Mode rules directly deletes your domain from live conversational answers—even if Googlebot passes cleanly.

awk -F'"' '$6 ~ /(OAI-SearchBot|ChatGPT-User|PerplexityBot|Claude-SearchBot)/ {
  print $2, $3, $6
}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 40

Detecting Silent WAF Throttling and Bot Spoofing

Verify official bot IP ranges against published JSON allowlists from OpenAI, Anthropic, and Perplexity before relaxing rate limits at the edge.

  • Alert whenever HTTP `403`, `429`, or `5xx` share exceeds `1%` for verified `OAI-SearchBot` or `PerplexityBot` requests.
  • Track latency percentiles (p95 TTFB) separately for user-triggered AI fetchers—slow origins time out mid-answer generation.
  • Correlate `ChatGPT-User` fetch spikes with UTM-less direct conversions and branded search lift.

AI Bot Classification & Telemetry Matrix

Classify user-agents in your BigQuery or ClickHouse log pipeline for clear executive reporting.

Crawler Class Example User-Agents Recommended Edge Policy
Live Citation Indexers OAI-SearchBot, PerplexityBot, Claude-SearchBot Allow + fast CDN HTML cache
User-Triggered Fetchers ChatGPT-User, Perplexity-User, Claude-User Zero-challenge fast bypass (<300ms)
Training Crawlers GPTBot, ClaudeBot, CCBot Governed by IP licensing strategy

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.