Crawl & Logs Updated 2026-11-16 3 min read Googlebot · Bingbot · Crawlers

robots.txt Engineering: Wildcards, Specificity Precedence & Crawl-Delay Traps

One misplaced Disallow rule in robots.txt can deindex an entire catalog or blind Google WRS to rendering bundles. Understand RFC 9309 longest-match behavior.

Dmitry K., Technical SEO Lead · SEO Space

1. Longest-Match Precedence Under RFC 9309

Search engines do not evaluate robots.txt sequentially from top to bottom inside a User-agent group. Under RFC 9309, both Googlebot and Bingbot score every matching Allow and Disallow path by character length; the longest matching rule wins regardless of line order, and when lengths tie, Allow overrides Disallow.

Engineers frequently assume placing Disallow: /app/ below Allow: /app/pricing/ will block everything, or vice versa. Testing every wildcard against concrete URLs prevents accidental exclusion of commercial landing pages while still containing infinite parameter traps.

  • Use the dollar anchor ($) strictly for exact file extensions or query-less folder roots.
  • Remember that blocking a URL via robots.txt stops crawling, not indexation—linked URLs can still appear as URL-only SERP listings.
  • Never place Crawl-delay directives expecting Googlebot compliance; Google ignores Crawl-delay and manages rate via Search Console and 429/503 signals.
User-agent: *
Allow: /assets/
Allow: /*.css$
Allow: /*.js$
Disallow: /checkout/
Disallow: /account/
Disallow: /*?sort=
Disallow: /*?filter=
Allow: /*?page=

2. Why Blocking /assets/, APIs & Build Chunks Breaks Rendering

When robots.txt blocks /_next/static/, /wp-content/themes/, or read-only JSON endpoints required to hydrate product prices and navigation, Google Web Rendering Service (WRS) renders a broken layout with missing copy and invisible internal links.

Worst of all, blocked scripts never trigger obvious 404 alerts in server logs—you only discover the loss when core keywords slide because primary text failed client hydration.

Resource Pattern Recommended Directive Impact if Disallowed
/assets/*.css, *.js, *.woff2 Allow WRS fails layout check & mobile usability
/api/v1/public/catalog/* Allow (GET) Client components render empty product skeletons
/search?q=* & /filter?* Disallow Protects crawl budget from infinite combinations
/admin/, /cart/, /checkout/ Disallow + noindex Keeps session & private endpoints out of queue

3. Automated CI Validation & 5xx Fail-Safe Behavior

If your origin returns HTTP 500 or 503 for /robots.txt during a deploy, Googlebot interprets the outage as a complete crawl halt across the entire domain for up to 12 hours. Serve robots.txt directly from CDN edge storage with immutable fallback caching.

Integrate unit tests into your CI pipeline using an RFC 9309 parser against 50+ sample URLs covering product pages, paginated archives, tracking parameters, and checkout routes before every release.

  • Assert HTTP 200 and text/plain; charset=utf-8 header on edge CDN.
  • Keep robots.txt strictly under 500 KiB as required by Google Search.
  • Verify sitemap XML URLs in robots.txt use the exact canonical HTTPS host.

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.