AI Search (GEO) Updated 2026-11-16 3 min read LLMs · Edge Workers · Content Negotiation

Markdown Mirrors & Accept Header Negotiation: Serving Clean Context to AI Agents

Stripping navbar chrome, mega-menus, and modal noise into crisp Markdown cuts token bloat by 80% while preserving strict canonical parity for web search.

Dmitry K., Technical SEO Lead · SEO Space

Indexation benchmark targets

Header check
Accept: text/markdown
Token reduction
70-85% smaller payload
Index safety
X-Robots-Tag: noindex on .md

Why HTML Boilerplate Wastes RAG Token Budgets

Typical SaaS documentation and product pages ship 120 KB of HTML where 80% consists of navigation drawers, SVG icon paths, footer links, and hydration state scripts. When an AI agent fetches five competing URLs inside a tight context window, noisy DOMs truncate crucial specifications.

Serving clean Markdown either via `Accept: text/markdown` HTTP content negotiation (paired with `Vary: Accept`) or dedicated `.md` URL mirrors linked in `llms.txt` delivers 100% factual signal density in a fraction of the bytes.

export default {
  async fetch(request) {
    const accept = request.headers.get('Accept') || '';
    if (accept.includes('text/markdown')) {
      const mdUrl = new URL(request.url.replace(/\/$/, '') + '/index.md');
      const res = await fetch(mdUrl);
      return new Response(res.body, {
        headers: { 'Content-Type': 'text/markdown; charset=utf-8', 'Vary': 'Accept', 'X-Robots-Tag': 'noindex' }
      });
    }
    return fetch(request);
  }
};

Preventing Markdown Endpoints from Competing in Organic SERPs

Markdown mirrors are built strictly for AI consumption and RAG pipelines—never let search engines index `.md` endpoints alongside HTML pages.

  • Attach `X-Robots-Tag: noindex` and a `<link>` or HTTP `Link: <canonical-html-url>; rel="canonical"` header on every `.md` route.
  • Always emit `Vary: Accept` when performing content negotiation at the same URL so CDN caches never serve raw Markdown to human browsers or Googlebot.
  • Auto-generate Markdown files from the exact same CMS or TypeScript content source during CI builds to prevent factual drift.

Content Negotiation Implementation Guardrails

Compare standalone `.md` routes versus edge `Accept` header negotiation.

Approach Cache & SEO Safety Best Use Case
Standalone /slug/index.md + llms.txt Zero Vary cache collision risk; add X-Robots-Tag: noindex Documentation hubs & developer portals
Accept: text/markdown Negotiation Requires Vary: Accept on CDN cache key Unified canonical URLs for agentic fetchers
Static HTML -> Readability Stripper Lower maintenance, moderate token savings Legacy CMS stacks without Markdown source

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.