Markdown Mirrors & Accept Header Negotiation: Serving Clean Context to AI Agents
Stripping navbar chrome, mega-menus, and modal noise into crisp Markdown cuts token bloat by 80% while preserving strict canonical parity for web search.
Dmitry K., Technical SEO Lead · SEO Space
Indexation benchmark targets
- Header check
- Accept: text/markdown
- Token reduction
- 70-85% smaller payload
- Index safety
- X-Robots-Tag: noindex on .md
Why HTML Boilerplate Wastes RAG Token Budgets
Typical SaaS documentation and product pages ship 120 KB of HTML where 80% consists of navigation drawers, SVG icon paths, footer links, and hydration state scripts. When an AI agent fetches five competing URLs inside a tight context window, noisy DOMs truncate crucial specifications.
Serving clean Markdown either via `Accept: text/markdown` HTTP content negotiation (paired with `Vary: Accept`) or dedicated `.md` URL mirrors linked in `llms.txt` delivers 100% factual signal density in a fraction of the bytes.
export default {
async fetch(request) {
const accept = request.headers.get('Accept') || '';
if (accept.includes('text/markdown')) {
const mdUrl = new URL(request.url.replace(/\/$/, '') + '/index.md');
const res = await fetch(mdUrl);
return new Response(res.body, {
headers: { 'Content-Type': 'text/markdown; charset=utf-8', 'Vary': 'Accept', 'X-Robots-Tag': 'noindex' }
});
}
return fetch(request);
}
}; Preventing Markdown Endpoints from Competing in Organic SERPs
Markdown mirrors are built strictly for AI consumption and RAG pipelines—never let search engines index `.md` endpoints alongside HTML pages.
- Attach `X-Robots-Tag: noindex` and a `<link>` or HTTP `Link: <canonical-html-url>; rel="canonical"` header on every `.md` route.
- Always emit `Vary: Accept` when performing content negotiation at the same URL so CDN caches never serve raw Markdown to human browsers or Googlebot.
- Auto-generate Markdown files from the exact same CMS or TypeScript content source during CI builds to prevent factual drift.
Content Negotiation Implementation Guardrails
Compare standalone `.md` routes versus edge `Accept` header negotiation.
| Approach | Cache & SEO Safety | Best Use Case |
|---|---|---|
| Standalone /slug/index.md + llms.txt | Zero Vary cache collision risk; add X-Robots-Tag: noindex | Documentation hubs & developer portals |
| Accept: text/markdown Negotiation | Requires Vary: Accept on CDN cache key | Unified canonical URLs for agentic fetchers |
| Static HTML -> Readability Stripper | Lower maintenance, moderate token savings | Legacy CMS stacks without Markdown source |
Need this indexation setup audited on your domain?
Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.