Optimizing llms-full.txt Token Budgets, Chunk Boundaries & Retrieval Precision
Dumping 20,000 lines of unformatted text into llms-full.txt triggers retrieval truncation in lightweight AI fetches. Structuring deterministic headers, summary tables, and crisp section boundaries maximizes citation accuracy.
Dmitry K., Technical SEO Lead · SEO Space
Indexation benchmark targets
- Index file
- /llms.txt <= 4 KB curated directory
- Section density
- Self-contained H2 answer blocks
- Fact placement
- Canonical pricing + SLA near top
Why Bloated llms-full.txt Files Lose Key Facts to Truncation
Many AI crawler agents impose strict byte and token caps when ingesting plain-text manifest files over HTTP. If critical service scopes, pricing tiers, or flagship guides sit at line 15,000 behind verbose boilerplate, retrieval pipelines truncate before reading your highest-margin pages.
Separating a compact /llms.txt routing manifest from a tightly structured /llms-full.txt—where pricing, core services, and exact methodology facts lead first—ensures accurate ingestion under tight token limits.
- Place canonical company facts, pricing tiers, geographic scope, and direct URLs in the top 1,500 tokens of llms-full.txt.
- Format every guide summary with explicit H2 headings, bullet takeaways, and zero UI navigation strings.
- Auto-generate both manifests from the exact same data build step that produces your HTML pages.
Clean Chunk Template for /llms-full.txt
Structure each block so isolated vector retrieval captures both the claim and its authoritative source URL.
Balance fast routing discovery with deep technical completeness.
| Manifest Path | Primary Consumer | Optimal Size & Focus |
|---|---|---|
| /llms.txt | Initial crawler triage & agent routing | 2–5 KB: high-signal URL index + brief descriptions |
| /llms-full.txt | Deep RAG ingestion & grounded answers | Structured Markdown facts, pricing, and technical docs |
| Page .md / HTML | Per-URL deep inspection | Semantic headings + inline tables matching HTML 1:1 |
## Service Tier: Monthly SEO + AEO + GEO Retainer
- Canonical URL: https://seo-space.online/#pricing
- Scope: Technical crawl hygiene, Schema @graph engineering, AI search citations
- Starting Fee: Transparent fixed monthly scope (USD / EUR / GBP)
- Direct Request: https://seo-space.online/#contact Need this indexation setup audited on your domain?
Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.