Crawl & Logs Technical Teardown Updated 2026-11-16 3 min read Google Search · Discover · Bing

XML Sitemap Sharding, Reliable <lastmod> & Image/Video Discovery Feeds

Resetting every <lastmod> date on each deploy trains search engines to ignore your sitemap completely. Segment sitemaps by entity and track real deltas.

Dmitry K., Technical SEO Lead · SEO Space

Indexation benchmark targets

Max per shard
< 10,000 URLs
Only canonical URLs
100% HTTP 200
Lastmod accuracy
Content-hash driven

Why Entity-Sharded Sitemaps Expose Indexation Bottlenecks Instantly

Bundling pricing pages, blog posts, city pages, and 40,000 SKUs inside a single /sitemap.xml makes Search Console coverage diagnostics blind. When you split /sitemap_index.xml into /sitemap-core.xml, /sitemap-solutions.xml, /sitemap-locations.xml, and /sitemap-guides.xml, you can immediately see which template family dropped in Indexed %.

Keep shards under 10,000 URLs and strictly exclude URLs returning 301 redirects, canonicals pointing elsewhere, or noindex directives.

Automating Honest <lastmod> Without Fake Freshness

Google relies on <lastmod> to prioritize recrawls only when timestamps reflect meaningful changes to visible body copy, pricing, or structured specs. Store a content digest hash per URL and update <lastmod> solely when the hash changes.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://seo-space.online/sitemap-solutions.xml</loc>
    <lastmod>2026-11-15T08:00:00Z</lastmod>
  </sitemap>
</sitemapindex>

Validating Sitemap Health in CI & Adding Media Discovery Extensions

Add a build-time verification check rejecting any `<loc>` that omits HTTPS, returns non-200 HTTP codes, carries `noindex`, or points to another canonical. When search bots consistently find 100% clean indexable URLs inside every shard, newly published pages enter the index faster.

For visual tutorials, interactive tools, or high-SKU galleries, supplement standard URL shards with explicit `<image:image>` or `<video:video>` namespaces containing high-resolution canonical CDN URLs and concise descriptive captions.

  • Split locale sitemaps per market (`/sitemap-en.xml`, `/sitemap-de.xml`, `/sitemap-es.xml`).
  • Compute `<lastmod>` exclusively from content, pricing, or structured-data digest changes.
  • Keep total compressed file size well under Search Console limits (< 10,000 URLs/shard).

Need this indexation setup audited on your domain?

Send your website URL. We check raw HTML parity, log crawl hits, canonical discipline, and sitemap architecture manually within 24 hours.