Sitemaps for AI Search: lastmod, Sitemap Index and the Errors That Break Discovery

At minimum, a sitemap is a list of your real, live, canonical URLs — pages that return a normal 200 status, not redirects, not pages blocked by robots.txt or a noindex tag. A sitemap that lists a URL your own robots.txt disallows sends a crawler a contradictory signal and should be trimmed to match.

Beyond the URL list, the field that actually earns its place is `lastmod`: a per-URL date stating when that page's main content last changed. A crawler with limited budget to spend on your site uses lastmod to decide which pages are worth re-reading sooner rather than crawling everything on a fixed, wasteful schedule.

lastmod should reflect a real, verifiable change to the page's main content, structured data, or links — not a cosmetic edit like a copyright-year footer update, and not a timestamp that silently updates on every deploy regardless of whether that specific page changed.

Google is explicit that it only trusts a lastmod value it can verify is accurate and consistent with a page's real history; a sitemap where every single URL carries today's date, every single day, teaches a crawler to stop trusting the field at all — which defeats the entire reason to include it.

Add a line anywhere in robots.txt in the form `Sitemap: https://example.com/sitemap.xml` — it is a standalone directive, not tied to any `User-agent` group, so every crawler that reads robots.txt at all (search engine or AI bot) can find your sitemap from the same file. See AI crawlers and robots.txt for the rest of what belongs in that file.

Submitting a sitemap URL directly to a platform (where that option exists) is a useful addition, not a replacement — the robots.txt line is the one path every crawler, including ones you have never registered with, can discover entirely on its own, with no account or submission step required on your part.

Once a site approaches 50,000 URLs or 50MB uncompressed in a single sitemap file — the hard limit every format Google supports enforces — split the URLs across multiple sitemap files and list them in one sitemap index file, which is itself just a small XML file pointing at the others.

Splitting by logical section (products, blog posts, industry pages) rather than an arbitrary alphabetical cut makes the index easier to audit later, and lets you spot immediately which section of the site is growing the fastest over time.

On most platforms, no. WordPress with Yoast SEO or RankMath auto-generates a sitemap index with lastmod dates the moment either plugin is active — the usual failure mode is the sitemap setting getting toggled off, not a missing feature. Shopify auto-generates and maintains `/sitemap.xml` for every store with no configuration at all, including lastmod values it manages itself.

Static-site generators and most modern frameworks (Next.js, Hugo, Jekyll) can generate a sitemap at build time from the page list, which keeps lastmod tied to real deploys rather than a guess. A hand-maintained sitemap is really only common on older, custom-built sites — and it is also where lastmod most often goes stale, since nothing regenerates it automatically when content changes.