How Search works: from discovery to the result
Understand the stages that move a page from a link on the web to a possible search result, then diagnose the first stage that is actually failing.
What to know before you change anything.
Include canonical, indexable URLs that return the intended public response. Use lastmod only when a meaningful page change occurred, publish the sitemap reliably, submit it in Search Console, and diagnose discrepancies without treating submission as an indexing guarantee.
- Exclude redirects, error pages, and intentionally noindexed URLs.
- Use one canonical representation of each included URL.
- Do not update lastmod for cosmetic or unrelated deployment changes.
Use this process in the next release.
Keep every observation dated and every change tied to the earliest demonstrated issue.
- 01
Define inclusion criteria
Include only the canonical URLs that are intended to be public and considered for indexing.
- 02
Generate reliable entries
Use absolute URLs, a stable format, and lastmod values only when a material update actually happened.
- 03
Publish and submit
Make the sitemap available at a reliable URL, reference it where appropriate, and submit it in Search Console.
- 04
Monitor and reconcile
Review reported URLs against redirects, directives, canonicals, internal links, and the actual response before making changes.
A small checklist before you ship.
These checks make the diagnosis repeatable for another teammate.
- Exclude redirects, error pages, and intentionally noindexed URLs.
- Use one canonical representation of each included URL.
- Do not update lastmod for cosmetic or unrelated deployment changes.
- Retest the feed after a route or CMS migration.
Treat the sitemap as a maintained output of your URL policy.
A sitemap cannot repair hidden pages, broken responses, contradictory canonicals, or thin content. Use it alongside meaningful internal links and a clear public URL policy.
When a sitemap report shows an issue, compare the declared URL with the actual response and the page’s intended state before changing the feed.
Continue with the related guide.
Use URL Inspection without confusing live and indexed states
A disciplined URL Inspection workflow keeps the processed record, the current live response, and the work you still need to observe separate.
Write robots.txt rules that match your crawl policy
robots.txt tells compliant crawlers which paths they may fetch. It is a crawl-management file—not authentication and not a dependable way to remove already discoverable URLs from an index.
Use noindex safely without blocking the evidence you need
A noindex directive tells crawlers not to include a page or resource in their index when they can fetch and process the directive. It is not a security control and it should match the page’s intended public state.
Common questions
Bounded answers for the checks that most often get conflated.
Do I need a sitemap for every site?
A sitemap is particularly useful for larger, newer, or more complex sites, but important pages should still be discoverable through ordinary links.
Will submitting a sitemap index every URL?
No. Submission helps discovery and processing. Each URL still has to meet the other conditions needed for indexing and serving.