How Search works: from discovery to the result
Understand the stages that move a page from a link on the web to a possible search result, then diagnose the first stage that is actually failing.
SITE DISCOVERY
Build an XML sitemap that supports discovery
An XML sitemap is a structured discovery hint. It works best when it contains the clean canonical URLs you want crawlers to find—not every address your application can generate.
Start for free- Practical release checks
- Clear limits
- Built for operators
THE SHORT ANSWER
What to know before you change anything.
Include canonical, indexable URLs that return the intended public response. Use lastmod only when a meaningful page change occurred, publish the sitemap reliably, submit it in Search Console, and diagnose discrepancies without treating submission as an indexing guarantee.
- Exclude redirects, error pages, and intentionally noindexed URLs.
- Use one canonical representation of each included URL.
- Do not update lastmod for cosmetic or unrelated deployment changes.
PRACTICAL WORKFLOW
Use this process in the next release.
Keep every observation dated and every change tied to the earliest demonstrated issue.
- 01
Define inclusion criteria
Include only the canonical URLs that are intended to be public and considered for indexing.
- 02
Generate reliable entries
Use absolute URLs, a stable format, and lastmod values only when a material update actually happened.
- 03
Publish and submit
Make the sitemap available at a reliable URL, reference it where appropriate, and submit it in Search Console.
- 04
Monitor and reconcile
Review reported URLs against redirects, directives, canonicals, internal links, and the actual response before making changes.
RELEASE CHECKS
A small checklist before you ship.
These checks make the diagnosis repeatable for another teammate.
- Exclude redirects, error pages, and intentionally noindexed URLs.
- Use one canonical representation of each included URL.
- Do not update lastmod for cosmetic or unrelated deployment changes.
- Retest the feed after a route or CMS migration.
COPYABLE TEMPLATE
Capture the evidence in one place.
Copy this into a release ticket or investigation record, then complete it for a representative URL.
Sitemap URL: [address]
Included URL rule: [canonical and indexable criteria]
Excluded URL rule: [redirects, noindex, duplicates, errors]
lastmod source: [meaningful change signal]
Validation sample: [URLs]
Search Console submission: [date]
Owner and monitoring: [details]WHY THIS MATTERS
Treat the sitemap as a maintained output of your URL policy.
A sitemap cannot repair hidden pages, broken responses, contradictory canonicals, or thin content. Use it alongside meaningful internal links and a clear public URL policy. When a sitemap report shows an issue, compare the declared URL with the actual response and the page’s intended state before changing the feed.
KEEP LEARNING
Continue with the related guide.
Use URL Inspection without confusing live and indexed states
A disciplined URL Inspection workflow keeps the processed record, the current live response, and the work you still need to observe separate.
Write robots.txt rules that match your crawl policy
robots.txt tells compliant crawlers which paths they may fetch. It is a crawl-management file—not authentication and not a dependable way to remove already discoverable URLs from an index.
Use noindex safely without blocking the evidence you need
A noindex directive tells crawlers not to include a page or resource in their index when they can fetch and process the directive. It is not a security control and it should match the page’s intended public state.
Frequently asked questions
Common questions
Bounded answers for the checks that most often get conflated.
Do I need a sitemap for every site?
A sitemap is particularly useful for larger, newer, or more complex sites, but important pages should still be discoverable through ordinary links.
Will submitting a sitemap index every URL?
No. Submission helps discovery and processing. Each URL still has to meet the other conditions needed for indexing and serving.
PUT THE WORKFLOW TO USE
Turn technical SEO checks into a repeatable workflow.
Use Spacebrain to organize the work, the evidence, and the next owner around every release.
Start for free