How Search works: from discovery to the result
Understand the stages that move a page from a link on the web to a possible search result, then diagnose the first stage that is actually failing.
What to know before you change anything.
The indexed result describes what Search Console knows from a processed version of a URL. A live test checks the URL as it can be fetched now. A healthy live test is useful release evidence, but it does not prove that the current version is already indexed or will appear for a query.
- Store indexed and live timestamps in different fields.
- Check X-Robots-Tag as well as HTML directives.
- Record declared and selected canonicals separately.
Use this process in the next release.
Keep every observation dated and every change tied to the earliest demonstrated issue.
- 01
Inspect the indexed record
Record the status, last crawl, selected canonical, and any reported reason before changing the page.
- 02
Run the live test
Check the current response after a controlled change. Review headers and rendered output, not only the on-screen summary.
- 03
Name the first mismatch
State the earliest observed failure—such as access, a directive, canonical conflict, or delivery issue—and keep later questions open.
- 04
Verify a bounded release
Ship the smallest reversible change, test a representative cohort, and return to the indexed record only after an appropriate observation window.
A small checklist before you ship.
These checks make the diagnosis repeatable for another teammate.
- Store indexed and live timestamps in different fields.
- Check X-Robots-Tag as well as HTML directives.
- Record declared and selected canonicals separately.
- Do not treat a recrawl request as an indexing promise.
Use the report to narrow the next decision.
Start with the question you need to answer. If the page is unexpectedly absent, look for access, directives, canonical signals, and content delivery before asking for another crawl.
If you ship a fix, use the live test to prove the fix reached production, then allow time to observe the later indexed record.
Continue with the related guide.
Write robots.txt rules that match your crawl policy
robots.txt tells compliant crawlers which paths they may fetch. It is a crawl-management file—not authentication and not a dependable way to remove already discoverable URLs from an index.
Use noindex safely without blocking the evidence you need
A noindex directive tells crawlers not to include a page or resource in their index when they can fetch and process the directive. It is not a security control and it should match the page’s intended public state.
Build an XML sitemap that supports discovery
An XML sitemap is a structured discovery hint. It works best when it contains the clean canonical URLs you want crawlers to find—not every address your application can generate.
Common questions
Bounded answers for the checks that most often get conflated.
Why do the indexed result and live test disagree?
They represent different points in time. The indexed result is based on a processed version, while the live test checks the current URL.
Should I request indexing after every change?
Use requests selectively after a meaningful, shipped repair. They do not replace a technical diagnosis or guarantee an outcome.