CRAWL CONTROL

Write robots.txt rules that match your crawl policy

robots.txt tells compliant crawlers which paths they may fetch. It is a crawl-management file—not authentication and not a dependable way to remove already discoverable URLs from an index.

Start for free
  • Practical release checks
  • Clear limits
  • Built for operators

THE SHORT ANSWER

What to know before you change anything.

Use robots.txt to express an intentional crawl policy at the site root. Do not use it to protect sensitive content, and do not block a URL when a crawler must fetch it to see a new noindex directive.

  • Use authentication for private information.
  • Do not pair a blocked URL with a noindex directive you expect the crawler to read.
  • Test public CSS, JavaScript, and critical page paths.

PRACTICAL WORKFLOW

Use this process in the next release.

Keep every observation dated and every change tied to the earliest demonstrated issue.

  1. 01

    Define the desired state

    Decide whether each path is public, private, useful only in a session, or intentionally unavailable before writing a rule.

  2. 02

    Write minimal path rules

    Target only the paths that match the crawl policy. Avoid broad rules that accidentally hide public assets or pages.

  3. 03

    Test representative URLs

    Check expected allow and disallow cases, including query patterns and critical resources used to render public pages.

  4. 04

    Deploy and observe

    Publish the file at the site root, retain the previous version, and monitor the affected paths after release.

RELEASE CHECKS

A small checklist before you ship.

These checks make the diagnosis repeatable for another teammate.

  • Use authentication for private information.
  • Do not pair a blocked URL with a noindex directive you expect the crawler to read.
  • Test public CSS, JavaScript, and critical page paths.
  • Keep a dated record of the policy and release owner.

COPYABLE TEMPLATE

Capture the evidence in one place.

Copy this into a release ticket or investigation record, then complete it for a representative URL.

Goal: [what should or should not be fetched]
Affected paths: [examples]
User agent: [agent or *]
Allowed public pages: [examples]
Blocked crawl paths: [examples]
Test URLs: [allow and disallow cases]
Owner and rollback: [details]

WHY THIS MATTERS

Separate privacy, crawling, and indexing decisions.

A private page needs real access control. A page you want removed from search needs an appropriate removal or indexing-control path that can be observed by a crawler when required. Put robots.txt in the smaller job it can reliably do: limiting fetches from paths that should not consume crawl activity.

KEEP LEARNING

Continue with the related guide.

Build an XML sitemap that supports discovery

An XML sitemap is a structured discovery hint. It works best when it contains the clean canonical URLs you want crawlers to find—not every address your application can generate.

Frequently asked questions

Common questions

Bounded answers for the checks that most often get conflated.

Can robots.txt keep a page private?

No. It is not an access-control system. Require authentication or another appropriate protection for private material.

Can I use robots.txt to remove a page from search?

Not reliably. If a crawler cannot fetch the page, it cannot read page-level noindex instructions. Use the removal approach appropriate to the page state instead.

PUT THE WORKFLOW TO USE

Turn technical SEO checks into a repeatable workflow.

Use Spacebrain to organize the work, the evidence, and the next owner around every release.

Start for free