Robots.txt is a public set of crawler instructions located at the root of a host. It can reduce crawling of selected paths for compliant crawlers, but it does not authenticate users, protect confidential files, or reliably remove known URLs from search results. Small syntax changes can affect an entire site.
Decision snapshot
| Decision | Practical approach | Watch for |
|---|---|---|
| Rules are host-specific | Protocol, hostname, and port each have their own robots.txt scope. | Testing one hostname does not prove rules on another. |
| Disallow is not noindex | A blocked URL may still be known through links and cannot expose page-level noindex to the crawler. | Use the indexing control appropriate to the goal. |
| Public means discoverable | Do not list secret administrative or backup paths as a protective measure. | The file itself can advertise sensitive locations. |
Define the exact crawl problem
Name the crawler class, URL pattern, server or duplicate-load issue, desired behavior, and why canonicalization, authentication, status codes, or noindex is not the better control.
Inventory current hosts and rules
Fetch robots.txt from apex, www, subdomains, HTTP and HTTPS behavior, production and accidental public environments. Review user-agent groups, allow and disallow paths, wildcards, end anchors, and sitemap lines.
Model affected URLs
Test representative allowed, blocked, valuable, parameterized, case-sensitive, encoded, asset, API, and trailing-slash variants. Remember that path matching begins from the URL root.
Deploy with a narrow change
Back up the current file, avoid broad root rules unless intentionally taking the site out of crawl, preserve required CSS and JavaScript access, publish as plain text, and keep a rollback command.
Verify and monitor
Fetch as an external client, test important patterns, inspect logs and Search Console where available, confirm sitemaps remain reachable, and review after migrations or new route structures.
Action checklist
- Robots file is reachable as plain text at each relevant host root
- Each rule has a documented crawler and URL-pattern purpose
- Important canonical pages and rendering resources remain crawlable
- Sensitive content is protected by authentication or authorization, not robots.txt
- Removal and duplicate goals use appropriate index or canonical controls
- Change has representative tests, owner, deploy record, and rollback
Working worksheet
Record these fields in the same working document so the decision can be reviewed and handed off:
- Host and environment
- Crawler group and URL pattern
- Current match, desired match, and reason
- Representative allow and disallow test URLs
- Deployment, external verification, owner, and rollback result
Common failure patterns
- Publishing Disallow: / during a migration and forgetting to remove it
- Blocking parameters that are also required to discover canonical content
- Using robots.txt to hide private documents, staging sites, or admin exports
Connect this work
The two files should send consistent host and URL signals. Read keep sitemap discovery aligned with crawl controls.
A block can affect rendering and canonical processing. Read test robots rules in the wider indexability audit.
Environment rules often leak into production during deployment. Read include robots review in launch preflight.