Publishing
Robots, sitemaps, and indexability without the mystery
If a page should be public, it needs more than a good design. Crawlers need to reach it, understand the preferred URL, and avoid mixed signals from robots, sitemaps, canonicals, and noindex rules.
Extractly6 min read
Key takeaways
- Robots.txt answers whether crawlers may request a path.
- A sitemap is a discovery hint, not a promise of indexing.
- Noindex and canonical tags can change what happens after a page is found.
Check whether crawlers can enter
The robots.txt file tells crawlers which paths they may request. A broad disallow rule can keep valuable pages from being fetched, while a missing file is usually not a problem when the site is intentionally public.
Use the sitemap as a clean map
An XML sitemap lists canonical URLs that the site owner wants crawlers to discover. It should include stable public pages that return clean responses, not old staging URLs, duplicate filters, or temporary experiments.
Look for the signals on the page too
A page can be linked in a sitemap and still ask search engines not to index it through a meta robots tag or HTTP header. Redirects, canonical tags, blocked assets, and server errors also shape how a page is understood.
Recheck after hosting or security changes
Routing, hosting, security rules, and CDN settings can change crawler access. Recheck important pages after moving domains, changing frameworks, editing security settings, or adding a new content section.
A short crawl checklist
- Homepage, tools, guides, contact, and policy pages return 200 responses.
- Robots.txt does not block important public paths.
- Sitemap lists only canonical public URLs.
- Important pages do not contain accidental noindex directives.