Website checks
Indexability checks after changing a site, CDN, or security setup
A site can look fine to a normal browser and still send confusing signals to crawlers. After changing hosting, CDN settings, security rules, or routes, check the public signals that affect discovery and indexing.
Extractly6 min read
Key takeaways
- Check the live public URL, not only the local build.
- Look for robots, noindex, canonical, redirects, and response status together.
- Recheck important pages after security or CDN edits.
Start with the final public response
Open the final URL that visitors and crawlers will use. Confirm it returns the intended page, not a challenge screen, redirect loop, stale route, or custom error page.
Compare robots and sitemap
Robots.txt should not block pages that appear in the sitemap. The sitemap should list stable canonical URLs and avoid pages that are private, duplicated, or removed.
Read page-level signals
A page may be discoverable but still contain a noindex directive, a canonical to another page, or metadata that no longer matches the content. Check those fields on the live page.
Watch security tools carefully
CDN rules, bot challenges, rate limits, and custom firewall pages can affect crawlers and preview bots. Use targeted rules where possible, then verify the public pages still load normally.
Indexability debug checklist
- Important pages return clean 200 responses.
- Robots.txt and sitemap do not conflict.
- No important page has accidental noindex.
- CDN or security rules do not replace public content with a challenge page.