Separate crawling from indexing

An allowed crawl does not mean a page is indexed. Robots.txt tells a crawler whether it can request the page; noindex is an indexing instruction in page HTML or an HTTP header. Canonical suggests the representative address among duplicates. Each has a distinct purpose.

If you block a page in robots.txt, the crawler may not read its noindex instruction. An address can sometimes still appear in search results without a content snippet. For confidential material, require proper authentication and access control rather than relying on robots.txt.

Use the rule tester for crawl permission and the page checker for the current server HTML and X-Robots-Tag header. JavaScript-rendered changes are outside the page checker. A fetched HTML snapshot also does not show what a crawler saw last week.

To confirm actual status, inspect the URL in the verified search engine webmaster property. Record when the report was fetched, whether crawling was allowed, which canonical was selected and the indexing decision. Do not infer guaranteed ranking from a clean technical check.