The direct answer
Decide whether content may be public before choosing its search policy. Robots crawl restrictions and noindex instructions do not replace authorization for private AI results; a crawler must be able to read noindex for that indexing instruction to take effect.
Classify three surfaces before changing a rule
A fictional AI application has three HTML surfaces: a reviewed educational guide P-P, a deliberately public synthetic example P-U, and a workspace answer P-X. The team wants the guide discoverable, the example reachable without appearing in Search, and the workspace answer available only to authorized users. One “hide from bots” setting cannot express all three decisions.
Start with disclosure, not search appearance. P-P and P-U contain only independently admitted public material. P-X requires current recipient, workspace and object authorization before releasing its answer or revealing metadata. These are proposed requirements; no actual route or access mechanism was inspected or created.
This is separate from tenant authorization and download-link access. Those controls remain necessary. The question here is what Google Search crawl and indexing instructions establish after the content’s access class is known.
A crawl restriction is not an indexing verdict
Google’s robots introduction describes robots.txt as crawler-traffic control rather than a way to keep web pages out of Google. A URL blocked from crawling can still appear in results when discovered through other pages, potentially with publicly available link information. The document also explains that robots instructions cannot force every crawler to obey.
For P-X, therefore, a disallow decision cannot become “private result protected.” The application must enforce access independently. Nor can a crawl-blocked URL be labeled undiscoverable. A descriptive public link can reveal information without publishing the protected answer itself.
Keep result identifiers, sensitive titles and usable capabilities out of public navigation and diagnostic examples under the application’s disclosure policy. Do not publish a revealing route list as an access-control substitute. This guide provides no robots syntax, crawler identity exception or bot-specific login bypass.
A readable noindex instruction belongs on already public content
Google’s noindex documentation says that after Googlebot crawls a page and extracts its noindex instruction, Google drops it from Search. The page must not be blocked by robots.txt and must otherwise be accessible to that crawler. A rule the crawler cannot read does not establish that result.
This dependency can support P-U because its content is deliberately public. The selected HTML plan requires an effective supported noindex meta instruction in the document head, accessible through its admitted public route. Google documents that placing noindex in robots.txt is unsupported.
The same reasoning must not be inverted for P-X. Do not temporarily remove authentication, disclose its answer or grant a privileged crawler path merely to expose an indexing instruction. Keep access protection first; any search-related recovery is a separate authorized task with its own evidence. The public example’s plan is not an exception for private results.
Preserve a useful public path and a clear private boundary
The following original sheet uses labels only. It creates no URL, page, robots file, meta tag or response header.
| Surface | May anonymous readers receive its contents? | Separate search decision |
|---|---|---|
| P-P: reviewed public guide G-A | Yes, under explicit editorial release. | Ordinary supported crawl/index eligibility; no indexing or traffic outcome claimed. |
| P-U: admitted synthetic example U-A | Yes. It contains no protected result data. | Readable supported HTML noindex plan; still publicly accessible. |
| P-X: workspace result X-A | No. Enforce current object authorization before content release. | Index policy never authorizes a crawl or disclosure of private contents. |
The positive keeps both an ordinary public guide and a useful public demonstration available, without presenting the latter as private. For P-X, the positive is an authorized user’s normal result access under independently established controls, not publication to search engines.
Actual authorization, origin and CDN behavior, effective HTML instructions and public-content classification are unimplemented prerequisites. Unknown evidence blocks the relevant release plan. A policy label alone supplies neither an enforced permission nor a verified indexing disposition.
Treat a changed policy and an observed result as different evidence
Google’s debugging guidance explains that it must crawl again to see changed tags or headers, and revisits can take months. A locally changed page therefore supplies no immediate removal observation. This article performs no inspection, recrawl request, submission or removal operation.
Keep separate records for content classification, access-control evidence, intended crawl policy, intended index instruction and actually observed Search state. Unknown stays unknown. A missing result in a casual observation would not establish that all copies, references or other search systems have disappeared.
If previously public content becomes restricted, first contain further unauthorized distribution through its responsible application workflow. Preserve what was disclosed and which copies remain uncertain. The incident-response guide covers a different disclosure route; neither a later noindex change nor an authorization repair recalls bytes already obtained.
Walk eight publication-policy cards
| Condition | Required decision |
|---|---|
| P-P is reviewed public content with its admitted normal search plan | Conditionally admit public release; discovery, indexing and traffic remain unobserved. |
| P-U is deliberately public and its supported HTML noindex instruction is readable | Conditionally admit that search-excluded public example plan; do not label it private. |
| P-X relies only on robots or noindex | Reject the private-release claim; require actual current authorization. |
| Crawl blocking prevents reading a newly added noindex instruction | Do not claim the indexing rule was seen; resolve only for already public content. |
| A blocked URL is discovered from an external reference | No automatic no-index or hidden-existence conclusion. |
| A previously public answer now needs restriction | Contain further disclosure first; keep index updates and retained copies unresolved until evidenced. |
| Public metadata contains a sensitive title or usable capability | Reject that exposure independently of crawler instructions. |
| Another crawler, content format or effective serving path is introduced | Outside the selected profile; re-review without assuming equivalent behavior. |
Keep the evidence and product scope explicit
Both selected Google Search Central pages displayed an update date of December 10, 2025. Their documented Google Search behavior is not a universal promise about other crawlers, AI training, citations or content reuse. This guide establishes no ranking, traffic, removal timing or indexing result.
For these proposed cases, no site, crawler, URL, robots file, meta tag, header, Search Console action, authorization mechanism, browser or runtime fixture was exercised. Only the declared documentation was read; ordinary private editorial file handling is separate.
Boundary is a prelaunch model router with planned request-path checks, not a shipped access-control or search-removal service. Its synthetic action exercise and existing mock evidence do not test these route classes, crawler rules or Google Search outcomes.
Primary sources
Official references supporting the guidance. Examples in this guide are synthetic and are not reports of a production incident.