THE SHORT ANSWER
An AI crawler policy can restrict training while preserving selected search access, where the provider offers separate controls. Decide by purpose and provider, then check both robots.txt and your hosting rules. Access makes retrieval possible; it does not guarantee a citation, recommendation or visit.
Start an AI crawler policy with three decisions
A marketing team may want its product pages found in AI search without approving every use of its content. An SEO team may also need to control unwanted server load. Those goals need a written access policy before someone changes a broad bot setting.
A September 27, 2026 content-marketing discussion asks whether teams distinguish training crawlers from search access. A July 9, 2026 technical SEO discussion asks which bots to allow, block or monitor. These are public questions, not evidence that any particular policy improves visibility.
- Training: which documented model-development uses will you permit?
- Search: which services should be able to discover your public pages?
- User requests: which assistants may fetch a page on someone’s behalf?
Check the provider, not just the bot name
OpenAI’s crawler documentation separates GPTBot training controls from OAI-SearchBot search controls. ChatGPT-User handles user-triggered visits; its robots.txt behavior differs and it does not control search inclusion. Record these as separate decisions.
Google’s Google-Extended documentation describes a control covering future Gemini training and certain grounding uses. It does not control Google Search inclusion or ranking. Do not describe this as a universal training-only switch: grounding is also within its documented scope.
Perplexity’s crawler guidance distinguishes PerplexityBot search discovery from Perplexity-User requests. For Google AI Overviews and AI Mode, consult Google’s AI search requirements. Keep each provider’s rules attached to its own entry in your policy.
Reconcile robots.txt with hosting controls
Your policy needs to cover the content delivery network (CDN), web application firewall (WAF) and origin server. A permissive robots.txt file cannot undo a blocked request at the hosting layer. Our guide to robots.txt, sitemaps and llms.txt explains what the discovery files themselves can do.
Cloudflare’s AI bot policy documentation distinguishes Search, Agent and Training behavior. It also describes mixed-purpose crawlers. Inspect your actual settings and rule matches; the label “block training” is not enough to establish which search access remains.
Before editing, save the current configuration and choose a few public pages to test. Include the homepage, a service page and an article. Ask the technical owner to review crawler-specific groups and path restrictions together. Keep private areas protected by authentication.
Verify access before reporting visibility
Collect a dated request sample at the CDN or server. Separate claimed bot names from verified identities. Perplexity’s linked guidance recommends combining user-agent and published IP checks. A request that merely copies a bot’s user-agent string is not proof of that provider’s access.
Check the response body as well as the status. A successful status carrying a challenge page does not deliver the article. Preserve the matched rule and redirect chain when available. If no verified request appears, report access as unobserved rather than inventing a crawl outcome.
- Record: time, hostname, path, claimed agent and identity-verification method.
- Inspect: response status, actual content, redirects and the rule responsible for any block.
- Decide: keep, narrow or reverse the change using the agreed policy and observed evidence.
Give the policy an owner and review trigger
Keep one record per provider: purpose, affected paths, chosen control, source documentation, owner and review date. Revisit it after a hosting change or provider update. Where a control combines training and retrieval, record the tradeoff instead of promising both outcomes.
Use our AI visibility audit playbook to investigate answers after access checks. Report crawling, citations and visits separately with the AI visibility measurement guide. Citevia’s SEO service can help coordinate the technical review. The first deliverable is a verified access decision, not a promised ranking.
AI crawler policy: practical questions
Does blocking GPTBot also block ChatGPT search?
No. OpenAI documents separate controls for GPTBot and OAI-SearchBot. Review both entries and the hosting rules before concluding that search access is open. Allowing access still does not guarantee inclusion.
Should I allow a bot just because it identifies itself by name?
No. Check the provider’s current verification guidance, including published IP ranges where available. Keep unverified requests separate. A broad exception based only on a claimed name can admit unrelated traffic.
Can robots.txt protect confidential documents?
No. Use authentication and access controls for private documents. A crawler preference is not an access barrier. Review public files and downloads separately from the pages that link to them.
Why do I still see no citations after allowing search crawlers?
Access is only one condition. Confirm actual retrieval first, then inspect the relevance and accuracy of your pages against a documented question sample. No observed citation does not, by itself, prove that a crawler was blocked.
What should a small team check first?
Choose one important public page and one provider. Compare the intended policy, deployed robots.txt and hosting rules, then inspect verified requests. Save the evidence and assign an owner before expanding to more providers.