AI crawlers: separate training, search, and user fetches
AI providers may use separate agents for training, search indexing, and user-triggered fetches. Verify official documentation, robots rules, IP evidence, server logs, and outcomes.
AI crawlers
“AI crawler” is a broad label for automated web access associated with AI providers. One company may use different user agents for model-development crawling, search discovery, and a fetch initiated by a user. Those roles are not interchangeable.
Allowing an agent means a compliant crawler may request permitted URLs. It does not guarantee indexing, training use, retrieval, citation, or traffic. Blocking one agent does not necessarily govern a provider’s other agents, licensed datasets, existing copies, or user-supplied content.
Build a role-specific register
For every agent relevant to your site, record:
| Field | Why it matters |
|---|---|
| Provider and exact user-agent token | Similar names can have different purposes |
| Documented purpose | Training, search, user fetch, or another product function |
| robots.txt behavior | Rules may differ by agent and provider |
| Verification method | Published IP ranges, reverse DNS, signatures, or no method |
| Policy URL and review date | Documentation changes over time |
| Observed requests | Logs show actual access, not intended behavior |
Current examples include OpenAI’s separate GPTBot, OAI-SearchBot, and ChatGPT-User roles; Anthropic’s documented bots; PerplexityBot and Perplexity-User; Google’s Google-Extended product token; and Applebot-Extended. Do not copy a static list without checking each provider.
Use the current primary documentation: OpenAI’s publisher and developer FAQ, Anthropic crawler controls, Perplexity crawlers, Google’s crawler list, and Applebot.
Make an access decision
- Define the content class: public marketing, documentation, licensed material, paywalled work, personal data, or confidential content.
- Identify the business objective and the exact agent related to it.
- Review contracts, copyright, privacy, security, and publisher policy—not only SEO traffic hopes.
- Apply the narrowest rule that reflects the decision, and repeat it for relevant subdomains.
- Test
robots.txt, edge rules, authentication, and caching behavior. - Monitor verified requests, status codes, byte volume, latency, and abuse patterns.
Robots.txt is a crawl directive for cooperating agents, not access control. Sensitive content requires authentication and authorization. Schema does not control what a crawler may fetch, while llms.txt is not a permission mechanism.
Interpret logs correctly
A user-agent string can be spoofed. Use the provider’s documented verification method where available and label unverified traffic separately. A successful request proves only that a request occurred and received a response. Keep crawler access, indexing, citations, referral sessions, and conversions as separate measurements.