ClaudeBot: Anthropic's model-development crawler
ClaudeBot collects public web content for potential model development. Compare it with Claude-SearchBot and Claude-User before changing robots.txt.
ClaudeBot
ClaudeBot is Anthropic’s crawler for public web content that could contribute to future model-training datasets. Anthropic separates it from Claude-SearchBot, which supports search-result quality, and Claude-User, which retrieves content for user-directed requests.
The separation lets site owners express different policies for model development, search discovery, and user-initiated retrieval.
Anthropic’s three bot tokens
| User-agent | Documented purpose | If blocked |
|---|---|---|
ClaudeBot | Potential model-training collection | Signals that future site material should be excluded from training datasets |
Claude-SearchBot | Search indexing and result quality | May reduce visibility and accuracy in Claude search results |
Claude-User | Retrieval initiated by a Claude user | Prevents retrieval for those user-directed requests |
This means “block ClaudeBot” and “block Claude” are not equivalent statements.
A split robots.txt policy
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Use this only if it matches the organization’s legal and product policy. Repeat the policy on each hostname where it should apply.
Implementation and verification
- Fetch the live
/robots.txtwithout a browser cache and confirm the groups are not accidentally merged. - Check CDN and WAF events for all three tokens. An edge denial can override a permissive robots.txt rule.
- Inspect response codes for actual content URLs, not only
/robots.txt. - Monitor crawl volume after deployment and record the change date.
- Review Anthropic’s documentation periodically because crawler behavior and names can change.
Anthropic currently says it does not publish stable IP ranges for these bots. Therefore, this page does not recommend an IP allowlist as proof of identity. Use user-agent observations as operational evidence while recognizing that user-agent strings can be spoofed.
What access does not guarantee
Allowing Claude-SearchBot does not guarantee indexing, citation, ranking, or AI referral traffic. Those are downstream selection decisions. Measure them separately with logs, analytics referrals, and a repeatable prompt set; do not label a citation as a direct result of one robots.txt change without a controlled before-and-after test.
robots.txt is also not a privacy boundary. Protect sensitive or licensed material with authentication, authorization, and contractual controls.
Primary source
See also: AI crawlers, robots.txt, and citation rate.