ClaudeBot

ClaudeBot is Anthropic’s web crawler, used to keep Claude’s knowledge current and to power citations in its answers. Decide how to handle it in your robots.txt.

Claude is widely used by knowledge workers and developers. Being cited by Claude is a high-value signal for B2B and technical content.

How to control ClaudeBot

• Allow. Default. Lets Anthropic crawl your content for Claude’s answers and ongoing training
• Block. Add to robots.txt:

  User-agent: ClaudeBot
  Disallow: /
  

• Block only newer model training. Use Claude-User (which is used for user-initiated fetches) and Claude-SearchBot (which powers live search) separately:

  User-agent: ClaudeBot
  Disallow: /
  User-agent: Claude-User
  Allow: /
  User-agent: Claude-SearchBot
  Allow: /
  

Why allow it

• Your content can be cited in Claude’s answers
• It drives AI referral traffic from a high-intent audience
• It contributes to your citation rate across the Anthropic surface

Why block it

• You do not want your content used for training future Claude models
• You have content licensing restrictions
• You are in a regulated industry

How to verify

• Check your server logs for User-Agent: ClaudeBot, User-Agent: Claude-User, and User-Agent: Claude-SearchBot
• Use Anthropic’s published IP ranges to confirm
• Test by asking Claude about a topic you have published on