anthropic.com
Scanned
Access by AI user-agent
Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.
OpenAI
GPTBotinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
OAI-SearchBotinherited
Search / answer index
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
ChatGPT-Userinherited
On-demand user fetch
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Anthropic
ClaudeBotinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Claude-Userinherited
On-demand user fetch
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Claude-SearchBotinherited
Search / answer index
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Google AI
Google-Extendedinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Perplexity
PerplexityBotinherited
Search / answer index
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Perplexity-Userinherited
On-demand user fetch
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
ByteDance
Bytespiderinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Common Crawl
CCBotinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Meta
Meta-ExternalAgentinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed