Skip to content
SIbotaccess

cloudflare.com

Scanned

/api/site/cloudflare.com

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBot
Training crawler
Allow: /
Allowed
OAI-SearchBotinherited
Search / answer index
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
ChatGPT-User
On-demand user fetch
Allow: /
Allowed

Anthropic

ClaudeBotinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Claude-Userinherited
On-demand user fetch
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Claude-SearchBotinherited
Search / answer index
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed

Google AI

Google-Extended
Training crawler
Allow: /
Allowed

Perplexity

PerplexityBot
Search / answer index
Allow: /
Allowed
Perplexity-Userinherited
On-demand user fetch
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed

ByteDance

Bytespiderinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed

Common Crawl

CCBot
Training crawler
Allow: /
Allowed

Meta

Meta-ExternalAgentinherited
Training crawler
inherits User-agent: *
Allow: /
Unspecifiedeffective: allowed
Back to the directory