Skip to content
SIbotaccess

theguardian.com

Scanned

/api/site/theguardian.com

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBotinherited
Training crawler
inherits User-agent: *
Disallow: /sendarticle/
Disallow: /Users/
Disallow: /users/
Disallow: /*/print$
Disallow: /email/
Disallow: /contactus/
+42 more disallow rules
Unspecifiedeffective: partial
OAI-SearchBotinherited
Search / answer index
inherits User-agent: *
Disallow: /sendarticle/
Disallow: /Users/
Disallow: /users/
Disallow: /*/print$
Disallow: /email/
Disallow: /contactus/
+42 more disallow rules
Unspecifiedeffective: partial
ChatGPT-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /sendarticle/
Disallow: /Users/
Disallow: /users/
Disallow: /*/print$
Disallow: /email/
Disallow: /contactus/
+42 more disallow rules
Unspecifiedeffective: partial

Anthropic

ClaudeBot
Training crawler
Disallow: /
Blocked
Claude-User
On-demand user fetch
Disallow: /
Blocked
Claude-SearchBot
Search / answer index
Disallow: /
Blocked

Google AI

Google-Extendedinherited
Training crawler
inherits User-agent: *
Disallow: /sendarticle/
Disallow: /Users/
Disallow: /users/
Disallow: /*/print$
Disallow: /email/
Disallow: /contactus/
+42 more disallow rules
Unspecifiedeffective: partial

Perplexity

PerplexityBot
Search / answer index
Disallow: /
Blocked
Perplexity-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /sendarticle/
Disallow: /Users/
Disallow: /users/
Disallow: /*/print$
Disallow: /email/
Disallow: /contactus/
+42 more disallow rules
Unspecifiedeffective: partial

ByteDance

Bytespider
Training crawler
Disallow: /
Blocked

Common Crawl

CCBot
Training crawler
Disallow: /
Blocked

Meta

Meta-ExternalAgent
Training crawler
Disallow: /
Blocked
Back to the directory