stackoverflow.com
Scanned
Access by AI user-agent
Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.
OpenAI
GPTBotinherited
Training crawler
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
OAI-SearchBotinherited
Search / answer index
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
ChatGPT-Userinherited
On-demand user fetch
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Anthropic
ClaudeBotinherited
Training crawler
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Claude-Userinherited
On-demand user fetch
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Claude-SearchBotinherited
Search / answer index
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Google AI
Google-Extendedinherited
Training crawler
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Perplexity
PerplexityBotinherited
Search / answer index
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Perplexity-Userinherited
On-demand user fetch
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
ByteDance
Bytespiderinherited
Training crawler
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Common Crawl
CCBotinherited
Training crawler
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed
Meta
Meta-ExternalAgentinherited
Training crawler
robots.txt unreadable (HTTP 403) — RFC 9309 treats 4xx as allow-all
Allowed