Skip to content
SIbotaccess

airbnb.com

Scanned

/api/site/airbnb.com

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBot
Training crawler
Disallow: /
Blocked
OAI-SearchBot
Search / answer index
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Partial
ChatGPT-User
On-demand user fetch
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Partial

Anthropic

ClaudeBot
Training crawler
Disallow: /
Blocked
Claude-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Unspecifiedeffective: partial
Claude-SearchBotinherited
Search / answer index
inherits User-agent: *
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Unspecifiedeffective: partial

Google AI

Google-Extendedinherited
Training crawler
inherits User-agent: *
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Unspecifiedeffective: partial

Perplexity

PerplexityBot
Search / answer index
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+57 more disallow rules
Partial
Perplexity-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Unspecifiedeffective: partial

ByteDance

Bytespiderinherited
Training crawler
inherits User-agent: *
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Unspecifiedeffective: partial

Common Crawl

CCBotinherited
Training crawler
inherits User-agent: *
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Unspecifiedeffective: partial

Meta

Meta-ExternalAgent
Training crawler
Disallow: /.well-known/assetlinks.json
Disallow: /*/skeleton
Disallow: /*/sw_skeleton
Disallow: /500
Disallow: /account
Disallow: /alumni
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
+55 more disallow rules
Partial
Back to the directory