20 sites · scanned live
The state of the agent web
Where 20 well-known sites stand today on AI access — who opens the door to agents and who slams it. A solid dot is an explicit rule for that vendor; a ring is the outcome inherited from User-agent: *.
AllowedPartialBlockedUnspecified (inherits *)
| Site | OpenAI | Anthropic | Google AI | Perplexity | ByteDance | Common Crawl | Meta | llms.txt | Manifest | |
|---|---|---|---|---|---|---|---|---|---|---|
| airbnb.com | — | — | View | |||||||
| amazon.com | — | — | View | |||||||
| anthropic.com | — | — | View | |||||||
| booking.com | — | — | View | |||||||
| cloudflare.com | yes | yes | View | |||||||
| ebay.com | — | — | View | |||||||
| expedia.com | yes | — | View | |||||||
| github.com | yes | yes | View | |||||||
| linkedin.com | — | — | View | |||||||
| nytimes.com | — | — | View | |||||||
| openai.com | — | — | View | |||||||
| reddit.com | — | — | View | |||||||
| ryanair.com | — | — | View | |||||||
| shopify.com | yes | — | View | |||||||
| skyscanner.net | — | — | View | |||||||
| stackoverflow.com | — | — | View | |||||||
| stripe.com | yes | — | View | |||||||
| theguardian.com | — | — | View | |||||||
| vercel.com | yes | yes | View | |||||||
| wikipedia.org | — | — | View |
Scanned 20/20 reachable · · generator SIbotaccessScanner