amazon.com

Scanned 2026-10-22 11:28:36 UTC

/api/site/amazon.com

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBot

Training crawler

Disallow: /

Blocked

OAI-SearchBot

Search / answer index

Disallow: /

Blocked

ChatGPT-User

On-demand user fetch

Disallow: /

Blocked

Anthropic

ClaudeBot

Training crawler

Disallow: /

Blocked

Claude-User

On-demand user fetch

Disallow: /

Blocked

Claude-SearchBot

Search / answer index

Disallow: /

Blocked

Google AI

Google-Extended

Training crawler

Disallow: /

Blocked

Perplexity

PerplexityBot

Search / answer index

Disallow: /

Blocked

Perplexity-User

On-demand user fetch

Disallow: /

Blocked

ByteDance

Bytespider

Training crawler

Disallow: /

Blocked

Common Crawl

CCBot

Training crawler

Disallow: /

Blocked

Files found

/robots.txt

Crawl rules

HTTP 200present

/llms.txt

Content index for LLMs

HTTP 404absent

/.well-known/agents.json

Agent manifest (A2A)

HTTP 404absent

/.well-known/ai-catalog.json

AI content catalog

HTTP 404absent

/.well-known/botaccess.json

Action declaration (this spec)

HTTP 404absent

Relevant robots.txt

Only the groups matching the AI user-agents we track (plus the wildcard).

User-agent: *

Disallow: /exec/obidos/account-access-login
Disallow: /exec/obidos/change-style
Disallow: /exec/obidos/flex-sign-in
Disallow: /exec/obidos/handle-buy-box
Disallow: /exec/obidos/tg/cm/member/
Disallow: /gp/aw/help/id=sss
Disallow: /gp/cart
Disallow: /gp/flex
Disallow: /gp/product/e-mail-friend
Disallow: /gp/product/product-availability
Disallow: /gp/product/rate-this-item
Disallow: /gp/sign-in
Disallow: /gp/reader
Disallow: /gp/sitbv3/reader
Disallow: /gp/richpub/syltguides/create
Disallow: /gp/gfix
Disallow: /gp/associations/wizard.html
Disallow: /gp/dmusic/order
Disallow: /gp/legacy-handle-buy-box.html
Disallow: /gp/aws/ssop
Disallow: /gp/yourstore
Disallow: /gp/gift-central/organizer/add-wishlist
Disallow: /gp/vote
Disallow: /gp/voting/
Disallow: /gp/music/wma-pop-up
Disallow: /gp/customer-images
Disallow: /gp/richpub/listmania/createpipeline
Disallow: /gp/content-form
Disallow: /gp/pdp/invitation/invite
Disallow: /gp/customer-reviews/common/du
Disallow: /gp/customer-reviews/write-a-review.html
Disallow: /gp/associations/wizard.html
Disallow: /gp/music/clipserve
Disallow: /gp/customer-media/upload
Disallow: /gp/history
Disallow: /gp/item-dispatch
Disallow: /gp/dmusic/order/handle-buy-box.html
Disallow: /gp/recsradio
Disallow: /gp/slredirect
Disallow: /dp/shipping/

User-agent: gptbot

Disallow: /

User-agent: ccbot

Disallow: /

User-agent: perplexitybot

Disallow: /

User-agent: google-extended

Disallow: /

User-agent: claudebot

Disallow: /
Disallow: /

User-agent: meta-externalagent

Disallow: /

User-agent: bytespider

Disallow: /

User-agent: claude-user

Disallow: /

User-agent: claude-searchbot

Disallow: /

User-agent: perplexity-user

Disallow: /

User-agent: chatgpt-user

Disallow: /

User-agent: oai-searchbot

Disallow: /

Back to the directory

amazon.com

Access by AI user-agent

OpenAI

Anthropic

Google AI

Perplexity

ByteDance

Common Crawl

Meta