nytimes.com

Scanned 2026-10-22 11:28:35 UTC

/api/site/nytimes.com

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBot

Training crawler

Disallow: /

Blocked

OAI-SearchBot

Search / answer index

Disallow: /

Blocked

ChatGPT-User

On-demand user fetch

Disallow: /

Blocked

Anthropic

ClaudeBot

Training crawler

Disallow: /

Blocked

Claude-User

On-demand user fetch

Disallow: /

Blocked

Claude-SearchBot

Search / answer index

Disallow: /

Blocked

Google AI

Google-Extended

Training crawler

Disallow: /

Blocked

Perplexity

PerplexityBot

Search / answer index

Disallow: /

Blocked

Perplexity-User

On-demand user fetch

Disallow: /

Blocked

ByteDance

Bytespider

Training crawler

Disallow: /

Blocked

Common Crawl

CCBot

Training crawler

Disallow: /

Blocked

Files found

/robots.txt

Crawl rules

HTTP 200present

/llms.txt

Content index for LLMs

HTTP 404absent

/.well-known/agents.json

Agent manifest (A2A)

HTTP 404absent

/.well-known/ai-catalog.json

AI content catalog

HTTP 404absent

/.well-known/botaccess.json

Action declaration (this spec)

HTTP 404absent

Relevant robots.txt

Only the groups matching the AI user-agents we track (plus the wildcard).

User-agent: *

Disallow: /ads/
Disallow: /adx/bin/
Disallow: /athletic/wp/wp-admin/
Allow: /athletic/wp/wp-admin/admin-ajax.php
Disallow: /athletic/async-*
Disallow: /athletic/search/*
Allow: /athletic/search/$
Disallow: /athletic/checkout/
Disallow: /athletic/checkout?plan_id*
Allow: /athletic/checkout/$
Disallow: /athletic/checkout2*
Disallow: /athletic/login/
Disallow: /athletic/login?login_source*
Disallow: /athletic/login?ref_page*
Allow: /athletic/login/$
Disallow: /athletic/login2/
Disallow: /athletic/login2?login_source*
Disallow: /athletic/login2?ref_page*
Allow: /athletic/login2/$
Disallow: /athletic/report/
Disallow: /athletic/*/discuss/*
Allow: /athletic/live-blogs/discuss/
Allow: /athletic/nfl/game/discuss/
Disallow: /athletic/register/
Disallow: /athletic/register?welcome_redirect*
Disallow: /athletic/register2/
Disallow: /athletic/register2?welcome_redirect*
Disallow: /athletic/betmgm-redirect*
Disallow: /athletic/cdn-cgi/
Disallow: /athletic/verizon/*
Disallow: /athletic/forgot-password/*
Disallow: /athletic/forgot-password2/*
Disallow: /athletic/amp-social-login*
Disallow: /athletic/track-analytics/
Disallow: /athletic/amp-auth/
Disallow: /athletic/rss-feed/
Disallow: /athletic/*?*rss=1
Disallow: /athletic/global-color-test.php
Disallow: /athletic/global-font-test.php
Disallow: /athletic/graphql*

User-agent: bytespider

Disallow: /

User-agent: ccbot

Disallow: /

User-agent: chatgpt-user

Disallow: /

User-agent: claudebot

Disallow: /

User-agent: claude-searchbot

Disallow: /

User-agent: claude-user

Disallow: /

User-agent: google-extended

Disallow: /

User-agent: gptbot

Disallow: /

User-agent: meta-externalagent

Disallow: /
Disallow: /

User-agent: oai-searchbot

Disallow: /

User-agent: perplexitybot

Disallow: /

User-agent: perplexity-user

Disallow: /

Back to the directory

nytimes.com

Access by AI user-agent

OpenAI

Anthropic

Google AI

Perplexity

ByteDance

Common Crawl

Meta